[jira] [Updated] (SPARK-47319) Improve missingInput calculation

2024-03-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47319: --- Description: {{QueryPlan.missingInput()}} calculation seems to be the root cause of

[jira] [Updated] (SPARK-47319) Improve missingInput calculation

2024-03-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47319: --- Description: {{QueryPlan.missingInput()}} calculation seems to be the root cause of

[jira] [Updated] (SPARK-47319) Improve missingInput calculation

2024-03-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47319: --- Summary: Improve missingInput calculation (was: Fix missingInput calculation) > Improve

[jira] [Created] (SPARK-47319) Fix missingInput calculation

2024-03-07 Thread Peter Toth (Jira)
Peter Toth created SPARK-47319: -- Summary: Fix missingInput calculation Key: SPARK-47319 URL: https://issues.apache.org/jira/browse/SPARK-47319 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-03-01 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-47217: --- Shepherd: (was: Peter Toth) > De-duplication of Relations in Joins, can result in plan resolution

[jira] [Resolved] (SPARK-45805) Eliminate magic numbers in withOrigin

2023-11-06 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-45805. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43671

[jira] [Resolved] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-45354. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43146

[jira] [Assigned] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth reassigned SPARK-45354: -- Assignee: Peter Toth > Resolve functions bottom-up > --- > >

[jira] [Updated] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45354: --- Description: This PR proposes bottum-up resolution in  {{{}ResolveFunctions{}}}, which is much

[jira] [Created] (SPARK-45354) Resolve functions bottom-up

2023-09-27 Thread Peter Toth (Jira)
Peter Toth created SPARK-45354: -- Summary: Resolve functions bottom-up Key: SPARK-45354 URL: https://issues.apache.org/jira/browse/SPARK-45354 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-45216) Fix non-deterministic seeded Dataset APIs

2023-09-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45216: --- Description: If we run the following example the result is the expected equal 2 columns:

[jira] [Created] (SPARK-45216) Fix non-deterministic seeded Dataset APIs

2023-09-19 Thread Peter Toth (Jira)
Peter Toth created SPARK-45216: -- Summary: Fix non-deterministic seeded Dataset APIs Key: SPARK-45216 URL: https://issues.apache.org/jira/browse/SPARK-45216 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45112) Use UnresolvedFunction based resolution in SQL Dataset functions

2023-09-11 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45112: --- Summary: Use UnresolvedFunction based resolution in SQL Dataset functions (was: Use

[jira] [Created] (SPARK-45112) Use UnresolvedFunction in dataset functions

2023-09-10 Thread Peter Toth (Jira)
Peter Toth created SPARK-45112: -- Summary: Use UnresolvedFunction in dataset functions Key: SPARK-45112 URL: https://issues.apache.org/jira/browse/SPARK-45112 Project: Spark Issue Type:

[jira] [Updated] (SPARK-45109) Fix eas_decrypt and ln in connect

2023-09-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45109: --- Description: The current {{eas_descrypt}} reference to {{aes_encrypt}} is clearly a bug. The {{ln}} 

[jira] [Created] (SPARK-45109) Fix eas_decrypt and ln in connect

2023-09-10 Thread Peter Toth (Jira)
Peter Toth created SPARK-45109: -- Summary: Fix eas_decrypt and ln in connect Key: SPARK-45109 URL: https://issues.apache.org/jira/browse/SPARK-45109 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45022) Provide context for dataset API errors

2023-08-31 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-45022: --- Description: SQL failures already provide nice error context when there is a failure: {noformat}

[jira] [Created] (SPARK-45034) Support deterministic mode function

2023-08-31 Thread Peter Toth (Jira)
Peter Toth created SPARK-45034: -- Summary: Support deterministic mode function Key: SPARK-45034 URL: https://issues.apache.org/jira/browse/SPARK-45034 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-45022) Provide context for dataset API errors

2023-08-30 Thread Peter Toth (Jira)
Peter Toth created SPARK-45022: -- Summary: Provide context for dataset API errors Key: SPARK-45022 URL: https://issues.apache.org/jira/browse/SPARK-45022 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

2023-08-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth reassigned SPARK-44934: -- Assignee: Wen Yuen Pang > PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan

[jira] [Updated] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

2023-08-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44934: --- Affects Version/s: 3.3.3 > PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when

[jira] [Resolved] (SPARK-44934) PushdownPredicatesAndPruneColumnsForCTEDef creates invalid plan when called over CTE with duplicate attributes

2023-08-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-44934. Fix Version/s: 3.3.4 3.5.0 4.0.0 3.4.2

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Fix Version/s: 3.5.0 > Fix PERCENTILE_DISC behaviour > - > >

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Affects Version/s: 3.4.0 3.3.2 3.3.1

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Affects Version/s: 3.4.1 3.3.3 (was: 3.3.0)

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Fix Version/s: 3.5.0 4.0.0 > Fix PERCENTILE_DISC behaviour >

[jira] [Commented] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756097#comment-17756097 ] Peter Toth commented on SPARK-44871: [~tgraves], sure, I've just updated it. It looks like my PR

[jira] [Updated] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-44871: --- Description: Currently {{percentile_disc()}} returns incorrect results in some cases: E.g.:

[jira] [Created] (SPARK-44871) Fix PERCENTILE_DISC behaviour

2023-08-18 Thread Peter Toth (Jira)
Peter Toth created SPARK-44871: -- Summary: Fix PERCENTILE_DISC behaviour Key: SPARK-44871 URL: https://issues.apache.org/jira/browse/SPARK-44871 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-43266) Move MergeScalarSubqueries to spark-sql

2023-04-24 Thread Peter Toth (Jira)
Peter Toth created SPARK-43266: -- Summary: Move MergeScalarSubqueries to spark-sql Key: SPARK-43266 URL: https://issues.apache.org/jira/browse/SPARK-43266 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-43199) Make InlineCTE idempotent

2023-04-19 Thread Peter Toth (Jira)
Peter Toth created SPARK-43199: -- Summary: Make InlineCTE idempotent Key: SPARK-43199 URL: https://issues.apache.org/jira/browse/SPARK-43199 Project: Spark Issue Type: Bug Components:

[jira] [Comment Edited] (SPARK-24497) ANSI SQL: Recursive query

2023-04-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714114#comment-17714114 ] Peter Toth edited comment on SPARK-24497 at 4/19/23 2:00 PM: - I've opened a

[jira] [Commented] (SPARK-24497) ANSI SQL: Recursive query

2023-04-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714114#comment-17714114 ] Peter Toth commented on SPARK-24497: I've opened a new PR:

[jira] [Created] (SPARK-43124) Dataset.show should not trigger job execution on CommandResults

2023-04-13 Thread Peter Toth (Jira)
Peter Toth created SPARK-43124: -- Summary: Dataset.show should not trigger job execution on CommandResults Key: SPARK-43124 URL: https://issues.apache.org/jira/browse/SPARK-43124 Project: Spark

[jira] [Updated] (SPARK-42852) Revert NamedLambdaVariable related changes from EquivalentExpressions

2023-03-20 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42852: --- Affects Version/s: (was: 3.3.2) > Revert NamedLambdaVariable related changes from

[jira] [Resolved] (SPARK-42836) Support for recursive queries

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-42836. Resolution: Duplicate > Support for recursive queries > - > >

[jira] [Commented] (SPARK-42836) Support for recursive queries

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17702138#comment-17702138 ] Peter Toth commented on SPARK-42836: Please ask on the already existing ticket to let others know

[jira] [Updated] (SPARK-42852) Revert NamedLambdaVariable related changes from EquivalentExpressions

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42852: --- Summary: Revert NamedLambdaVariable related changes from EquivalentExpressions (was: Revert

[jira] [Updated] (SPARK-42852) Revert NamedLambdaVariables related changes from EquivalentExpressions

2023-03-18 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42852: --- Summary: Revert NamedLambdaVariables related changes from EquivalentExpressions (was: Rervert

[jira] [Created] (SPARK-42852) Rervert NamedLambdaVariables related changes from EquivalentExpressions

2023-03-18 Thread Peter Toth (Jira)
Peter Toth created SPARK-42852: -- Summary: Rervert NamedLambdaVariables related changes from EquivalentExpressions Key: SPARK-42852 URL: https://issues.apache.org/jira/browse/SPARK-42852 Project: Spark

[jira] [Updated] (SPARK-42745) Improved AliasAwareOutputExpression works with DSv2

2023-03-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42745: --- Description: After SPARK-40086 / SPARK-42049 the following, simple subselect expression containing

[jira] [Updated] (SPARK-42745) Improved AliasAwareOutputExpression works with DSv2

2023-03-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42745: --- Summary: Improved AliasAwareOutputExpression works with DSv2 (was: Fix NPE after recent

[jira] [Created] (SPARK-42745) Fix NPE after recent AliasAwareOutputExpression changes

2023-03-10 Thread Peter Toth (Jira)
Peter Toth created SPARK-42745: -- Summary: Fix NPE after recent AliasAwareOutputExpression changes Key: SPARK-42745 URL: https://issues.apache.org/jira/browse/SPARK-42745 Project: Spark Issue

[jira] [Created] (SPARK-42438) Improve constraint propagation using multiTransform

2023-02-14 Thread Peter Toth (Jira)
Peter Toth created SPARK-42438: -- Summary: Improve constraint propagation using multiTransform Key: SPARK-42438 URL: https://issues.apache.org/jira/browse/SPARK-42438 Project: Spark Issue Type:

[jira] [Created] (SPARK-42436) Improve multiTransform to generate alternatives dynamically

2023-02-14 Thread Peter Toth (Jira)
Peter Toth created SPARK-42436: -- Summary: Improve multiTransform to generate alternatives dynamically Key: SPARK-42436 URL: https://issues.apache.org/jira/browse/SPARK-42436 Project: Spark

[jira] [Created] (SPARK-42435) Update DataTables to 1.13.2

2023-02-14 Thread Peter Toth (Jira)
Peter Toth created SPARK-42435: -- Summary: Update DataTables to 1.13.2 Key: SPARK-42435 URL: https://issues.apache.org/jira/browse/SPARK-42435 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685858#comment-17685858 ] Peter Toth edited comment on SPARK-42346 at 2/8/23 11:16 AM: - [~ritikam],

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-08 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685858#comment-17685858 ] Peter Toth commented on SPARK-42346: [~ritikam], you also need to disable the

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-06 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685124#comment-17685124 ] Peter Toth commented on SPARK-42346: [~ritikam], please use the Pyspark repro in description or add

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-05 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684272#comment-17684272 ] Peter Toth commented on SPARK-42346: [~yumwang], [~RobinLinacre],

[jira] [Updated] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-05 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-42346: --- Affects Version/s: 3.3.0 3.4.0 3.5.0

[jira] [Commented] (SPARK-42346) distinct(count colname) with UNION ALL causes query analyzer bug

2023-02-04 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684184#comment-17684184 ] Peter Toth commented on SPARK-42346: Thanks for pinging me [~yumwang], this might be subquery merge

[jira] [Created] (SPARK-42136) Refactor BroadcastHashJoinExec output partitioning generation

2023-01-20 Thread Peter Toth (Jira)
Peter Toth created SPARK-42136: -- Summary: Refactor BroadcastHashJoinExec output partitioning generation Key: SPARK-42136 URL: https://issues.apache.org/jira/browse/SPARK-42136 Project: Spark

[jira] [Created] (SPARK-42134) Fix getPartitionFiltersAndDataFilters() to handle filters without referenced attributes

2023-01-20 Thread Peter Toth (Jira)
Peter Toth created SPARK-42134: -- Summary: Fix getPartitionFiltersAndDataFilters() to handle filters without referenced attributes Key: SPARK-42134 URL: https://issues.apache.org/jira/browse/SPARK-42134

[jira] [Created] (SPARK-41468) Fix PlanExpression handling in EquivalentExpressions

2022-12-09 Thread Peter Toth (Jira)
Peter Toth created SPARK-41468: -- Summary: Fix PlanExpression handling in EquivalentExpressions Key: SPARK-41468 URL: https://issues.apache.org/jira/browse/SPARK-41468 Project: Spark Issue Type:

[jira] [Created] (SPARK-41367) Enable V2 file tables in read paths in session catalog

2022-12-02 Thread Peter Toth (Jira)
Peter Toth created SPARK-41367: -- Summary: Enable V2 file tables in read paths in session catalog Key: SPARK-41367 URL: https://issues.apache.org/jira/browse/SPARK-41367 Project: Spark Issue

[jira] [Created] (SPARK-41124) Add DSv2 PlanStabilitySuites

2022-11-13 Thread Peter Toth (Jira)
Peter Toth created SPARK-41124: -- Summary: Add DSv2 PlanStabilitySuites Key: SPARK-41124 URL: https://issues.apache.org/jira/browse/SPARK-41124 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40874: --- Description: The following Pyspark script: {noformat} bin/pyspark --conf

[jira] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874 ] Peter Toth deleted comment on SPARK-40874: was (Author: petertoth): The following Pyspark script: {noformat} bin/pyspark --conf spark.io.encryption.enabled=true ... bar = {"a": "aa", "b":

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40874: --- The following Pyspark script: {noformat} bin/pyspark --conf spark.io.encryption.enabled=true ... bar

[jira] [Updated] (SPARK-40874) Fix broadcasts in Python UDFs when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40874: --- Summary: Fix broadcasts in Python UDFs when encryption is enabled (was: Fix Python UDFs with

[jira] [Created] (SPARK-40874) Fix Python UDFs with broadcasts when encryption is enabled

2022-10-21 Thread Peter Toth (Jira)
Peter Toth created SPARK-40874: -- Summary: Fix Python UDFs with broadcasts when encryption is enabled Key: SPARK-40874 URL: https://issues.apache.org/jira/browse/SPARK-40874 Project: Spark Issue

[jira] [Updated] (SPARK-40599) Add multiTransform methods to TreeNode to generate alternatives

2022-09-28 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40599: --- Summary: Add multiTransform methods to TreeNode to generate alternatives (was: Add multiTransform

[jira] [Created] (SPARK-40599) Add multiTransform methods to TreeNode to generate alternative transformations

2022-09-28 Thread Peter Toth (Jira)
Peter Toth created SPARK-40599: -- Summary: Add multiTransform methods to TreeNode to generate alternative transformations Key: SPARK-40599 URL: https://issues.apache.org/jira/browse/SPARK-40599 Project:

[jira] [Updated] (SPARK-40259) Support Parquet DSv2 in subquery plan merge

2022-08-29 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40259: --- Description: We could improve SPARK-34079 with DSv2 support. (was: We could improve SPARK-34079 to

[jira] [Created] (SPARK-40259) Support Parquet DSv2 in subquery plan merge

2022-08-29 Thread Peter Toth (Jira)
Peter Toth created SPARK-40259: -- Summary: Support Parquet DSv2 in subquery plan merge Key: SPARK-40259 URL: https://issues.apache.org/jira/browse/SPARK-40259 Project: Spark Issue Type:

[jira] [Created] (SPARK-40247) Fix BitSet equality check

2022-08-28 Thread Peter Toth (Jira)
Peter Toth created SPARK-40247: -- Summary: Fix BitSet equality check Key: SPARK-40247 URL: https://issues.apache.org/jira/browse/SPARK-40247 Project: Spark Issue Type: Bug Components:

[jira] [Updated] (SPARK-40245) Fix FileScan equality check when partition or data filter columns are not read

2022-08-27 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40245: --- Summary: Fix FileScan equality check when partition or data filter columns are not read (was: Fix

[jira] [Created] (SPARK-40245) Fix FileScan canonicalization when partition or data filter columns are not read

2022-08-27 Thread Peter Toth (Jira)
Peter Toth created SPARK-40245: -- Summary: Fix FileScan canonicalization when partition or data filter columns are not read Key: SPARK-40245 URL: https://issues.apache.org/jira/browse/SPARK-40245

[jira] [Updated] (SPARK-40193) Merge subquery plans with different filters

2022-08-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-40193: --- Summary: Merge subquery plans with different filters (was: Merge different filters when merging

[jira] [Created] (SPARK-40193) Merge different filters when merging subquery plans

2022-08-23 Thread Peter Toth (Jira)
Peter Toth created SPARK-40193: -- Summary: Merge different filters when merging subquery plans Key: SPARK-40193 URL: https://issues.apache.org/jira/browse/SPARK-40193 Project: Spark Issue Type:

[jira] [Created] (SPARK-40086) Improve AliasAwareOutputPartitioning to take all aliases into account

2022-08-15 Thread Peter Toth (Jira)
Peter Toth created SPARK-40086: -- Summary: Improve AliasAwareOutputPartitioning to take all aliases into account Key: SPARK-40086 URL: https://issues.apache.org/jira/browse/SPARK-40086 Project: Spark

[jira] [Created] (SPARK-38717) Handle Hive's bucket spec case preserving behaviour

2022-03-31 Thread Peter Toth (Jira)
Peter Toth created SPARK-38717: -- Summary: Handle Hive's bucket spec case preserving behaviour Key: SPARK-38717 URL: https://issues.apache.org/jira/browse/SPARK-38717 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26639) The reuse subquery function maybe does not work in SPARK SQL

2022-03-25 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512270#comment-17512270 ] Peter Toth commented on SPARK-26639: [~stubartmess], that's a different issue but it is fixed in

[jira] [Resolved] (SPARK-28299) Evaluation of multiple CTE uses

2021-11-24 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth resolved SPARK-28299. Resolution: Duplicate > Evaluation of multiple CTE uses > - > >

[jira] [Commented] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

2021-11-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447980#comment-17447980 ] Peter Toth commented on SPARK-37259: I've opened a PR: [https://github.com/apache/spark/pull/34693]

[jira] [Commented] (SPARK-37259) JDBC read is always going to wrap the query in a select statement

2021-11-19 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446554#comment-17446554 ] Peter Toth commented on SPARK-37259: [~KevinAppelBofa], how about adding a new `withClause` to the

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-09-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419388#comment-17419388 ] Peter Toth commented on SPARK-35672: I put up a revert PR:

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-09-23 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419285#comment-17419285 ] Peter Toth commented on SPARK-35672: [~xkrogen], [~tgraves], unfortunately, I think this is a

[jira] [Commented] (SPARK-36065) date_trunc returns incorrect output

2021-07-30 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390421#comment-17390421 ] Peter Toth commented on SPARK-36065: I think the output is correct as there was a time zone change

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-12 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Issue Type: Improvement (was: Bug) > EquivalentExpressions fixes and improvements >

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Description: Currently `EquivalentExpressions` has 2 issues: - identifying common expressions in

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Issue Type: Bug (was: Improvement) > EquivalentExpressions fixes and improvements >

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Priority: Major (was: Minor) > EquivalentExpressions fixes and improvements >

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Description: Fixes an issue with identifying common expressions in conditional expressions (a

[jira] [Updated] (SPARK-36073) EquivalentExpressions fixes and improvements

2021-07-10 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-36073: --- Summary: EquivalentExpressions fixes and improvements (was: SubExpr elimination should include

[jira] [Created] (SPARK-36073) SubExpr elimination should include common child exprs of conditional expressions

2021-07-09 Thread Peter Toth (Jira)
Peter Toth created SPARK-36073: -- Summary: SubExpr elimination should include common child exprs of conditional expressions Key: SPARK-36073 URL: https://issues.apache.org/jira/browse/SPARK-36073

[jira] [Created] (SPARK-35855) Unify reuse map data structures in non-AQE and AQE rules

2021-06-22 Thread Peter Toth (Jira)
Peter Toth created SPARK-35855: -- Summary: Unify reuse map data structures in non-AQE and AQE rules Key: SPARK-35855 URL: https://issues.apache.org/jira/browse/SPARK-35855 Project: Spark Issue

[jira] [Created] (SPARK-35798) Fix SparkPlan.sqlContext usage

2021-06-17 Thread Peter Toth (Jira)
Peter Toth created SPARK-35798: -- Summary: Fix SparkPlan.sqlContext usage Key: SPARK-35798 URL: https://issues.apache.org/jira/browse/SPARK-35798 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-34801) java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.metadata.Hive.loadPartition

2021-03-22 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306093#comment-17306093 ] Peter Toth commented on SPARK-34801: Yes it is. Please use CDS 3 (Cloudera Distribution of Spark 3)

[jira] [Updated] (SPARK-34581) BoundAttribute issue after optimization by BooleanSimplification and PushFoldableIntoBranches

2021-03-21 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-34581: --- Affects Version/s: 3.0.2 3.1.1 > BoundAttribute issue after optimization by

[jira] [Updated] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2021-03-16 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-33482: --- Affects Version/s: 3.0.0 3.0.1 3.0.2

[jira] [Updated] (SPARK-34756) Fix FileScan equality check

2021-03-16 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-34756: --- Affects Version/s: 3.1.0 > Fix FileScan equality check > --- > >

[jira] [Created] (SPARK-34756) Fix FileScan equality check

2021-03-16 Thread Peter Toth (Jira)
Peter Toth created SPARK-34756: -- Summary: Fix FileScan equality check Key: SPARK-34756 URL: https://issues.apache.org/jira/browse/SPARK-34756 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-34726) Fix collectToPython timeouts

2021-03-12 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-34726: --- Affects Version/s: (was: 3.1.1) 2.4.7 > Fix collectToPython timeouts >

[jira] [Created] (SPARK-34726) Fix collectToPython timeouts

2021-03-12 Thread Peter Toth (Jira)
Peter Toth created SPARK-34726: -- Summary: Fix collectToPython timeouts Key: SPARK-34726 URL: https://issues.apache.org/jira/browse/SPARK-34726 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-30655) Update WebUI Bootstrap to 4.4.1

2021-03-09 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17298005#comment-17298005 ] Peter Toth commented on SPARK-30655: [~d.clarke], it looks like you already fixed this in 

[jira] [Commented] (SPARK-24497) ANSI SQL: Recursive query

2021-02-02 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277367#comment-17277367 ] Peter Toth commented on SPARK-24497: Thanks [~ilaurens] for your comment. Recursive queries are very

[jira] [Commented] (SPARK-32041) Exchange reuse won't work in cases when DPP, subqueries are involved

2021-01-26 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272102#comment-17272102 ] Peter Toth commented on SPARK-32041: Let me reopen this ticket as this is not a duplicate of

[jira] [Reopened] (SPARK-32041) Exchange reuse won't work in cases when DPP, subqueries are involved

2021-01-26 Thread Peter Toth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth reopened SPARK-32041: > Exchange reuse won't work in cases when DPP, subqueries are involved >

  1   2   3   >