[jira] [Commented] (SPARK-49690) UDT type is not expanded into its StructType in the schema definition

2024-09-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883148#comment-17883148 ] Asif commented on SPARK-49690: -- Not sure if this is a bug or expected behaviour. Given tha

[jira] [Commented] (SPARK-49727) A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset

2024-09-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883147#comment-17883147 ] Asif commented on SPARK-49727: -- This issue is addressed in the PR for [https://issues.apac

[jira] [Updated] (SPARK-49727) A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset

2024-09-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-49727: - Labels: pull-request-available (was: ) > A Bean class with a serializable POJO field looses data when converted

[jira] [Updated] (SPARK-49727) A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset

2024-09-19 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-49727: - Description: If a Bean class contains a serializable POJO  ( without any getter/setter) property,  the encoder

[jira] [Created] (SPARK-49727) A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset

2024-09-19 Thread Asif (Jira)
Asif created SPARK-49727: Summary: A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset Key: SPARK-49727 URL: https://issues.apache.org/jira/browse/SPARK-49727

[jira] [Commented] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

2024-09-18 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882866#comment-17882866 ] Asif commented on SPARK-46679: -- [~andoni.teso]  Hi. I have opened a PR for this. Though it

[jira] [Updated] (SPARK-49690) UDT type is not expanded into its StructType in the schema definition

2024-09-17 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-49690: - Description: A UDT type field does not show up as constituent struct type in the schema. Instead it shows up as

[jira] [Updated] (SPARK-49690) UDT type is not expanded into its StructType in the schema definition

2024-09-17 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-49690: - Description: A UDT type field does not show up as constituent struct type in the schema. Instead it shows up as

[jira] [Created] (SPARK-49690) UDT type is not expanded into its StructType in the schema definition

2024-09-17 Thread Asif (Jira)
Asif created SPARK-49690: Summary: UDT type is not expanded into its StructType in the schema definition Key: SPARK-49690 URL: https://issues.apache.org/jira/browse/SPARK-49690 Project: Spark Issue

[jira] [Created] (SPARK-49618) Union ( & UnionExec) nodes equality not take into account unaligned positions of branches causing NO ( reuse of exchange and cached plans)

2024-09-12 Thread Asif (Jira)
Asif created SPARK-49618: Summary: Union ( & UnionExec) nodes equality not take into account unaligned positions of branches causing NO ( reuse of exchange and cached plans) Key: SPARK-49618 URL: https://issues.apache.org

[jira] [Commented] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

2024-09-01 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878487#comment-17878487 ] Asif commented on SPARK-46679: -- Sorry.. there has been delay in internal review.. will subm

[jira] [Commented] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

2024-08-23 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876379#comment-17876379 ] Asif commented on SPARK-46679: -- [~andoni.teso] Hi. I will be opening a PR for this in a day

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Affects Version/s: 4.0.0 > SPIP: Constraint Propagation code causes OOM issues or increasing compilation > time

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Affects Version/s: 4.0.0 > Canonicalization of DynamicPruningSubquery is broken > --

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Affects Version/s: 4.0.0 > SPIP: Improving performance of BroadcastHashJoin queries with stream side > join key

[jira] [Updated] (SPARK-45926) The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45926: - Affects Version/s: 4.0.0 > The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing > equals an

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Affects Version/s: 4.0.0 > Minimizing calls to HiveMetaStore layer for getting partitions, when tables > are r

[jira] [Updated] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-46671: - Affects Version/s: 4.0.0 > InferFiltersFromConstraint rule is creating a redundant filter >

[jira] [Updated] (SPARK-45959) SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Affects Version/s: 4.0.0 > SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf > degradation

[jira] [Updated] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45866: - Affects Version/s: 4.0.0 > Reuse of exchange in AQE does not happen when run time filters are pushed > down to

[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Affects Version/s: 4.0.0 > CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan >

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-07-30 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Affects Version/s: 4.0.0 > Datasets involving self joins behave in an inconsistent and unintuitive > manner >

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-06-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Labels: SPIP pull-request-available (was: SPIP) > SPIP: Constraint Propagation code causes OOM issues or increa

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2024-06-07 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Shepherd: Peter Toth (was: Wenchen Fan) > Minimizing calls to HiveMetaStore layer for getting partitions, when

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2024-06-06 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Shepherd: Wenchen Fan > Minimizing calls to HiveMetaStore layer for getting partitions, when tables > are repe

[jira] [Updated] (SPARK-45959) SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2024-05-29 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Priority: Major (was: Minor) > SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf > degrada

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2024-05-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Priority: Major (was: Minor) > Minimizing calls to HiveMetaStore layer for getting partitions, when tables >

[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-03-27 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Description: This issue became apparent while bringing my PR  [https://github.com/apache/spark/pull/43854] in s

[jira] [Updated] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47609: - Description: This issue became apparent while bringing my PR  [https://github.com/apache/spark/pull/43854] in s

[jira] [Created] (SPARK-47609) CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan

2024-03-26 Thread Asif (Jira)
Asif created SPARK-47609: Summary: CacheManager Lookup can miss picking InMemoryRelation corresponding to subplan Key: SPARK-47609 URL: https://issues.apache.org/jira/browse/SPARK-47609 Project: Spark

[jira] [Comment Edited] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831116#comment-17831116 ] Asif edited comment on SPARK-26708 at 3/27/24 12:58 AM: I believ

[jira] [Comment Edited] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831117#comment-17831117 ] Asif edited comment on SPARK-26708 at 3/27/24 12:54 AM: Towards

[jira] [Commented] (SPARK-26708) Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

2024-03-26 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831116#comment-17831116 ] Asif commented on SPARK-26708: -- I believe the current caching logic is suboptimal and accor

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when Analy

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Labels: pull-request-available (was: ) > Datasets involving self joins behave in an inconsistent and unintuitiv

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when Analy

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when Analy

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when Analy

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when Analy

[jira] [Updated] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47320: - Description: The behaviour of Datasets involving self joins behave in an unintuitive manner in terms when Analy

[jira] [Commented] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824877#comment-17824877 ] Asif commented on SPARK-47320: -- Opened following PR [https://github.com/apache/spark/pull/4

[jira] [Commented] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-07 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824589#comment-17824589 ] Asif commented on SPARK-47320: -- will be linking the bug to an open PR > Datasets involving

[jira] [Created] (SPARK-47320) Datasets involving self joins behave in an inconsistent and unintuitive manner

2024-03-07 Thread Asif (Jira)
Asif created SPARK-47320: Summary: Datasets involving self joins behave in an inconsistent and unintuitive manner Key: SPARK-47320 URL: https://issues.apache.org/jira/browse/SPARK-47320 Project: Spark

[jira] [Commented] (SPARK-39441) Speed up DeduplicateRelations

2024-03-06 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824102#comment-17824102 ] Asif commented on SPARK-39441: -- this issue should be resolved by the PR for ticket [https:

[jira] [Comment Edited] (SPARK-39441) Speed up DeduplicateRelations

2024-03-06 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824102#comment-17824102 ] Asif edited comment on SPARK-39441 at 3/6/24 5:33 PM: -- this issue s

[jira] [Comment Edited] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823510#comment-17823510 ] Asif edited comment on SPARK-33152 at 3/5/24 6:43 PM: -- [~tedjenks]

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823512#comment-17823512 ] Asif commented on SPARK-33152: -- other than using my PR, the safe option would be to disable

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823510#comment-17823510 ] Asif commented on SPARK-33152: -- [~tedjenks] .. Unfortunately I am not a committer. As part

[jira] [Commented] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2024-03-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823344#comment-17823344 ] Asif commented on SPARK-33152: -- [~tedjenks]  The issue has always been there  because of th

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47217: - Description: In case of some flavours of  nested joins involving repetition of relation, the projected columns

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47217: - Description: In case of some flavours of self join queries or nested joins involving repetition of relation, th

[jira] [Updated] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-47217: - Description: In case of some flavours of nested self join queries, the projected columns when passed to the Da

[jira] [Created] (SPARK-47217) De-duplication of Relations in Joins, can result in plan resolution failure

2024-02-28 Thread Asif (Jira)
Asif created SPARK-47217: Summary: De-duplication of Relations in Joins, can result in plan resolution failure Key: SPARK-47217 URL: https://issues.apache.org/jira/browse/SPARK-47217 Project: Spark

[jira] [Updated] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-46671: - Description: while bring my old PR which uses a different approach to the ConstraintPropagation algorithm ( [

[jira] [Reopened] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif reopened SPARK-46671: -- After further analysis , I believe , that what I said originally in the ticket is valid and that the code Does cr

[jira] [Resolved] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-46671. -- Resolution: Not A Bug > InferFiltersFromConstraint rule is creating a redundant filter > -

[jira] [Commented] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805434#comment-17805434 ] Asif commented on SPARK-46671: -- on further thoughts , I am wrong.. There should be 2 separa

[jira] [Commented] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-11 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17805435#comment-17805435 ] Asif commented on SPARK-46671: -- so closing the ticket > InferFiltersFromConstraint rule is

[jira] [Created] (SPARK-46671) InferFiltersFromConstraint rule is creating a redundant filter

2024-01-10 Thread Asif (Jira)
Asif created SPARK-46671: Summary: InferFiltersFromConstraint rule is creating a redundant filter Key: SPARK-46671 URL: https://issues.apache.org/jira/browse/SPARK-46671 Project: Spark Issue Type: B

[jira] [Updated] (SPARK-45959) SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2024-01-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Description: Though documentation clearly recommends to add all columns in a single shot, but in reality is dif

[jira] [Updated] (SPARK-45959) SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2024-01-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Summary: SPIP: Abusing DataSet.withColumn can cause huge tree with severe perf degradation (was: Abusing DataSe

[jira] [Updated] (SPARK-45959) Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2023-11-16 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45959: - Priority: Minor (was: Major) > Abusing DataSet.withColumn can cause huge tree with severe perf degradation > --

[jira] [Commented] (SPARK-45959) Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2023-11-16 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786941#comment-17786941 ] Asif commented on SPARK-45959: -- will create a PR for the same.. > Abusing DataSet.withColu

[jira] [Created] (SPARK-45959) Abusing DataSet.withColumn can cause huge tree with severe perf degradation

2023-11-16 Thread Asif (Jira)
Asif created SPARK-45959: Summary: Abusing DataSet.withColumn can cause huge tree with severe perf degradation Key: SPARK-45959 URL: https://issues.apache.org/jira/browse/SPARK-45959 Project: Spark

[jira] [Commented] (SPARK-45943) DataSourceV2Relation.computeStats throws IllegalStateException in test mode

2023-11-16 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786652#comment-17786652 ] Asif commented on SPARK-45943: -- thanks [~wforget] for the input.. if you have solution pls

[jira] [Updated] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45866: - Labels: pull-request-available (was: ) > Reuse of exchange in AQE does not happen when run time filters are pus

[jira] [Created] (SPARK-45943) DataSourceV2Relation.computeStats throws IllegalStateException in test mode

2023-11-15 Thread Asif (Jira)
Asif created SPARK-45943: Summary: DataSourceV2Relation.computeStats throws IllegalStateException in test mode Key: SPARK-45943 URL: https://issues.apache.org/jira/browse/SPARK-45943 Project: Spark

[jira] [Closed] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif closed SPARK-45924. this is not a bug > Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not > equivalent with Subquer

[jira] [Closed] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif closed SPARK-45925. this is not an issue > SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec > causing re-use o

[jira] [Resolved] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-45924. -- Resolution: Not A Bug > Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not > equivalent w

[jira] [Resolved] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-15 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif resolved SPARK-45925. -- Resolution: Not A Problem > SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec > caus

[jira] [Commented] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786155#comment-17786155 ] Asif commented on SPARK-45866: -- Now that the other PRs on which this ticket itself is depen

[jira] [Updated] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45925: - Labels: pull-request-available (was: ) > SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcast

[jira] [Updated] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45924: - Labels: pull-request-available (was: ) > Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not

[jira] [Created] (SPARK-45926) The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly

2023-11-14 Thread Asif (Jira)
Asif created SPARK-45926: Summary: The InMemoryV2FilterBatchScan and InMemoryBatchScan are not implementing equals and hashCode correctly Key: SPARK-45926 URL: https://issues.apache.org/jira/browse/SPARK-45926

[jira] [Created] (SPARK-45925) SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE

2023-11-14 Thread Asif (Jira)
Asif created SPARK-45925: Summary: SubqueryBroadcastExec is not equivalent with SubqueryAdaptiveBroadcastExec causing re-use of exchange not happening in AQE Key: SPARK-45925 URL: https://issues.apache.org/jira/browse/SPA

[jira] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658 ] Asif deleted comment on SPARK-45658: -- was (Author: ashahid7): I also think that during canonicalization of DynamicPruningSubquery, the pruning key's canonicalization should be done on the basis of

[jira] [Updated] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-14 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45924: - Description: while writing bug test for [SPARK-45866|https://issues.apache.org/jira/projects/SPARK/issues/SPAR

[jira] [Created] (SPARK-45924) Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec

2023-11-14 Thread Asif (Jira)
Asif created SPARK-45924: Summary: Canonicalization of SubqueryAdaptiveBroadcastExec is broken and is not equivalent with SubqueryBroadcastExec Key: SPARK-45924 URL: https://issues.apache.org/jira/browse/SPARK-45924

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Shepherd: (was: Peter Toth) > Minimizing calls to HiveMetaStore layer for getting partitions, when tables >

[jira] [Updated] (SPARK-33152) SPIP: Constraint Propagation code causes OOM issues or increasing compilation time to hours

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-33152: - Affects Version/s: 3.5.0 (was: 2.4.0) (was: 3.0.1)

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Affects Version/s: 3.5.0 (was: 4.0.0) > Minimizing calls to HiveMetaStore layer for g

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Affects Version/s: 3.5.0 (was: 3.5.1) > SPIP: Improving performance of BroadcastHashJ

[jira] [Created] (SPARK-45866) Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg )

2023-11-09 Thread Asif (Jira)
Asif created SPARK-45866: Summary: Reuse of exchange in AQE does not happen when run time filters are pushed down to the underlying Scan ( like iceberg ) Key: SPARK-45866 URL: https://issues.apache.org/jira/browse/SPARK-4

[jira] [Commented] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-11-09 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784567#comment-17784567 ] Asif commented on SPARK-45658: -- I also think that during canonicalization of DynamicPruning

[jira] [Commented] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-08 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784282#comment-17784282 ] Asif commented on SPARK-44662: -- The changes for iceberg which support broadcast-var-pushdow

[jira] [Commented] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782927#comment-17782927 ] Asif commented on SPARK-44662: -- The majority of file changes are due to additional tpcds te

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Attachment: perf results broadcast var pushdown - Partitioned TPCDS.pdf > SPIP: Improving performance of Broadca

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On th

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Affects Version/s: 3.5.1 (was: 3.3.3) > SPIP: Improving performance of BroadcastHashJ

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On th

[jira] [Updated] (SPARK-44662) SPIP: Improving performance of BroadcastHashJoin queries with stream side join key on non partition columns

2023-11-04 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-44662: - Description: h2. *Q1. What are you trying to do? Articulate your objectives using absolutely no jargon.* On th

[jira] [Commented] (SPARK-36786) SPIP: Improving the compile time performance, by improving a couple of rules, from 24 hrs to under 8 minutes

2023-11-01 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781976#comment-17781976 ] Asif commented on SPARK-36786: -- I had put this on back burner as my changes were on 3.2, so

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Description: The canonicalization of (buildKeys: Seq[Expression]) in the class DynamicPruningSubquery is broken

[jira] [Updated] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45658: - Priority: Major (was: Critical) > Canonicalization of DynamicPruningSubquery is broken > --

[jira] [Created] (SPARK-45658) Canonicalization of DynamicPruningSubquery is broken

2023-10-24 Thread Asif (Jira)
Asif created SPARK-45658: Summary: Canonicalization of DynamicPruningSubquery is broken Key: SPARK-45658 URL: https://issues.apache.org/jira/browse/SPARK-45658 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-10-05 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Affects Version/s: 4.0.0 (was: 3.5.1) > Minimizing calls to HiveMetaStore layer for g

[jira] [Updated] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-29 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-45373: - Description: In the rule PruneFileSourcePartitions where the CatalogFileIndex gets converted to InMemoryFileInd

[jira] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-29 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373 ] Asif deleted comment on SPARK-45373: -- was (Author: ashahid7): Will be generating a PR for this. > Minimizing calls to HiveMetaStore layer for getting partitions, when tables > are repeated >

[jira] [Commented] (SPARK-45373) Minimizing calls to HiveMetaStore layer for getting partitions, when tables are repeated

2023-09-28 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17770220#comment-17770220 ] Asif commented on SPARK-45373: -- Will be generating a PR for this. > Minimizing calls to Hi

  1   2   >