[jira] [Created] (SPARK-38160) Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend

2022-02-09 Thread EdisonWang (Jira)
EdisonWang created SPARK-38160: -- Summary: Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend Key: SPARK-38160 URL: https://issues.apache.org/jira/browse/SPARK-38160 Project:

[jira] [Assigned] (SPARK-37952) Add missing statements to ALTER TABLE document.

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37952: --- Assignee: Yuto Akutsu > Add missing statements to ALTER TABLE document. >

[jira] [Resolved] (SPARK-37952) Add missing statements to ALTER TABLE document.

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37952. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35239

[jira] [Resolved] (SPARK-37770) Performance improvements for ColumnVector `putByteArray`

2022-02-09 Thread Yaohua Zhao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yaohua Zhao resolved SPARK-37770. - Resolution: Fixed > Performance improvements for ColumnVector `putByteArray` >

[jira] [Created] (SPARK-38161) when clean data hope to spilt one dataframe or dataset to two dataframe

2022-02-09 Thread gaokui (Jira)
gaokui created SPARK-38161: -- Summary: when clean data hope to spilt one dataframe or dataset to two dataframe Key: SPARK-38161 URL: https://issues.apache.org/jira/browse/SPARK-38161 Project: Spark

[jira] [Assigned] (SPARK-38150) Update comment of RelationConversions

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38150: --- Assignee: angerszhu > Update comment of RelationConversions >

[jira] [Commented] (SPARK-38159) Minor refactor of MetadataAttribute unapply method

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489359#comment-17489359 ] Apache Spark commented on SPARK-38159: -- User 'Yaohua628' has created a pull request for this issue:

[jira] [Resolved] (SPARK-38150) Update comment of RelationConversions

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38150. - Resolution: Fixed Issue resolved by pull request 35453

[jira] [Assigned] (SPARK-38120) HiveExternalCatalog.listPartitions is failing when partition column name is upper case and dot in partition value

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38120: --- Assignee: Khalid Mammadov > HiveExternalCatalog.listPartitions is failing when partition

[jira] [Resolved] (SPARK-38120) HiveExternalCatalog.listPartitions is failing when partition column name is upper case and dot in partition value

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38120. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-38159) Minor refactor of MetadataAttribute unapply method

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38159: Assignee: Apache Spark > Minor refactor of MetadataAttribute unapply method >

[jira] [Assigned] (SPARK-38159) Minor refactor of MetadataAttribute unapply method

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38159: Assignee: (was: Apache Spark) > Minor refactor of MetadataAttribute unapply method >

[jira] [Resolved] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-38056. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35356

[jira] [Assigned] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-38056: Assignee: wy > Structured streaming not working in history server when using LevelDB >

[jira] [Created] (SPARK-38159) Minor refactor of MetadataAttribute unapply method

2022-02-09 Thread Yaohua Zhao (Jira)
Yaohua Zhao created SPARK-38159: --- Summary: Minor refactor of MetadataAttribute unapply method Key: SPARK-38159 URL: https://issues.apache.org/jira/browse/SPARK-38159 Project: Spark Issue Type:

[jira] [Created] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Max Gekk (Jira)
Max Gekk created SPARK-38163: Summary: Preserve the error class of `AnalysisException` while constructing of function builder Key: SPARK-38163 URL: https://issues.apache.org/jira/browse/SPARK-38163

[jira] [Commented] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489482#comment-17489482 ] Max Gekk commented on SPARK-38163: -- I am working on the fix. > Preserve the error class of

[jira] [Commented] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489523#comment-17489523 ] Willi Raschkowski commented on SPARK-38166: --- Attaching driver logs: [^driver.log] Notable

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Abhinav Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489537#comment-17489537 ] Abhinav Kumar commented on SPARK-38061: --- [~hyukjin.kwon] [~sujitbiswas] Are we agreeing to track

[jira] [Created] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-09 Thread Marnix van den Broek (Jira)
Marnix van den Broek created SPARK-38167: Summary: CSV parsing error when using escape='"' Key: SPARK-38167 URL: https://issues.apache.org/jira/browse/SPARK-38167 Project: Spark

[jira] [Commented] (SPARK-34452) OneVsRest with GBTClassifier throws InternalCompilerException in 3.1.0

2022-02-09 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489517#comment-17489517 ] zhengruifeng commented on SPARK-34452: -- I can not reproduce this issue in 3.1.2, could you please

[jira] [Created] (SPARK-38162) Remove distinct in aggregate if its child is empty

2022-02-09 Thread XiDuo You (Jira)
XiDuo You created SPARK-38162: - Summary: Remove distinct in aggregate if its child is empty Key: SPARK-38162 URL: https://issues.apache.org/jira/browse/SPARK-38162 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38165) private classes fail at runtime in scala 2.12.13+

2022-02-09 Thread Johnny Everson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johnny Everson updated SPARK-38165: --- Description: h2. reproduction steps {code:java} git clone

[jira] [Assigned] (SPARK-38164) New SQL function: try_subtract and try_multiply

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38164: Assignee: Gengliang Wang (was: Apache Spark) > New SQL function: try_subtract and

[jira] [Assigned] (SPARK-38164) New SQL function: try_subtract and try_multiply

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38164: Assignee: Apache Spark (was: Gengliang Wang) > New SQL function: try_subtract and

[jira] [Commented] (SPARK-38164) New SQL function: try_subtract and try_multiply

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489490#comment-17489490 ] Apache Spark commented on SPARK-38164: -- User 'gengliangwang' has created a pull request for this

[jira] [Created] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
Willi Raschkowski created SPARK-38166: - Summary: Duplicates after task failure in dropDuplicates and repartition Key: SPARK-38166 URL: https://issues.apache.org/jira/browse/SPARK-38166 Project:

[jira] [Commented] (SPARK-34160) pyspark.ml.stat.Summarizer should allow sparse vector results

2022-02-09 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489518#comment-17489518 ] zhengruifeng commented on SPARK-34160: -- you can get a sparse vector by calling

[jira] [Resolved] (SPARK-34160) pyspark.ml.stat.Summarizer should allow sparse vector results

2022-02-09 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-34160. -- Resolution: Not A Problem > pyspark.ml.stat.Summarizer should allow sparse vector results >

[jira] [Commented] (SPARK-34511) Current Security vulnerabilities in spark libraries

2022-02-09 Thread Abhinav Kumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489544#comment-17489544 ] Abhinav Kumar commented on SPARK-34511: --- Updating to Spark 3.2.1 does solve most of the issues.

[jira] [Updated] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-09 Thread Marnix van den Broek (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marnix van den Broek updated SPARK-38167: - Description: hi all, When reading CSV files with Spark, I ran into a parsing

[jira] [Commented] (SPARK-31709) Proper base path for location when it is a relative path

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489528#comment-17489528 ] Apache Spark commented on SPARK-31709: -- User 'bozhang2820' has created a pull request for this

[jira] [Updated] (SPARK-36145) Remove Python 3.6 support in codebase and CI

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36145: - Labels: release-notes (was: ) > Remove Python 3.6 support in codebase and CI >

[jira] [Resolved] (SPARK-36145) Remove Python 3.6 support in codebase and CI

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36145. -- Assignee: Maciej Szymkiewicz Resolution: Done > Remove Python 3.6 support in codebase

[jira] [Commented] (SPARK-38139) ml.recommendation.ALS doctests failures

2022-02-09 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489521#comment-17489521 ] zhengruifeng commented on SPARK-38139: -- I think it is ok to adjust the tol in this case >

[jira] [Updated] (SPARK-38167) CSV parsing error when using escape='"'

2022-02-09 Thread Marnix van den Broek (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marnix van den Broek updated SPARK-38167: - Description: hi all, When reading CSV files with Spark, I ran into a parsing

[jira] [Created] (SPARK-38164) New SQL function: try_subtract and try_multiply

2022-02-09 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-38164: -- Summary: New SQL function: try_subtract and try_multiply Key: SPARK-38164 URL: https://issues.apache.org/jira/browse/SPARK-38164 Project: Spark Issue

[jira] [Created] (SPARK-38165) private classes fail at runtime in scala 2.12.13+

2022-02-09 Thread Johnny Everson (Jira)
Johnny Everson created SPARK-38165: -- Summary: private classes fail at runtime in scala 2.12.13+ Key: SPARK-38165 URL: https://issues.apache.org/jira/browse/SPARK-38165 Project: Spark Issue

[jira] [Updated] (SPARK-38165) private classes fail at runtime in scala 2.12.13+

2022-02-09 Thread Johnny Everson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johnny Everson updated SPARK-38165: --- Description: h2. reproduction steps {code:java} git clone

[jira] [Commented] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489579#comment-17489579 ] Willi Raschkowski commented on SPARK-38166: --- Linking SPARK-23207 (which is closed but looks

[jira] [Updated] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-38163: - Description: When the cause exception is `AnalysisException` at

[jira] [Assigned] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38163: Assignee: Max Gekk (was: Apache Spark) > Preserve the error class of

[jira] [Assigned] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38163: Assignee: Apache Spark (was: Max Gekk) > Preserve the error class of

[jira] [Commented] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489668#comment-17489668 ] Apache Spark commented on SPARK-38163: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489589#comment-17489589 ] Apache Spark commented on SPARK-38056: -- User 'kuwii' has created a pull request for this issue:

[jira] [Commented] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489588#comment-17489588 ] Apache Spark commented on SPARK-38056: -- User 'kuwii' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37969) Hive Serde insert should check schema before execution

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37969: --- Assignee: angerszhu > Hive Serde insert should check schema before execution >

[jira] [Resolved] (SPARK-37969) Hive Serde insert should check schema before execution

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37969. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35258

[jira] [Created] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Dooyoung Hwang (Jira)
Dooyoung Hwang created SPARK-38168: -- Summary: LikeSimplification handles escape character Key: SPARK-38168 URL: https://issues.apache.org/jira/browse/SPARK-38168 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38163) Preserve the error class of `AnalysisException` while constructing of function builder

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489670#comment-17489670 ] Apache Spark commented on SPARK-38163: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Resolved] (SPARK-37585) DSV2 InputMetrics are not getting update in corner case

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37585. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35432

[jira] [Assigned] (SPARK-37585) DSV2 InputMetrics are not getting update in corner case

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37585: --- Assignee: Sandeep Katta > DSV2 InputMetrics are not getting update in corner case >

[jira] [Commented] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489612#comment-17489612 ] Apache Spark commented on SPARK-38168: -- User 'Dooyoung-Hwang' has created a pull request for this

[jira] [Commented] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489591#comment-17489591 ] Apache Spark commented on SPARK-38056: -- User 'kuwii' has created a pull request for this issue:

[jira] [Resolved] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38061. -- Resolution: Incomplete > security scan issue with htrace-core4-4.1.0-incubating >

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489608#comment-17489608 ] Hyukjin Kwon commented on SPARK-38061: -- Again, this doesn't directly affect Spark since Spark

[jira] [Assigned] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38168: Assignee: (was: Apache Spark) > LikeSimplification handles escape character >

[jira] [Commented] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489611#comment-17489611 ] Apache Spark commented on SPARK-38168: -- User 'Dooyoung-Hwang' has created a pull request for this

[jira] [Assigned] (SPARK-38168) LikeSimplification handles escape character

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38168: Assignee: Apache Spark > LikeSimplification handles escape character >

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Sujit Biswas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489711#comment-17489711 ] Sujit Biswas commented on SPARK-38061: -- [~abhinavofficial] there are multiple vulnerabilities, as

[jira] [Updated] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-38155: - Description: Block lateral subqueries in CheckAnalysis that contains DISTINCT aggregate and

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Description: **MASTER** -

[jira] [Assigned] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38151: Assignee: (was: Apache Spark) > Flaky Test: DateTimeUtilsSuite.`daysToMicros and

[jira] [Assigned] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38151: Assignee: Apache Spark > Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

[jira] [Commented] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489731#comment-17489731 ] Apache Spark commented on SPARK-38151: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-38155: - Description: Block lateral subqueries in CheckAnalysis that contain DISTINCT aggregate and

[jira] [Updated] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-38166: -- Description: We're seeing duplicates after running the following {code:java} def

[jira] [Updated] (SPARK-38120) HiveExternalCatalog.listPartitions is failing when partition column name is upper case and dot in partition value

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38120: -- Fix Version/s: 3.1.3 > HiveExternalCatalog.listPartitions is failing when partition column

[jira] [Created] (SPARK-38169) Use OffHeap memory if configured in vectorized DeltaByteArray reader

2022-02-09 Thread Parth Chandra (Jira)
Parth Chandra created SPARK-38169: - Summary: Use OffHeap memory if configured in vectorized DeltaByteArray reader Key: SPARK-38169 URL: https://issues.apache.org/jira/browse/SPARK-38169 Project:

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Affects Version/s: 3.1.3 > Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays` >

[jira] [Commented] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489732#comment-17489732 ] Apache Spark commented on SPARK-38151: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38056: -- Fix Version/s: 3.1.3 3.2.2 > Structured streaming not working in history

[jira] [Updated] (SPARK-38056) Structured streaming not working in history server when using LevelDB

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38056: -- Target Version/s: (was: 3.1.2, 3.2.0) > Structured streaming not working in history server

[jira] [Comment Edited] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Sujit Biswas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489711#comment-17489711 ] Sujit Biswas edited comment on SPARK-38061 at 2/9/22, 5:56 PM: ---

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Description: **MASTER** -

[jira] [Updated] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Affects Version/s: 3.2.2 > Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays` >

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Stephen L. De Rudder (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489742#comment-17489742 ] Stephen L. De Rudder commented on SPARK-37814: -- Thank you for the responses, [~dongjoon]

[jira] [Commented] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489747#comment-17489747 ] Apache Spark commented on SPARK-38155: -- User 'allisonwang-db' has created a pull request for this

[jira] [Assigned] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38155: Assignee: Apache Spark > Disallow distinct aggregate in lateral subqueries with

[jira] [Assigned] (SPARK-38155) Disallow distinct aggregate in lateral subqueries with unsupported correlated predicates

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38155: Assignee: (was: Apache Spark) > Disallow distinct aggregate in lateral subqueries

[jira] [Commented] (SPARK-38061) security scan issue with htrace-core4-4.1.0-incubating

2022-02-09 Thread Sujit Biswas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489793#comment-17489793 ] Sujit Biswas commented on SPARK-38061: -- [~abhinavofficial] fyi, I was able to build the

[jira] [Updated] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38146: -- Summary: UDAF fails to aggregate TIMESTAMP_NTZ column (was: UDAF fails with unsafe row

[jira] [Assigned] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38146: Assignee: Apache Spark > UDAF fails to aggregate TIMESTAMP_NTZ column >

[jira] [Commented] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489878#comment-17489878 ] Apache Spark commented on SPARK-38146: -- User 'bersprockets' has created a pull request for this

[jira] [Assigned] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38146: Assignee: (was: Apache Spark) > UDAF fails to aggregate TIMESTAMP_NTZ column >

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489880#comment-17489880 ] Dongjoon Hyun commented on SPARK-37814: --- [~kabhwan]. Yes, it could be. Let's wait and see the way

[jira] [Commented] (SPARK-38139) ml.recommendation.ALS doctests failures

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489884#comment-17489884 ] Hyukjin Kwon commented on SPARK-38139: -- yup agree > ml.recommendation.ALS doctests failures >

[jira] [Resolved] (SPARK-38164) New SQL function: try_subtract and try_multiply

2022-02-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38164. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35461

[jira] [Assigned] (SPARK-38160) Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38160: Assignee: Apache Spark > Shuffle by rand could lead to incorrect answers when

[jira] [Commented] (SPARK-38160) Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489433#comment-17489433 ] Apache Spark commented on SPARK-38160: -- User 'WangGuangxin' has created a pull request for this

[jira] [Assigned] (SPARK-38160) Shuffle by rand could lead to incorrect answers when ShuffleFetchFailed happend

2022-02-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38160: Assignee: (was: Apache Spark) > Shuffle by rand could lead to incorrect answers when

[jira] [Updated] (SPARK-38166) Duplicates after task failure in dropDuplicates and repartition

2022-02-09 Thread Willi Raschkowski (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Willi Raschkowski updated SPARK-38166: -- Attachment: driver.log > Duplicates after task failure in dropDuplicates and

[jira] [Resolved] (SPARK-37652) Support optimize skewed join through union

2022-02-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37652. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34908

[jira] [Commented] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489857#comment-17489857 ] Jungtaek Lim commented on SPARK-37814: -- If we are open to release for maintenance for versions

[jira] [Comment Edited] (SPARK-37814) Migrating from log4j 1 to log4j 2

2022-02-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489857#comment-17489857 ] Jungtaek Lim edited comment on SPARK-37814 at 2/9/22, 11:03 PM: If we

[jira] [Updated] (SPARK-38151) Handle `Pacific/Kanton` in DateTimeUtilsSuite

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Summary: Handle `Pacific/Kanton` in DateTimeUtilsSuite (was: Flaky Test:

[jira] [Resolved] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38151. --- Fix Version/s: 3.3.0 3.2.2 3.1.3 Resolution:

[jira] [Updated] (SPARK-38151) Handle `Pacific/Kanton` in DateTimeUtilsSuite

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38151: -- Description: This issue aims to fix the flaky UT failures due to

[jira] [Assigned] (SPARK-38151) Flaky Test: DateTimeUtilsSuite.`daysToMicros and microsToDays`

2022-02-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38151: - Assignee: Dongjoon Hyun > Flaky Test: DateTimeUtilsSuite.`daysToMicros and

[jira] [Updated] (SPARK-38146) UDAF fails with unsafe row buffer containing a TIMESTAMP_NTZ column

2022-02-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38146: -- Summary: UDAF fails with unsafe row buffer containing a TIMESTAMP_NTZ column (was: UDAF

  1   2   >