[jira] [Commented] (SPARK-37503) Improve SparkSession/PySpark SparkSession startup

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465987#comment-17465987 ] angerszhu commented on SPARK-37503: --- [~hyukjin.kwon] Move all related pr as subtask to

[jira] [Resolved] (SPARK-37503) Improve SparkSession/PySpark SparkSession startup

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu resolved SPARK-37503. --- Fix Version/s: 3.3.0 Resolution: Fixed > Improve SparkSession/PySpark SparkSession startup >

[jira] [Updated] (SPARK-37638) Use existing active Spark session instead of SparkSession.getOrCreate in pandas API on Spark

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37638: -- Parent: SPARK-37503 Issue Type: Sub-task (was: Improvement) > Use existing active Spark sessi

[jira] [Updated] (SPARK-37503) Improve SparkSession/PySpark SparkSession startup

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37503: -- Summary: Improve SparkSession/PySpark SparkSession startup (was: Improve SparkSession startup issue)

[jira] [Updated] (SPARK-37651) Use existing active Spark session in all places of pandas API on Spark

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37651: -- Parent: SPARK-37503 Issue Type: Sub-task (was: Improvement) > Use existing active Spark sessi

[jira] [Updated] (SPARK-37727) Show ignored confs & hide warnings for conf already set in SparkSession.builder.getOrCreate

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-37727: -- Parent: SPARK-37503 Issue Type: Sub-task (was: Improvement) > Show ignored confs & hide warni

[jira] [Updated] (SPARK-37728) reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException

2021-12-27 Thread Yimin Yang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yimin Yang updated SPARK-37728: --- Affects Version/s: (was: 3.1.2) (was: 3.0.3) > reading nested columns

[jira] [Assigned] (SPARK-37763) Upgrade jackson to 2.13.1

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37763: - Assignee: William Hyun > Upgrade jackson to 2.13.1 > - > >

[jira] [Resolved] (SPARK-37763) Upgrade jackson to 2.13.1

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37763. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35037 [https://

[jira] [Commented] (SPARK-37752) Python UDF fails when it should not get evaluated

2021-12-27 Thread Ohad Raviv (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465972#comment-17465972 ] Ohad Raviv commented on SPARK-37752: That is what I deduced. thanks for the answer!

[jira] [Commented] (SPARK-37691) Support ANSI Aggregation Function: percentile_disc

2021-12-27 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465971#comment-17465971 ] jiaan.geng commented on SPARK-37691: [~subkanthi] I'm sorry! I'm working on it. > S

[jira] [Resolved] (SPARK-37715) Docker integration tests: Tweak docs and remove unneeded dependency

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37715. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34979 [https://

[jira] [Assigned] (SPARK-37715) Docker integration tests: Tweak docs and remove unneeded dependency

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37715: - Assignee: Danny Guinther > Docker integration tests: Tweak docs and remove unneeded dep

[jira] [Updated] (SPARK-37715) Remove ojdbc6 dependency and update docker-integration test docs

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37715: -- Summary: Remove ojdbc6 dependency and update docker-integration test docs (was: Docker integr

[jira] [Resolved] (SPARK-37762) Close SessonState when each hive request done

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu resolved SPARK-37762. --- Resolution: Duplicate > Close SessonState when each hive request done >

[jira] [Commented] (SPARK-37762) Close SessonState when each hive request done

2021-12-27 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465965#comment-17465965 ] angerszhu commented on SPARK-37762: --- Issue has been resolve in spark-3.2, close this

[jira] [Resolved] (SPARK-37728) reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37728. --- Fix Version/s: 3.3.0 Assignee: Yimin Yang Resolution: Fixed > reading nested

[jira] [Commented] (SPARK-37728) reading nested columns with ORC vectorized reader can cause ArrayIndexOutOfBoundsException

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465953#comment-17465953 ] Dongjoon Hyun commented on SPARK-37728: --- This is resolved via https://github.com/a

[jira] [Assigned] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37756: Assignee: Haejoon Lee > Enable matplotlib test for pandas API on Spark >

[jira] [Resolved] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37756. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35036 [https://gi

[jira] [Resolved] (SPARK-37760) Upgrade SBT to 1.6.0

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37760. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35035 [https://

[jira] [Assigned] (SPARK-37763) Upgrade jackson to 2.13.1

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37763: Assignee: (was: Apache Spark) > Upgrade jackson to 2.13.1 > -

[jira] [Assigned] (SPARK-37763) Upgrade jackson to 2.13.1

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37763: Assignee: Apache Spark > Upgrade jackson to 2.13.1 > - > >

[jira] [Commented] (SPARK-37763) Upgrade jackson to 2.13.1

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465942#comment-17465942 ] Apache Spark commented on SPARK-37763: -- User 'williamhyun' has created a pull reque

[jira] [Created] (SPARK-37763) Upgrade jackson to 2.13.1

2021-12-27 Thread William Hyun (Jira)
William Hyun created SPARK-37763: Summary: Upgrade jackson to 2.13.1 Key: SPARK-37763 URL: https://issues.apache.org/jira/browse/SPARK-37763 Project: Spark Issue Type: Bug Component

[jira] [Commented] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465933#comment-17465933 ] Apache Spark commented on SPARK-37756: -- User 'itholic' has created a pull request f

[jira] [Commented] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465932#comment-17465932 ] Apache Spark commented on SPARK-37756: -- User 'itholic' has created a pull request f

[jira] [Assigned] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37756: Assignee: (was: Apache Spark) > Enable matplotlib test for pandas API on Spark >

[jira] [Assigned] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37756: Assignee: Apache Spark > Enable matplotlib test for pandas API on Spark > ---

[jira] [Created] (SPARK-37762) Close SessonState when each hive request done

2021-12-27 Thread angerszhu (Jira)
angerszhu created SPARK-37762: - Summary: Close SessonState when each hive request done Key: SPARK-37762 URL: https://issues.apache.org/jira/browse/SPARK-37762 Project: Spark Issue Type: Improveme

[jira] [Commented] (SPARK-37761) Install matplotlib in Python 3.9 and PyPy 3 in GitHub Actions image

2021-12-27 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465929#comment-17465929 ] Haejoon Lee commented on SPARK-37761: - [~dongjoon] , could we have Python package `m

[jira] [Created] (SPARK-37761) Install matplotlib in Python 3.9 and PyPy 3 in GitHub Actions image

2021-12-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-37761: --- Summary: Install matplotlib in Python 3.9 and PyPy 3 in GitHub Actions image Key: SPARK-37761 URL: https://issues.apache.org/jira/browse/SPARK-37761 Project: Spark

[jira] [Assigned] (SPARK-37760) Upgrade SBT to 1.6.0

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37760: - Assignee: Dongjoon Hyun > Upgrade SBT to 1.6.0 > > >

[jira] [Assigned] (SPARK-37760) Upgrade SBT to 1.6.0

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37760: Assignee: Apache Spark > Upgrade SBT to 1.6.0 > > >

[jira] [Assigned] (SPARK-37760) Upgrade SBT to 1.6.0

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37760: Assignee: (was: Apache Spark) > Upgrade SBT to 1.6.0 > > >

[jira] [Commented] (SPARK-37760) Upgrade SBT to 1.6.0

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465927#comment-17465927 ] Apache Spark commented on SPARK-37760: -- User 'dongjoon-hyun' has created a pull req

[jira] [Created] (SPARK-37760) Upgrade SBT to 1.6.0

2021-12-27 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37760: - Summary: Upgrade SBT to 1.6.0 Key: SPARK-37760 URL: https://issues.apache.org/jira/browse/SPARK-37760 Project: Spark Issue Type: Sub-task Compone

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2021-12-27 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-35607: Description: There were two Arm CI jobs in AMP lab: [https://amplab.cs.berkeley.edu/jenkins/job/s

[jira] [Created] (SPARK-37759) Add ARM based self-hosted runner to Apache Spark

2021-12-27 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37759: --- Summary: Add ARM based self-hosted runner to Apache Spark Key: SPARK-37759 URL: https://issues.apache.org/jira/browse/SPARK-37759 Project: Spark Issue Type: Su

[jira] [Updated] (SPARK-37757) Enable Spark test scheduled job on ARM based self-hosted runner

2021-12-27 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37757: Summary: Enable Spark test scheduled job on ARM based self-hosted runner (was: Enable spark test

[jira] [Updated] (SPARK-37757) Enable spark test scheduled job on ARM based self-hosted runner

2021-12-27 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-37757: Summary: Enable spark test scheduled job on ARM based self-hosted runner (was: Enable scala test

[jira] [Created] (SPARK-37758) Enable PySpark scheduled job on ARM based self-hosted runner

2021-12-27 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37758: --- Summary: Enable PySpark scheduled job on ARM based self-hosted runner Key: SPARK-37758 URL: https://issues.apache.org/jira/browse/SPARK-37758 Project: Spark I

[jira] [Created] (SPARK-37757) Enable scala test on ARM based self-hosted runner

2021-12-27 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-37757: --- Summary: Enable scala test on ARM based self-hosted runner Key: SPARK-37757 URL: https://issues.apache.org/jira/browse/SPARK-37757 Project: Spark Issue Type: S

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2021-12-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35607: - Affects Version/s: 3.3.0 (was: 3.1.2) > Migrate Spark Arm Job from Je

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2021-12-27 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-35607: Description: There were two Arm CI jobs in AMP lab: [https://amplab.cs.berkeley.edu/jenkins/job/s

[jira] [Updated] (SPARK-35607) Migrate Spark Arm Job from Jenkins to GitHub Actions

2021-12-27 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-35607: Issue Type: Umbrella (was: Test) > Migrate Spark Arm Job from Jenkins to GitHub Actions > ---

[jira] [Created] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-37756: --- Summary: Enable matplotlib test for pandas API on Spark Key: SPARK-37756 URL: https://issues.apache.org/jira/browse/SPARK-37756 Project: Spark Issue Type: Impr

[jira] [Commented] (SPARK-37756) Enable matplotlib test for pandas API on Spark

2021-12-27 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465923#comment-17465923 ] Haejoon Lee commented on SPARK-37756: - I'm taking a look into this > Enable matplot

[jira] [Assigned] (SPARK-37730) plot.hist throws AttributeError on pandas=1.3.5

2021-12-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-37730: Assignee: Michał Słapek > plot.hist throws AttributeError on pandas=1.3.5 > -

[jira] [Resolved] (SPARK-37730) plot.hist throws AttributeError on pandas=1.3.5

2021-12-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-37730. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35000 [https://gi

[jira] [Resolved] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-37755. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35034 [https://

[jira] [Assigned] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-37755: - Assignee: Dongjoon Hyun > Optimize RocksDB KVStore configurations > ---

[jira] [Commented] (SPARK-37752) Python UDF fails when it should not get evaluated

2021-12-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465883#comment-17465883 ] L. C. Hsieh commented on SPARK-37752: - This is known limitation of PySpark UDF. You

[jira] [Commented] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465878#comment-17465878 ] Apache Spark commented on SPARK-37755: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37755: Assignee: (was: Apache Spark) > Optimize RocksDB KVStore configurations > ---

[jira] [Commented] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465877#comment-17465877 ] Apache Spark commented on SPARK-37755: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37755: Assignee: Apache Spark > Optimize RocksDB KVStore configurations > --

[jira] [Updated] (SPARK-37755) Optimize RocksDB KVStore configurations

2021-12-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37755: -- Summary: Optimize RocksDB KVStore configurations (was: Optimize RocksDB configurations) > Op

[jira] [Created] (SPARK-37755) Optimize RocksDB configurations

2021-12-27 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-37755: - Summary: Optimize RocksDB configurations Key: SPARK-37755 URL: https://issues.apache.org/jira/browse/SPARK-37755 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-37754) Fix black version in dev/reformat-python

2021-12-27 Thread Chia-Ping Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated SPARK-37754: --- Summary: Fix black version in dev/reformat-python (was: the version of black in reformat-py

[jira] [Assigned] (SPARK-37754) the version of black in reformat-python is inconsistent to build_and_test.yml

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37754: Assignee: Apache Spark > the version of black in reformat-python is inconsistent to build

[jira] [Assigned] (SPARK-37754) the version of black in reformat-python is inconsistent to build_and_test.yml

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37754: Assignee: (was: Apache Spark) > the version of black in reformat-python is inconsiste

[jira] [Commented] (SPARK-37754) the version of black in reformat-python is inconsistent to build_and_test.yml

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465856#comment-17465856 ] Apache Spark commented on SPARK-37754: -- User 'chia7712' has created a pull request

[jira] [Commented] (SPARK-37754) the version of black in reformat-python is inconsistent to build_and_test.yml

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465855#comment-17465855 ] Apache Spark commented on SPARK-37754: -- User 'chia7712' has created a pull request

[jira] [Created] (SPARK-37754) the version of black in reformat-python is inconsistent to build_and_test.yml

2021-12-27 Thread Chia-Ping Tsai (Jira)
Chia-Ping Tsai created SPARK-37754: -- Summary: the version of black in reformat-python is inconsistent to build_and_test.yml Key: SPARK-37754 URL: https://issues.apache.org/jira/browse/SPARK-37754 Pro

[jira] [Resolved] (SPARK-37746) log4j2-defaults.properties is not working since log4j 2 is always initialized by default

2021-12-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-37746. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35026 [https://gith

[jira] [Assigned] (SPARK-37746) log4j2-defaults.properties is not working since log4j 2 is always initialized by default

2021-12-27 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-37746: --- Assignee: Chia-Ping Tsai > log4j2-defaults.properties is not working since log4j 2 is alway

[jira] [Created] (SPARK-37753) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should apply to left outer with join many empty partitions on the left

2021-12-27 Thread Eugene Koifman (Jira)
Eugene Koifman created SPARK-37753: -- Summary: DynamicJoinSelection.shouldDemoteBroadcastHashJoin should apply to left outer with join many empty partitions on the left Key: SPARK-37753 URL: https://issues.apache.

[jira] [Commented] (SPARK-37691) Support ANSI Aggregation Function: percentile_disc

2021-12-27 Thread Kanthi Subramanian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465764#comment-17465764 ] Kanthi Subramanian commented on SPARK-37691: Is this something that I can wo

[jira] [Commented] (SPARK-37691) Support ANSI Aggregation Function: percentile_disc

2021-12-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465746#comment-17465746 ] Max Gekk commented on SPARK-37691: -- [~beliefer] Are you going to work on this soon? >

[jira] [Resolved] (SPARK-37676) Support ANSI Aggregation Function: percentile_cont

2021-12-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-37676. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34936 [https://github.com

[jira] [Assigned] (SPARK-37676) Support ANSI Aggregation Function: percentile_cont

2021-12-27 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-37676: Assignee: jiaan.geng > Support ANSI Aggregation Function: percentile_cont > -

[jira] [Assigned] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37738: Assignee: Apache Spark > PySpark date_add only accepts an integer as it's second paramete

[jira] [Assigned] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37738: Assignee: (was: Apache Spark) > PySpark date_add only accepts an integer as it's seco

[jira] [Commented] (SPARK-37738) PySpark date_add only accepts an integer as it's second parameter

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465733#comment-17465733 ] Apache Spark commented on SPARK-37738: -- User 'Daniel-Davies' has created a pull req

[jira] [Assigned] (SPARK-35703) Relax constraint for Spark bucket join and remove HashClusteredDistribution

2021-12-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35703: --- Assignee: Chao Sun > Relax constraint for Spark bucket join and remove HashClusteredDistrib

[jira] [Resolved] (SPARK-35703) Relax constraint for Spark bucket join and remove HashClusteredDistribution

2021-12-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35703. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 32875 [https://gith

[jira] [Updated] (SPARK-37751) Apache Commons Crypto doesn't support Java 11

2021-12-27 Thread Shipeng Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shipeng Feng updated SPARK-37751: - Description: For kubernetes, we are using Java 11 in docker, [https://github.com/apache/spark/b

[jira] [Created] (SPARK-37752) Python UDF fails when it should not get evaluated

2021-12-27 Thread Ohad Raviv (Jira)
Ohad Raviv created SPARK-37752: -- Summary: Python UDF fails when it should not get evaluated Key: SPARK-37752 URL: https://issues.apache.org/jira/browse/SPARK-37752 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-37751) Apache Commons Crypto doesn't support Java 11

2021-12-27 Thread Shipeng Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shipeng Feng updated SPARK-37751: - Summary: Apache Commons Crypto doesn't support Java 11 (was: Apache Commons Crypto Doesn't supp

[jira] [Created] (SPARK-37751) Apache Commons Crypto Doesn't support Java 11

2021-12-27 Thread Shipeng Feng (Jira)
Shipeng Feng created SPARK-37751: Summary: Apache Commons Crypto Doesn't support Java 11 Key: SPARK-37751 URL: https://issues.apache.org/jira/browse/SPARK-37751 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-37732) Improve the implement of JDBCV2Suite

2021-12-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37732. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35010 [https://gith

[jira] [Assigned] (SPARK-37732) Improve the implement of JDBCV2Suite

2021-12-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37732: --- Assignee: jiaan.geng > Improve the implement of JDBCV2Suite > -

[jira] [Resolved] (SPARK-37716) Allow LateralJoin node to host non-deterministic expressions when the outer query is a single row relation

2021-12-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-37716. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34987 [https://gith

[jira] [Assigned] (SPARK-37716) Allow LateralJoin node to host non-deterministic expressions when the outer query is a single row relation

2021-12-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-37716: --- Assignee: Allison Wang > Allow LateralJoin node to host non-deterministic expressions when

[jira] [Commented] (SPARK-37750) ANSI mode: optionally return null result if element not exists in array/map

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465576#comment-17465576 ] Apache Spark commented on SPARK-37750: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-37750) ANSI mode: optionally return null result if element not exists in array/map

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37750: Assignee: Apache Spark (was: Gengliang Wang) > ANSI mode: optionally return null result

[jira] [Assigned] (SPARK-37750) ANSI mode: optionally return null result if element not exists in array/map

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37750: Assignee: Gengliang Wang (was: Apache Spark) > ANSI mode: optionally return null result

[jira] [Commented] (SPARK-37750) ANSI mode: optionally return null result if element not exists in array/map

2021-12-27 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17465575#comment-17465575 ] Apache Spark commented on SPARK-37750: -- User 'gengliangwang' has created a pull req

[jira] [Updated] (SPARK-37750) ANSI mode: optionally return null result if element not exists in array/map

2021-12-27 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-37750: --- Summary: ANSI mode: optionally return null result if element not exists in array/map (was:

[jira] [Created] (SPARK-37750) ANSI mode: Add a config to optionally return null result if element not exists in array/map

2021-12-27 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-37750: -- Summary: ANSI mode: Add a config to optionally return null result if element not exists in array/map Key: SPARK-37750 URL: https://issues.apache.org/jira/browse/SPARK-37750