[jira] [Commented] (SPARK-20628) Keep track of nodes which are going to be shut down & avoid scheduling new tasks

2020-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139132#comment-17139132 ] Hyukjin Kwon commented on SPARK-20628: -- [~holden] Looks like it had to be a SPIP. D

[jira] [Commented] (SPARK-20624) Add better handling for node shutdown

2020-06-17 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17139133#comment-17139133 ] Hyukjin Kwon commented on SPARK-20624: -- [~holden] Looks like it had to be a SPIP. D

[jira] [Resolved] (SPARK-27280) infer filters from Join's OR condition

2020-06-17 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-27280. - Resolution: Duplicate > infer filters from Join's OR condition > ---

[jira] [Resolved] (SPARK-31960) Only populate Hadoop classpath for no-hadoop build

2020-06-17 Thread DB Tsai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai resolved SPARK-31960. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28788 [https://github.com/a

[jira] [Updated] (SPARK-32016) Why spark does not preserve the original timestamp format while writing dataset to file or hdfs

2020-06-17 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-32016: Fix Version/s: (was: 3.0.0) > Why spark does not preserve the original timestamp format while writing

[jira] [Updated] (SPARK-32016) Why spark does not preserve the original timestamp format while writing dataset to file or hdfs

2020-06-17 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-32016: Target Version/s: (was: 3.0.0) > Why spark does not preserve the original timestamp format while writing

[jira] [Assigned] (SPARK-32020) Refactor the logic to compute SPARK_HOME into a common place.

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32020: Assignee: Apache Spark > Refactor the logic to compute SPARK_HOME into a common place. >

[jira] [Commented] (SPARK-32020) Refactor the logic to compute SPARK_HOME into a common place.

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138965#comment-17138965 ] Apache Spark commented on SPARK-32020: -- User 'dilipbiswal' has created a pull reque

[jira] [Assigned] (SPARK-32020) Refactor the logic to compute SPARK_HOME into a common place.

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32020: Assignee: (was: Apache Spark) > Refactor the logic to compute SPARK_HOME into a commo

[jira] [Created] (SPARK-32020) Refactor the logic to compute SPARK_HOME into a common place.

2020-06-17 Thread Dilip Biswal (Jira)
Dilip Biswal created SPARK-32020: Summary: Refactor the logic to compute SPARK_HOME into a common place. Key: SPARK-32020 URL: https://issues.apache.org/jira/browse/SPARK-32020 Project: Spark

[jira] [Assigned] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32019: Assignee: (was: Apache Spark) > Add spark.sql.files.minPartitionNum config >

[jira] [Commented] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138954#comment-17138954 ] Apache Spark commented on SPARK-32019: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138953#comment-17138953 ] Apache Spark commented on SPARK-32019: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32019: Assignee: Apache Spark > Add spark.sql.files.minPartitionNum config > ---

[jira] [Created] (SPARK-32019) Add spark.sql.files.minPartitionNum config

2020-06-17 Thread ulysses you (Jira)
ulysses you created SPARK-32019: --- Summary: Add spark.sql.files.minPartitionNum config Key: SPARK-32019 URL: https://issues.apache.org/jira/browse/SPARK-32019 Project: Spark Issue Type: Improvem

[jira] [Created] (SPARK-32018) Fix UnsafeRow set overflowed decimal

2020-06-17 Thread Allison Wang (Jira)
Allison Wang created SPARK-32018: Summary: Fix UnsafeRow set overflowed decimal Key: SPARK-32018 URL: https://issues.apache.org/jira/browse/SPARK-32018 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2020-06-17 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138879#comment-17138879 ] Sean R. Owen commented on SPARK-26183: -- Yeah that's probably imprecise. From the us

[jira] [Created] (SPARK-32017) Make Pyspark Hadoop 3.2+ Variant available in PyPI

2020-06-17 Thread George Pongracz (Jira)
George Pongracz created SPARK-32017: --- Summary: Make Pyspark Hadoop 3.2+ Variant available in PyPI Key: SPARK-32017 URL: https://issues.apache.org/jira/browse/SPARK-32017 Project: Spark Issu

[jira] [Commented] (SPARK-30616) Introduce TTL config option for SQL Metadata Cache

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138841#comment-17138841 ] Apache Spark commented on SPARK-30616: -- User 'sap1ens' has created a pull request f

[jira] [Assigned] (SPARK-30616) Introduce TTL config option for SQL Metadata Cache

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30616: Assignee: (was: Apache Spark) > Introduce TTL config option for SQL Metadata Cache >

[jira] [Updated] (SPARK-30616) Introduce TTL config option for SQL Metadata Cache

2020-06-17 Thread Yaroslav Tkachenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yaroslav Tkachenko updated SPARK-30616: --- Description: >From >[documentation|https://spark.apache.org/docs/2.4.4/sql-data-sou

[jira] [Assigned] (SPARK-30616) Introduce TTL config option for SQL Metadata Cache

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-30616: Assignee: Apache Spark > Introduce TTL config option for SQL Metadata Cache > ---

[jira] [Commented] (SPARK-30616) Introduce TTL config option for SQL Metadata Cache

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138840#comment-17138840 ] Apache Spark commented on SPARK-30616: -- User 'sap1ens' has created a pull request f

[jira] [Updated] (SPARK-30616) Introduce TTL config option for SQL Metadata Cache

2020-06-17 Thread Yaroslav Tkachenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yaroslav Tkachenko updated SPARK-30616: --- Summary: Introduce TTL config option for SQL Metadata Cache (was: Introduce TTL con

[jira] [Commented] (SPARK-26183) ConcurrentModificationException when using Spark collectionAccumulator

2020-06-17 Thread Brandon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138810#comment-17138810 ] Brandon commented on SPARK-26183: - Hi [~srowen], I have also seen this issue with the sa

[jira] [Created] (SPARK-32016) Why spark does not preserve the original timestamp format while writing dataset to file or hdfs

2020-06-17 Thread Anupam Jain (Jira)
Anupam Jain created SPARK-32016: --- Summary: Why spark does not preserve the original timestamp format while writing dataset to file or hdfs Key: SPARK-32016 URL: https://issues.apache.org/jira/browse/SPARK-32016

[jira] [Resolved] (SPARK-31871) Display the canvas element icon for sorting column

2020-06-17 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-31871. Resolution: Fixed This issue is resolved in [https://github.com/apache/spark/pull/28799] .

[jira] [Assigned] (SPARK-31871) Display the canvas element icon for sorting column

2020-06-17 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta reassigned SPARK-31871: -- Assignee: liucht-inspur > Display the canvas element icon for sorting column > --

[jira] [Updated] (SPARK-31871) Display the canvas element icon for sorting column

2020-06-17 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-31871: --- Affects Version/s: 2.4.6 > Display the canvas element icon for sorting column >

[jira] [Updated] (SPARK-31871) Display the canvas element icon for sorting column

2020-06-17 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-31871: --- Fix Version/s: 2.4.7 > Display the canvas element icon for sorting column >

[jira] [Updated] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32000: -- Priority: Minor (was: Major) > Fix the flaky testcase for partially launched task in barrier-

[jira] [Resolved] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32000. --- Fix Version/s: 2.4.7 3.1.0 3.0.1 Assignee: wuy

[jira] [Commented] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138471#comment-17138471 ] Apache Spark commented on SPARK-32000: -- User 'Ngone51' has created a pull request f

[jira] [Updated] (SPARK-32007) Spark Driver Supervise does not work reliably

2020-06-17 Thread Suraj Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suraj Sharma updated SPARK-32007: - Issue Type: Bug (was: Question) > Spark Driver Supervise does not work reliably > -

[jira] [Commented] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138376#comment-17138376 ] Apache Spark commented on SPARK-32015: -- User 'wankunde' has created a pull request

[jira] [Assigned] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32015: Assignee: Apache Spark > Remote inheritable thread local variables after spark context is

[jira] [Assigned] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32015: Assignee: (was: Apache Spark) > Remote inheritable thread local variables after spark

[jira] [Commented] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138375#comment-17138375 ] Apache Spark commented on SPARK-32015: -- User 'wankunde' has created a pull request

[jira] [Updated] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-32015: Attachment: (was: screenshot-1.png) > Remote inheritable thread local variables after spark context is

[jira] [Updated] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-32015: Description: There is still a reference(eg. `java.lang.ApplicationShutdownHooks` -> `org.apache.hive.comm

[jira] [Created] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Wan Kun (Jira)
Wan Kun created SPARK-32015: --- Summary: Remote inheritable thread local variables after spark context is stopped Key: SPARK-32015 URL: https://issues.apache.org/jira/browse/SPARK-32015 Project: Spark

[jira] [Updated] (SPARK-32015) Remote inheritable thread local variables after spark context is stopped

2020-06-17 Thread Wan Kun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wan Kun updated SPARK-32015: Attachment: screenshot-1.png > Remote inheritable thread local variables after spark context is stopped >

[jira] [Commented] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138294#comment-17138294 ] Apache Spark commented on SPARK-32000: -- User 'Ngone51' has created a pull request f

[jira] [Commented] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138293#comment-17138293 ] Apache Spark commented on SPARK-32000: -- User 'Ngone51' has created a pull request f

[jira] [Assigned] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta reassigned SPARK-32000: -- Assignee: (was: Kousuke Saruta) > Fix the flaky testcase for partially launched t

[jira] [Updated] (SPARK-32000) Fix the flaky testcase for partially launched task in barrier-mode.

2020-06-17 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-32000: --- Description: I noticed sometimes the testcase for SPARK-31485 fails. (was: I noticed someti

[jira] [Updated] (SPARK-32008) 3.0.0 release build fails

2020-06-17 Thread Philipp Dallig (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philipp Dallig updated SPARK-32008: --- Description: Hi, I try to build the spark release 3.0.0 by myself. I got the following erro

[jira] [Closed] (SPARK-32008) 3.0.0 release build fails

2020-06-17 Thread Philipp Dallig (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philipp Dallig closed SPARK-32008. -- > 3.0.0 release build fails > - > > Key: SPARK-32008 >

[jira] [Resolved] (SPARK-32008) 3.0.0 release build fails

2020-06-17 Thread Philipp Dallig (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philipp Dallig resolved SPARK-32008. Fix Version/s: 3.0.0 Resolution: Fixed > 3.0.0 release build fails > --

[jira] [Commented] (SPARK-32008) 3.0.0 release build fails

2020-06-17 Thread Philipp Dallig (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138249#comment-17138249 ] Philipp Dallig commented on SPARK-32008: Hi [~shivaram] Thanks for your fast re

[jira] [Updated] (SPARK-32013) Support query execution before/after reading/writing over JDBC

2020-06-17 Thread Noritaka Sekiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-32013: -- Description: For ETL workload, there is a common requirement to perform SQL statement

[jira] [Created] (SPARK-32014) Support calling stored procedure on JDBC data source

2020-06-17 Thread Yoshi Matsuzaki (Jira)
Yoshi Matsuzaki created SPARK-32014: --- Summary: Support calling stored procedure on JDBC data source Key: SPARK-32014 URL: https://issues.apache.org/jira/browse/SPARK-32014 Project: Spark Is

[jira] [Updated] (SPARK-32013) Support query execution before/after reading/writing over JDBC

2020-06-17 Thread Noritaka Sekiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-32013: -- Description: For ETL workload, there is a common requirement to perform SQL statement

[jira] [Updated] (SPARK-32013) Support query execution before/after reading/writing over JDBC

2020-06-17 Thread Noritaka Sekiyama (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noritaka Sekiyama updated SPARK-32013: -- Description: For ETL workload, there is a common requirement to perform SQL statement

[jira] [Created] (SPARK-32013) Support query execution before/after reading/writing over JDBC

2020-06-17 Thread Noritaka Sekiyama (Jira)
Noritaka Sekiyama created SPARK-32013: - Summary: Support query execution before/after reading/writing over JDBC Key: SPARK-32013 URL: https://issues.apache.org/jira/browse/SPARK-32013 Project: Spa