[jira] [Commented] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

[ https://issues.apache.org/jira/browse/SPARK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937427#comment-16937427 ] shahid commented on SPARK-29235: I would like to analyze the issue. > CrossValidatorModel.avgMetrics

[jira] [Reopened] (SPARK-29172) Fix some exception issue of explain commands

[ https://issues.apache.org/jira/browse/SPARK-29172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta reopened SPARK-29172: This was accidentally closed because the issue number included in the title of PR #25848 was

[jira] [Assigned] (SPARK-29226) Upgrade jackson-databind to 2.9.10 and fix vulnerabilities.

[ https://issues.apache.org/jira/browse/SPARK-29226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29226: - Assignee: jiaan.geng > Upgrade jackson-databind to 2.9.10 and fix vulnerabilities. >

[jira] [Resolved] (SPARK-29226) Upgrade jackson-databind to 2.9.10 and fix vulnerabilities.

[ https://issues.apache.org/jira/browse/SPARK-29226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29226. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25912

[jira] [Commented] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937408#comment-16937408 ] Suchintak Patnaik commented on SPARK-29234: --- Is it possible to back port the PRs to version

[jira] [Commented] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937396#comment-16937396 ] Dongjoon Hyun commented on SPARK-29234: --- I'm fine with that. But, we need to back port two PRs,

[jira] [Commented] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937392#comment-16937392 ] Yuming Wang commented on SPARK-29234: - It's 3.0. [~dongjoon] Do you think we need to backport

[jira] [Resolved] (SPARK-28625) Shuffle Writer API: Indeterminate shuffle support in ShuffleMapOutputWriter

[ https://issues.apache.org/jira/browse/SPARK-28625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li resolved SPARK-28625. - Resolution: Resolved Done in SPARK-25341, which changes the writer creation API by using map

[jira] [Commented] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937378#comment-16937378 ] Suchintak Patnaik commented on SPARK-29234: --- [~yumwang] in which Spark version this is fixed?

[jira] [Updated] (SPARK-29233) Add regex expression checks for executorEnv in K8S mode

[ https://issues.apache.org/jira/browse/SPARK-29233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] merrily01 updated SPARK-29233: -- Description: In k8s mode, there are some naming regular expression requirements and checks for the

[jira] [Created] (SPARK-29236) Access 'executorDataMap' out of 'DriverEndpoint' should be protected by lock

Xianyang Liu created SPARK-29236: Summary: Access 'executorDataMap' out of 'DriverEndpoint' should be protected by lock Key: SPARK-29236 URL: https://issues.apache.org/jira/browse/SPARK-29236

[jira] [Closed] (SPARK-28915) The new keyword is not used when instantiating the WorkerOffer.

[ https://issues.apache.org/jira/browse/SPARK-28915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaqi Li closed SPARK-28915. > The new keyword is not used when instantiating the WorkerOffer. >

[jira] [Resolved] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-29234. - Resolution: Duplicate Issue resolved by pull request 24486

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937233#comment-16937233 ] Bryan Cutler commented on SPARK-28502: -- I was able to reproduce in Spark 2.4.3. The problem was

[jira] [Updated] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

[ https://issues.apache.org/jira/browse/SPARK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Bedford updated SPARK-29235: Description:   Right after a CrossValidatorModel is trained, it has avgMetrics.  After

[jira] [Commented] (SPARK-29206) Number of shuffle Netty server threads should be a multiple of number of chunk fetch handler threads

[ https://issues.apache.org/jira/browse/SPARK-29206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937219#comment-16937219 ] Min Shen commented on SPARK-29206: -- [~irashid], Would appreciate your thoughts on this ticket as well.

[jira] [Updated] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

[ https://issues.apache.org/jira/browse/SPARK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Bedford updated SPARK-29235: Affects Version/s: 2.4.3 > CrossValidatorModel.avgMetrics disappears after model is

[jira] [Updated] (SPARK-29230) Fix NullPointerException in the test class ProcfsMetricsGetterSuite

[ https://issues.apache.org/jira/browse/SPARK-29230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29230: -- Component/s: Tests > Fix NullPointerException in the test class ProcfsMetricsGetterSuite >

[jira] [Assigned] (SPARK-29230) Fix NullPointerException in the test class ProcfsMetricsGetterSuite

[ https://issues.apache.org/jira/browse/SPARK-29230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29230: - Assignee: Jiaqi Li > Fix NullPointerException in the test class

[jira] [Resolved] (SPARK-29230) Fix NullPointerException in the test class ProcfsMetricsGetterSuite

[ https://issues.apache.org/jira/browse/SPARK-29230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29230. --- Fix Version/s: 3.0.0 Resolution: Fixed This is resolved via

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937170#comment-16937170 ] Jungtaek Lim commented on SPARK-29220: -- Could we expose a way to store these unit-test.log files in

[jira] [Updated] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

[ https://issues.apache.org/jira/browse/SPARK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Bedford updated SPARK-29235: Description:   Right after a CrossValidatorModel is trained, it has avgMetrics.  After

[jira] [Updated] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

[ https://issues.apache.org/jira/browse/SPARK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Bedford updated SPARK-29235: Description:   Right after a CrossValidatorModel is trained, it has avgMetrics.  After

[jira] [Created] (SPARK-29235) CrossValidatorModel.avgMetrics disappears after model is written/read again

Matthew Bedford created SPARK-29235: --- Summary: CrossValidatorModel.avgMetrics disappears after model is written/read again Key: SPARK-29235 URL: https://issues.apache.org/jira/browse/SPARK-29235

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937155#comment-16937155 ] shane knapp commented on SPARK-29220: - it's ok... let's try and keep an eye on these tests for

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937154#comment-16937154 ] Dongjoon Hyun commented on SPARK-29220: --- Since I'm monitoring Jenkins jobs, I'll ping you if I see

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937152#comment-16937152 ] Dongjoon Hyun commented on SPARK-29220: --- Thank you for investigation, [~shaneknapp]. Yes. I merged

[jira] [Updated] (SPARK-29211) Second invocation of custom UDF results in exception (when invoked from shell)

[ https://issues.apache.org/jira/browse/SPARK-29211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29211: -- Affects Version/s: 2.3.4 > Second invocation of custom UDF results in exception (when invoked

[jira] [Commented] (SPARK-29211) Second invocation of custom UDF results in exception (when invoked from shell)

[ https://issues.apache.org/jira/browse/SPARK-29211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937149#comment-16937149 ] Dongjoon Hyun commented on SPARK-29211: --- Thank you, [~dkbiswal]. That's enough. > Second

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937126#comment-16937126 ] shane knapp commented on SPARK-29220: - hmm, i see that https://github.com/apache/spark/pull/25901

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937124#comment-16937124 ] shane knapp commented on SPARK-29220: - ARGH i just messed up and accidentally overwrote all of these

[jira] [Issue Comment Deleted] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shane knapp updated SPARK-29220: Comment: was deleted (was: for

[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]

[ https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937120#comment-16937120 ] shane knapp commented on SPARK-29220: - for

[jira] [Updated] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suchintak Patnaik updated SPARK-29234: -- Description: When we create a bucketed table as follows, it's input and output format

[jira] [Updated] (SPARK-29233) Add regex expression checks for executorEnv in K8S mode

[ https://issues.apache.org/jira/browse/SPARK-29233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] merrily01 updated SPARK-29233: -- Attachment: error.jpeg > Add regex expression checks for executorEnv in K8S mode >

[jira] [Updated] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

[ https://issues.apache.org/jira/browse/SPARK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suchintak Patnaik updated SPARK-29234: -- Description: When we create a bucketed table as follows, it's input and output format

[jira] [Updated] (SPARK-29233) Add regex expression checks for executorEnv in K8S mode

[ https://issues.apache.org/jira/browse/SPARK-29233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] merrily01 updated SPARK-29233: -- Attachment: podEnv.jpeg > Add regex expression checks for executorEnv in K8S mode >

[jira] [Created] (SPARK-29234) bucketed table created by Spark SQL DataFrame is in SequenceFile format

Suchintak Patnaik created SPARK-29234: - Summary: bucketed table created by Spark SQL DataFrame is in SequenceFile format Key: SPARK-29234 URL: https://issues.apache.org/jira/browse/SPARK-29234

[jira] [Created] (SPARK-29233) Add regex expression checks for executorEnv in K8S mode

merrily01 created SPARK-29233: - Summary: Add regex expression checks for executorEnv in K8S mode Key: SPARK-29233 URL: https://issues.apache.org/jira/browse/SPARK-29233 Project: Spark Issue

[jira] [Created] (SPARK-29232) RandomForestRegressionModel does not update the parameter maps of the DecisionTreeRegressionModels underneath

Jiaqi Guo created SPARK-29232: - Summary: RandomForestRegressionModel does not update the parameter maps of the DecisionTreeRegressionModels underneath Key: SPARK-29232 URL:

[jira] [Resolved] (SPARK-29082) Spark driver cannot start with only delegation tokens

[ https://issues.apache.org/jira/browse/SPARK-29082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29082. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25901

[jira] [Commented] (SPARK-29229) Change the additional remote repository in IsolatedClientLoader to google minor

[ https://issues.apache.org/jira/browse/SPARK-29229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937057#comment-16937057 ] Dongjoon Hyun commented on SPARK-29229: --- This is backported to `branch-2.4` via

[jira] [Updated] (SPARK-29229) Change the additional remote repository in IsolatedClientLoader to google minor

[ https://issues.apache.org/jira/browse/SPARK-29229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29229: -- Fix Version/s: 2.4.5 > Change the additional remote repository in IsolatedClientLoader to

[jira] [Updated] (SPARK-29229) Change the additional remote repository in IsolatedClientLoader to google minor

[ https://issues.apache.org/jira/browse/SPARK-29229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29229: -- Issue Type: Bug (was: Improvement) > Change the additional remote repository in

[jira] [Resolved] (SPARK-29229) Change the additional remote repository in IsolatedClientLoader to google minor

[ https://issues.apache.org/jira/browse/SPARK-29229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29229. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25915

[jira] [Assigned] (SPARK-29229) Change the additional remote repository in IsolatedClientLoader to google minor

[ https://issues.apache.org/jira/browse/SPARK-29229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29229: --- Assignee: Yuanjian Li > Change the additional remote repository in IsolatedClientLoader to

[jira] [Resolved] (SPARK-29228) Adding the ability to add headers to the KafkaRecords published to Kafka from spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-29228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randal Boyle resolved SPARK-29228. -- Resolution: Duplicate > Adding the ability to add headers to the KafkaRecords published to

[jira] [Created] (SPARK-29231) Can not infer an additional set of constraints if contains CAST

Yuming Wang created SPARK-29231: --- Summary: Can not infer an additional set of constraints if contains CAST Key: SPARK-29231 URL: https://issues.apache.org/jira/browse/SPARK-29231 Project: Spark

[jira] [Commented] (SPARK-29231) Can not infer an additional set of constraints if contains CAST

[ https://issues.apache.org/jira/browse/SPARK-29231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936978#comment-16936978 ] Yuming Wang commented on SPARK-29231: - I'm working on. > Can not infer an additional set of

[jira] [Updated] (SPARK-29230) Fix NullPointerException in the test class ProcfsMetricsGetterSuite

[ https://issues.apache.org/jira/browse/SPARK-29230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiaqi Li updated SPARK-29230: - Description: When I use `ProcfsMetricsGetterSuite` testing, always throw out 

[jira] [Created] (SPARK-29230) Fix NullPointerException in the test class ProcfsMetricsGetterSuite

Jiaqi Li created SPARK-29230: Summary: Fix NullPointerException in the test class ProcfsMetricsGetterSuite Key: SPARK-29230 URL: https://issues.apache.org/jira/browse/SPARK-29230 Project: Spark

[jira] [Resolved] (SPARK-29095) add extractInstances

[ https://issues.apache.org/jira/browse/SPARK-29095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-29095. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25802

[jira] [Assigned] (SPARK-29095) add extractInstances

[ https://issues.apache.org/jira/browse/SPARK-29095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-29095: - Assignee: zhengruifeng > add extractInstances > > > Key:

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Description: Current `DESC FORMATTED TABLE` show different table desc format, this problem cause HUE

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Description: Current `DESC FORMATTED TABLE` show different table desc format, this problem cause HUE

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Attachment: Screen Shot 2019-09-24 at 9.26.14 PM.png > Spark SQL 'DESC FORMATTED TABLE' show

[jira] [Commented] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936781#comment-16936781 ] angerszhu commented on SPARK-29225: --- [~hyukjin.kwon] Sorry for forget to add desc.  There are 4 main

[jira] [Commented] (SPARK-28990) SparkSQL invalid call to toAttribute on unresolved object, tree: *

[ https://issues.apache.org/jira/browse/SPARK-28990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936778#comment-16936778 ] fengchaoge commented on SPARK-28990: @[~726575...@qq.com] hello daile ,Can you send a link? thanks

[jira] [Resolved] (SPARK-29156) Hive has appending data as part of cdc, In write mode we should be able to write only changes captured to teradata or datasource.

[ https://issues.apache.org/jira/browse/SPARK-29156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29156. -- Resolution: Invalid no feedback > Hive has appending data as part of cdc, In write mode we

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Description: Current `DESC FORMATTED TABLE` show different table desc format, this problem cause HUE

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Description: Current `DESC FORMATTED TABLE` show different table desc format, this problem cause HUE

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Attachment: Screen Shot 2019-09-24 at 9.14.39 PM.png > Spark SQL 'DESC FORMATTED TABLE' show

[jira] [Updated] (SPARK-29174) LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source

[ https://issues.apache.org/jira/browse/SPARK-29174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29174: - Description: *using does not work for insert overwrite when in local  but works when insert

[jira] [Updated] (SPARK-29174) LOCAL is not supported in INSERT OVERWRITE DIRECTORY to data source

[ https://issues.apache.org/jira/browse/SPARK-29174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29174: - Description: *using does not work for insert overwrite when in local  but works when insert

[jira] [Resolved] (SPARK-29178) Improve shown message on sql show()

[ https://issues.apache.org/jira/browse/SPARK-29178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29178. -- Resolution: Won't Fix > Improve shown message on sql show() >

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Attachment: current saprk.jpg > Spark SQL 'DESC FORMATTED TABLE' show different format with hive >

[jira] [Updated] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29225: -- Description: Current `DESC FORMATTED TABLE` show different table desc format, this problem cause HUE

[jira] [Commented] (SPARK-29178) Improve shown message on sql show()

[ https://issues.apache.org/jira/browse/SPARK-29178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936772#comment-16936772 ] Hyukjin Kwon commented on SPARK-29178: -- In order to do that it requires to count all records

[jira] [Resolved] (SPARK-29184) I have installed pyspark and created code in notebook but when I am running it.its throwing error-Py4JJavaError: An error occurred while calling z:org.apache.spark.api.

[ https://issues.apache.org/jira/browse/SPARK-29184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29184. -- Resolution: Invalid Please ask questions to mailing list or stackoverflow. > I have

[jira] [Updated] (SPARK-29185) Add new SaveMode types for Spark SQL jdbc datasource

[ https://issues.apache.org/jira/browse/SPARK-29185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29185: - Component/s: (was: Input/Output) SQL > Add new SaveMode types for Spark

[jira] [Commented] (SPARK-29186) SubqueryAlias name value is null in Spark 2.4.3 Logical plan.

[ https://issues.apache.org/jira/browse/SPARK-29186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936770#comment-16936770 ] Hyukjin Kwon commented on SPARK-29186: -- Can you reformat the description with \{code\} ... \{code\}

[jira] [Updated] (SPARK-29186) SubqueryAlias name value is null in Spark 2.4.3 Logical plan.

[ https://issues.apache.org/jira/browse/SPARK-29186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29186: - Target Version/s: (was: 2.4.3) > SubqueryAlias name value is null in Spark 2.4.3 Logical

[jira] [Commented] (SPARK-29195) Can't config orc.compress.size option for native ORC writer

[ https://issues.apache.org/jira/browse/SPARK-29195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936767#comment-16936767 ] Hyukjin Kwon commented on SPARK-29195: -- Can you narrow down if this is a bug in ORC or Spark? >

[jira] [Commented] (SPARK-29205) Pyspark tests failed for suspected performance problem on ARM

[ https://issues.apache.org/jira/browse/SPARK-29205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936763#comment-16936763 ] Hyukjin Kwon commented on SPARK-29205: -- It seems to be flaky test, rather. and Yes, the pointer

[jira] [Resolved] (SPARK-29205) Pyspark tests failed for suspected performance problem on ARM

[ https://issues.apache.org/jira/browse/SPARK-29205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29205. -- Resolution: Duplicate Unfortunately, seems it's still flaky (although less flaky after the

[jira] [Resolved] (SPARK-29210) Spark 3.0 Migration guide should contain the details of to_utc_timestamp function and from_utc_timestamp

[ https://issues.apache.org/jira/browse/SPARK-29210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29210. -- Resolution: Won't Fix > Spark 3.0 Migration guide should contain the details of

[jira] [Commented] (SPARK-29210) Spark 3.0 Migration guide should contain the details of to_utc_timestamp function and from_utc_timestamp

[ https://issues.apache.org/jira/browse/SPARK-29210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936760#comment-16936760 ] Hyukjin Kwon commented on SPARK-29210: -- Deprecations don't have to be documented in the migration

[jira] [Resolved] (SPARK-29217) How to read streaming output path by ignoring metadata log files

[ https://issues.apache.org/jira/browse/SPARK-29217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29217. -- Resolution: Invalid Please ask questions to mailing list or stackoverflow. > How to read

[jira] [Commented] (SPARK-29222) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_parameter_convergence

[ https://issues.apache.org/jira/browse/SPARK-29222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936756#comment-16936756 ] Hyukjin Kwon commented on SPARK-29222: -- cc [~bryanc] > Flaky test: >

[jira] [Commented] (SPARK-29225) Spark SQL 'DESC FORMATTED TABLE' show different format with hive

[ https://issues.apache.org/jira/browse/SPARK-29225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936755#comment-16936755 ] Hyukjin Kwon commented on SPARK-29225: -- Can you show current & expected input/output? > Spark SQL

[jira] [Updated] (SPARK-29175) Make maven central repository in IsolatedClientLoader configurable

[ https://issues.apache.org/jira/browse/SPARK-29175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-29175: Description: We need to connect a central repository in IsolatedClientLoader for downloading Hive

[jira] [Commented] (SPARK-29166) Add parameters to limit the number of dynamic partitions for data source table

[ https://issues.apache.org/jira/browse/SPARK-29166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936732#comment-16936732 ] Ido Michael commented on SPARK-29166: - No problem [~cltlfcjin], great job! > Add parameters to

[jira] [Commented] (SPARK-29197) Remove saveModeForDSV2 in DataFrameWriter

[ https://issues.apache.org/jira/browse/SPARK-29197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936731#comment-16936731 ] Ido Michael commented on SPARK-29197: - Great job Burak! And thanks for the quick reply! > Remove

[jira] [Created] (SPARK-29229) Change the additional remote repository in IsolatedClientLoader to google minor

Yuanjian Li created SPARK-29229: --- Summary: Change the additional remote repository in IsolatedClientLoader to google minor Key: SPARK-29229 URL: https://issues.apache.org/jira/browse/SPARK-29229

[jira] [Commented] (SPARK-28558) DatasetWriter partitionBy is changing the group file permissions in 2.4 for parquets

[ https://issues.apache.org/jira/browse/SPARK-28558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936719#comment-16936719 ] Nicolas Laduguie commented on SPARK-28558: -- I debugged Spark and my understanding is that : *

[jira] [Commented] (SPARK-29228) Adding the ability to add headers to the KafkaRecords published to Kafka from spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-29228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936715#comment-16936715 ] Randal Boyle commented on SPARK-29228: -- I [~rjhsb1989] am working on this :) > Adding the ability

[jira] [Commented] (SPARK-29221) Flaky test: SQLQueryTestSuite.sql (subquery/scalar-subquery/scalar-subquery-select.sql)

[ https://issues.apache.org/jira/browse/SPARK-29221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936713#comment-16936713 ] Yuming Wang commented on SPARK-29221: -

[jira] [Created] (SPARK-29228) Adding the ability to add headers to the KafkaRecords published to Kafka from spark structured streaming

R JHS B created SPARK-29228: --- Summary: Adding the ability to add headers to the KafkaRecords published to Kafka from spark structured streaming Key: SPARK-29228 URL: https://issues.apache.org/jira/browse/SPARK-29228

[jira] [Assigned] (SPARK-28292) Enable inject user-defined Hint

[ https://issues.apache.org/jira/browse/SPARK-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28292: --- Assignee: Xiao Li > Enable inject user-defined Hint > --- > >

[jira] [Resolved] (SPARK-28292) Enable inject user-defined Hint

[ https://issues.apache.org/jira/browse/SPARK-28292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28292. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25746

[jira] [Resolved] (SPARK-28678) Specify that start index is 1-based in docstring of pyspark.sql.functions.slice

[ https://issues.apache.org/jira/browse/SPARK-28678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28678. -- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-28678) Specify that start index is 1-based in docstring of pyspark.sql.functions.slice

[ https://issues.apache.org/jira/browse/SPARK-28678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28678: Assignee: Ting Yang > Specify that start index is 1-based in docstring of >

[jira] [Commented] (SPARK-29119) DEFAULT option is not supported in Spark

[ https://issues.apache.org/jira/browse/SPARK-29119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936596#comment-16936596 ] jiaan.geng commented on SPARK-29119: This is duplicate with 

[jira] [Updated] (SPARK-29227) Track rule info in optimization phase

[ https://issues.apache.org/jira/browse/SPARK-29227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxuanguan updated SPARK-29227: Description: Track timing info for each rule in optimization phase using QueryPlanningTracker in

[jira] [Updated] (SPARK-29227) Track rule info in optimization phase

[ https://issues.apache.org/jira/browse/SPARK-29227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenxuanguan updated SPARK-29227: Summary: Track rule info in optimization phase (was: track rule info in optimization phase ) >

[jira] [Created] (SPARK-29227) track rule info in optimization phase

wenxuanguan created SPARK-29227: --- Summary: track rule info in optimization phase Key: SPARK-29227 URL: https://issues.apache.org/jira/browse/SPARK-29227 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-29226) Upgrade jackson-databind to 2.9.10 and fix vulnerabilities.

[ https://issues.apache.org/jira/browse/SPARK-29226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29226: Component/s: (was: Spark Core) Build > Upgrade jackson-databind to 2.9.10

[jira] [Updated] (SPARK-29226) Upgrade jackson-databind to 2.9.10 and fix vulnerabilities.

[ https://issues.apache.org/jira/browse/SPARK-29226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29226: Affects Version/s: (was: 2.4.5) > Upgrade jackson-databind to 2.9.10 and fix vulnerabilities.

[jira] [Commented] (SPARK-29211) Second invocation of custom UDF results in exception (when invoked from shell)

[ https://issues.apache.org/jira/browse/SPARK-29211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936538#comment-16936538 ] angerszhu commented on SPARK-29211: --- [~dkbiswal] Set a local repo follow Apache/spark and checkout

[jira] [Commented] (SPARK-29211) Second invocation of custom UDF results in exception (when invoked from shell)

[ https://issues.apache.org/jira/browse/SPARK-29211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936529#comment-16936529 ] Dilip Biswal commented on SPARK-29211: -- [~dongjoon] I tried with 2.3.4 and see the same issue. In

  1   2   >