[jira] [Commented] (SPARK-32264) More resources in Github Actions

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170571#comment-17170571 ] Hyukjin Kwon commented on SPARK-32264: -- Looks we can't allocate the locations for one specific repo

[jira] [Resolved] (SPARK-32264) More resources in Github Actions

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32264. -- Resolution: Invalid > More resources in Github Actions > > >

[jira] [Resolved] (SPARK-32520) Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32520. -- Fix Version/s: 3.1.0 Assignee: Jungtaek Lim Resolution: Fixed This was fixed

[jira] [Commented] (SPARK-32264) More resources in Github Actions

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170040#comment-17170040 ] Hyukjin Kwon commented on SPARK-32264: -- Okay .. actually I contacted GitHub team. I brought one

[jira] [Commented] (SPARK-32520) Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170031#comment-17170031 ] Hyukjin Kwon commented on SPARK-32520: -- cc [~kabhwan] and [~gsomogyi] FYI. Would you guys mind

[jira] [Updated] (SPARK-32520) Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32520: - Description: {{KafkaSourceStressSuite.stress test with multiple topics and partitions}} seems

[jira] [Created] (SPARK-32520) Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions

2020-08-03 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32520: Summary: Flaky Test: KafkaSourceStressSuite.stress test with multiple topics and partitions Key: SPARK-32520 URL: https://issues.apache.org/jira/browse/SPARK-32520

[jira] [Resolved] (SPARK-32513) Rename classes/files with the Jdbc prefix to JDBC

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32513. -- Resolution: Won't Fix > Rename classes/files with the Jdbc prefix to JDBC >

[jira] [Resolved] (SPARK-32253) Make readability better in the test result logs

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32253. -- Resolution: Duplicate This was reverted in https://github.com/apache/spark/pull/29219. It

[jira] [Reopened] (SPARK-32253) Make readability better in the test result logs

2020-08-03 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-32253: -- > Make readability better in the test result logs >

[jira] [Updated] (SPARK-32507) Main Page

2020-07-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32507: - Description: We should make a main package to overview PySpark properly. See the demo example:

[jira] [Updated] (SPARK-32507) Main Page

2020-07-31 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32507: - Description: We should make a main package to overview PySpark properly. See the demo example:

[jira] [Created] (SPARK-32507) Main Page

2020-07-31 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32507: Summary: Main Page Key: SPARK-32507 URL: https://issues.apache.org/jira/browse/SPARK-32507 Project: Spark Issue Type: Sub-task Components:

[jira] [Resolved] (SPARK-32497) Installs qpdf package for CRAN check in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32497. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29306

[jira] [Assigned] (SPARK-32497) Installs qpdf package for CRAN check in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32497: Assignee: Hyukjin Kwon > Installs qpdf package for CRAN check in GitHub Actions >

[jira] [Resolved] (SPARK-32227) Bug in load-spark-env.cmd with Spark 3.0.0

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32227. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Fixed in

[jira] [Assigned] (SPARK-32227) Bug in load-spark-env.cmd with Spark 3.0.0

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32227: Assignee: Ihor Bobak > Bug in load-spark-env.cmd with Spark 3.0.0 >

[jira] [Resolved] (SPARK-32496) Include GitHub Action file as the changes in testing

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32496. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29305

[jira] [Assigned] (SPARK-32496) Include GitHub Action file as the changes in testing

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32496: Assignee: Hyukjin Kwon > Include GitHub Action file as the changes in testing >

[jira] [Updated] (SPARK-32497) Installs qpdf package for CRAN check in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32497: - Parent: SPARK-32244 Issue Type: Sub-task (was: Bug) > Installs qpdf package for CRAN

[jira] [Created] (SPARK-32497) Installs qpdf package for CRAN check in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32497: Summary: Installs qpdf package for CRAN check in GitHub Actions Key: SPARK-32497 URL: https://issues.apache.org/jira/browse/SPARK-32497 Project: Spark Issue

[jira] [Updated] (SPARK-32497) Installs qpdf package for CRAN check in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32497: - Affects Version/s: (was: 3.0.0) 3.1.0 > Installs qpdf package for

[jira] [Resolved] (SPARK-32493) Manually install R instead of using setup-r in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32493. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29302

[jira] [Assigned] (SPARK-32493) Manually install R instead of using setup-r in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32493: Assignee: Hyukjin Kwon > Manually install R instead of using setup-r in GitHub Actions >

[jira] [Created] (SPARK-32496) Include GitHub Action file as the changes in testing

2020-07-30 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32496: Summary: Include GitHub Action file as the changes in testing Key: SPARK-32496 URL: https://issues.apache.org/jira/browse/SPARK-32496 Project: Spark Issue

[jira] [Resolved] (SPARK-32491) Do not install SparkR in test-only mode in testing script

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32491. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29300

[jira] [Assigned] (SPARK-32491) Do not install SparkR in test-only mode in testing script

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32491: Assignee: Hyukjin Kwon > Do not install SparkR in test-only mode in testing script >

[jira] [Created] (SPARK-32493) Manually install R instead of using setup-r in GitHub Actions

2020-07-30 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32493: Summary: Manually install R instead of using setup-r in GitHub Actions Key: SPARK-32493 URL: https://issues.apache.org/jira/browse/SPARK-32493 Project: Spark

[jira] [Commented] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167736#comment-17167736 ] Hyukjin Kwon commented on SPARK-32475: -- Looks like you're Java paths are mixed up. It happens when

[jira] [Commented] (SPARK-32475) java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167737#comment-17167737 ] Hyukjin Kwon commented on SPARK-32475: -- Oh right. > java.lang.NoSuchMethodError:

[jira] [Commented] (SPARK-32483) spark-shell: error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))]

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167735#comment-17167735 ] Hyukjin Kwon commented on SPARK-32483: -- Looks [~JinxinTang]'s way is working. I am resolving this.

[jira] [Resolved] (SPARK-32483) spark-shell: error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))]

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32483. -- Resolution: Invalid > spark-shell: error: value topByKey is not a member of >

[jira] [Comment Edited] (SPARK-32483) spark-shell: error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))]

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167734#comment-17167734 ] Hyukjin Kwon edited comment on SPARK-32483 at 7/30/20, 8:00 AM: Please

[jira] [Commented] (SPARK-32483) spark-shell: error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))]

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167734#comment-17167734 ] Hyukjin Kwon commented on SPARK-32483: -- Please avoid setting the Priority to Critical+ which is

[jira] [Updated] (SPARK-32483) spark-shell: error: value topByKey is not a member of org.apache.spark.rdd.RDD[(String, (String, Double))]

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32483: - Priority: Major (was: Critical) > spark-shell: error: value topByKey is not a member of >

[jira] [Updated] (SPARK-32491) Do not install SparkR in test-only mode in testing script

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32491: - Parent: SPARK-32244 Issue Type: Sub-task (was: Improvement) > Do not install SparkR in

[jira] [Created] (SPARK-32491) Do not install SparkR in test-only mode in testing script

2020-07-30 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32491: Summary: Do not install SparkR in test-only mode in testing script Key: SPARK-32491 URL: https://issues.apache.org/jira/browse/SPARK-32491 Project: Spark

[jira] [Resolved] (SPARK-32478) Error message to show the schema mismatch in gapply with Arrow vectorization

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32478. -- Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32478) Error message to show the schema mismatch in gapply with Arrow vectorization

2020-07-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32478: Assignee: Hyukjin Kwon > Error message to show the schema mismatch in gapply with Arrow

[jira] [Resolved] (SPARK-32010) Thread leaks in pinned thread mode

2020-07-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32010. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28968

[jira] [Commented] (SPARK-30255) Support explain mode in SparkR df.explain

2020-07-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167320#comment-17167320 ] Hyukjin Kwon commented on SPARK-30255: -- Please go ahead and directly open a PR. We don't usually

[jira] [Resolved] (SPARK-32355) 使用Structured Streaming窗口统计不能实现topN

2020-07-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32355. -- Resolution: Incomplete > 使用Structured Streaming窗口统计不能实现topN >

[jira] [Created] (SPARK-32478) Error message to show the schema mismatch in gapply with Arrow vectorization

2020-07-29 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32478: Summary: Error message to show the schema mismatch in gapply with Arrow vectorization Key: SPARK-32478 URL: https://issues.apache.org/jira/browse/SPARK-32478

[jira] [Updated] (SPARK-32478) Error message to show the schema mismatch in gapply with Arrow vectorization

2020-07-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32478: - Description: Currently, the error message is confusing when the output schema type is not

[jira] [Commented] (SPARK-30817) SparkR ML algorithms parity

2020-07-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166940#comment-17166940 ] Hyukjin Kwon commented on SPARK-30817: -- [~zero323] I will leave this JIRA resolved but [~dan_z]

[jira] [Resolved] (SPARK-30817) SparkR ML algorithms parity

2020-07-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30817. -- Resolution: Done > SparkR ML algorithms parity > > >

[jira] [Assigned] (SPARK-32471) Describe JSON option `allowNonNumericNumbers`

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32471: Assignee: Maxim Gekk > Describe JSON option `allowNonNumericNumbers` >

[jira] [Resolved] (SPARK-32471) Describe JSON option `allowNonNumericNumbers`

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32471. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29275

[jira] [Reopened] (SPARK-31525) Inconsistent result of df.head(1) and df.head()

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-31525: -- Assignee: (was: Tianshi Zhu) Sorry I was confused. Let's keep it consistent with Scala

[jira] [Updated] (SPARK-31525) Inconsistent result of df.head(1) and df.head()

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31525: - Fix Version/s: (was: 3.1.0) > Inconsistent result of df.head(1) and df.head() >

[jira] [Assigned] (SPARK-31267) Flaky test: WholeStageCodegenSparkSubmitSuite.Generated code on driver should not embed platform-specific constant

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31267: Assignee: Tianshi Zhu > Flaky test: WholeStageCodegenSparkSubmitSuite.Generated code on

[jira] [Assigned] (SPARK-31525) Inconsistent result of df.head(1) and df.head()

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31525: Assignee: Tianshi Zhu > Inconsistent result of df.head(1) and df.head() >

[jira] [Commented] (SPARK-32101) The name in the with clause when it is same as table name. And when that table name is used in the other places, it is not taking the table, it is considering the with

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166256#comment-17166256 ] Hyukjin Kwon commented on SPARK-32101: -- Can you try it in higher version? Spark 2.2 and 2.3 are

[jira] [Resolved] (SPARK-32101) The name in the with clause when it is same as table name. And when that table name is used in the other places, it is not taking the table, it is considering the with

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32101. -- Resolution: Incomplete > The name in the with clause when it is same as table name. And when

[jira] [Resolved] (SPARK-32114) Change name of the slaves file, to something more acceptable

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32114. -- Resolution: Duplicate > Change name of the slaves file, to something more acceptable >

[jira] [Resolved] (SPARK-32116) Python RDD containing a 'pyarrow record_batch object' to java RDD conversion issue

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32116. -- Resolution: Invalid What you are using are internal APIs in Spark. It's not supposed to be

[jira] [Resolved] (SPARK-32122) Exception while writing dataframe with enum fields

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32122. -- Resolution: Cannot Reproduce Let's resolve it as cannot reproduce since it can't in 3.0. We

[jira] [Commented] (SPARK-32137) AttributeError: Can only use .dt accessor with datetimelike values

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166247#comment-17166247 ] Hyukjin Kwon commented on SPARK-32137: -- Can you share the codes to reproduce? > AttributeError:

[jira] [Resolved] (SPARK-32158) Add JSONOptions to toJSON

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32158. -- Resolution: Won't Fix > Add JSONOptions to toJSON > - > >

[jira] [Updated] (SPARK-32158) Add JSONOptions to toJSON

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32158: - Fix Version/s: (was: 3.0.1) (was: 3.1.0) > Add JSONOptions to toJSON

[jira] [Updated] (SPARK-32137) AttributeError: Can only use .dt accessor with datetimelike values

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32137: - Priority: Major (was: Critical) > AttributeError: Can only use .dt accessor with datetimelike

[jira] [Resolved] (SPARK-32176) Automatic type promotion to ArrayType in defined schema in from_json is broken

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32176. -- Resolution: Cannot Reproduce > Automatic type promotion to ArrayType in defined schema in

[jira] [Reopened] (SPARK-32369) pyspark foreach/foreachPartition send http request failed

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-32369: -- > pyspark foreach/foreachPartition send http request failed >

[jira] [Resolved] (SPARK-32369) pyspark foreach/foreachPartition send http request failed

2020-07-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32369. -- Resolution: Workaround > pyspark foreach/foreachPartition send http request failed >

[jira] [Commented] (SPARK-32208) SparkSQL throw Illegal character exception when load certain abnormal path of HDFS

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166147#comment-17166147 ] Hyukjin Kwon commented on SPARK-32208: -- Can you fill the PR description please? > SparkSQL throw

[jira] [Resolved] (SPARK-32213) saveAsTable deletes all files in path

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32213. -- Resolution: Not A Problem > saveAsTable deletes all files in path >

[jira] [Commented] (SPARK-32213) saveAsTable deletes all files in path

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166145#comment-17166145 ] Hyukjin Kwon commented on SPARK-32213: -- I think this is documented: {code} * In this method,

[jira] [Resolved] (SPARK-32261) PySpark regexp_replace not replacing JSON çlçlçl

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32261. -- Resolution: Invalid > PySpark regexp_replace not replacing JSON çlçlçl >

[jira] [Resolved] (SPARK-32263) PySpark regexp_replace not replacing JSON çlçlçlçl

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32263. -- Resolution: Invalid > PySpark regexp_replace not replacing JSON çlçlçlçl >

[jira] [Resolved] (SPARK-32260) PySpark regexp_replace not replacing JSON çlçl

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32260. -- Resolution: Invalid > PySpark regexp_replace not replacing JSON çlçl >

[jira] [Resolved] (SPARK-32262) PySpark regexp_replace not replacing JSON çlçlçl

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32262. -- Resolution: Invalid > PySpark regexp_replace not replacing JSON çlçlçl >

[jira] [Commented] (SPARK-32269) Failed to rename delta file on checkpoint path

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166135#comment-17166135 ] Hyukjin Kwon commented on SPARK-32269: -- [~ElvisQaQ] can you check the image? Seems that's broken.

[jira] [Commented] (SPARK-32275) "None.org.apache.spark.api.java.JavaSparkContext" Issue With Spark-Mllib Algorithm and JDBC Connectors

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166132#comment-17166132 ] Hyukjin Kwon commented on SPARK-32275: -- Looks like it tries to access to JVM instances within UDFs

[jira] [Issue Comment Deleted] (SPARK-32323) Javascript/HTML bug in spark application UI

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32323: - Comment: was deleted (was: [~ibobak] can you double check if the image is uploaded or not?) >

[jira] [Commented] (SPARK-32323) Javascript/HTML bug in spark application UI

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166130#comment-17166130 ] Hyukjin Kwon commented on SPARK-32323: -- [~ibobak] can you double check if the image is uploaded or

[jira] [Commented] (SPARK-32341) add mutiple filter in rdd function

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166129#comment-17166129 ] Hyukjin Kwon commented on SPARK-32341: -- You can do it via DataFrame and SQL APIs. I think such

[jira] [Resolved] (SPARK-32341) add mutiple filter in rdd function

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32341. -- Resolution: Won't Do > add mutiple filter in rdd function >

[jira] [Resolved] (SPARK-31525) Inconsistent result of df.head(1) and df.head()

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31525. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29214

[jira] [Commented] (SPARK-32312) Upgrade Apache Arrow to 1.0.0

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166117#comment-17166117 ] Hyukjin Kwon commented on SPARK-32312: -- Nice! > Upgrade Apache Arrow to 1.0.0 >

[jira] [Commented] (SPARK-32385) Publish a "bill of materials" (BOM) descriptor for Spark with correct versions of various dependencies

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166115#comment-17166115 ] Hyukjin Kwon commented on SPARK-32385: -- Do you mean something like this?

[jira] [Commented] (SPARK-32423) class 'DataFrame' returns instance of type(self) instead of DataFrame

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166114#comment-17166114 ] Hyukjin Kwon commented on SPARK-32423: -- Can you show some pseudo codes? It's a bit difficult to

[jira] [Resolved] (SPARK-32460) how spark collects non-match results after performing broadcast left outer join

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32460. -- Resolution: Invalid > how spark collects non-match results after performing broadcast left

[jira] [Commented] (SPARK-32433) Spark Web UI shows Nan undefined in Shuffle Read Size / Records

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166109#comment-17166109 ] Hyukjin Kwon commented on SPARK-32433: -- Seems the attached image is broken. Can you check

[jira] [Commented] (SPARK-32460) how spark collects non-match results after performing broadcast left outer join

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166110#comment-17166110 ] Hyukjin Kwon commented on SPARK-32460: -- Let's ask questions into mailing list or stackoverflow

[jira] [Commented] (SPARK-32400) Test coverage of HiveScripTransformationExec

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166108#comment-17166108 ] Hyukjin Kwon commented on SPARK-32400: -- Please fill JIRA description. > Test coverage of

[jira] [Commented] (SPARK-32388) TRANSFORM when schema less should keep same with hive

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166106#comment-17166106 ] Hyukjin Kwon commented on SPARK-32388: -- Please fill JIRA description. > TRANSFORM when schema less

[jira] [Commented] (SPARK-32390) TRANSFORM with hive serde support CalendarIntervalType and UserDefinedType

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166107#comment-17166107 ] Hyukjin Kwon commented on SPARK-32390: -- Please fill JIRA description. > TRANSFORM with hive serde

[jira] [Commented] (SPARK-32355) 使用Structured Streaming窗口统计不能实现topN

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166104#comment-17166104 ] Hyukjin Kwon commented on SPARK-32355: -- Can you write it in English which most of dev people use to

[jira] [Resolved] (SPARK-32369) pyspark foreach/foreachPartition send http request failed

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32369. -- Resolution: Invalid > pyspark foreach/foreachPartition send http request failed >

[jira] [Resolved] (SPARK-32370) pyspark foreach/foreachPartition send http request failed

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32370. -- Resolution: Duplicate > pyspark foreach/foreachPartition send http request failed >

[jira] [Commented] (SPARK-32369) pyspark foreach/foreachPartition send http request failed

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166096#comment-17166096 ] Hyukjin Kwon commented on SPARK-32369: -- Seems you're running it on Mac. You should probably set

[jira] [Updated] (SPARK-32369) pyspark foreach/foreachPartition send http request failed

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32369: - Description: I use urllib.request to send http request in foreach/foreachPartition. pyspark

[jira] [Commented] (SPARK-32361) Remove project if output is subset of child

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166094#comment-17166094 ] Hyukjin Kwon commented on SPARK-32361: -- Please fill JIRA description. > Remove project if output

[jira] [Commented] (SPARK-32359) Implement max_error metric evaluator for spark regression mllib

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166095#comment-17166095 ] Hyukjin Kwon commented on SPARK-32359: -- Please fill JIRA description. > Implement max_error metric

[jira] [Resolved] (SPARK-32439) Override datasource implementation during look up via configuration

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32439. -- Resolution: Won't Fix > Override datasource implementation during look up via configuration >

[jira] [Resolved] (SPARK-32435) Remove heapq3 port from Python 3

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32435. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29229

[jira] [Assigned] (SPARK-32435) Remove heapq3 port from Python 3

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32435: Assignee: Hyukjin Kwon > Remove heapq3 port from Python 3 >

[jira] [Comment Edited] (SPARK-32453) Remove SPARK_SCALA_VERSION environment and let load-spark-env scripts detect it in AppVeyor

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165549#comment-17165549 ] Hyukjin Kwon edited comment on SPARK-32453 at 7/27/20, 8:57 AM: It will

[jira] [Resolved] (SPARK-32453) Remove SPARK_SCALA_VERSION environment and let load-spark-env scripts detect it in AppVeyor

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32453. -- Resolution: Invalid > Remove SPARK_SCALA_VERSION environment and let load-spark-env scripts

[jira] [Commented] (SPARK-32453) Remove SPARK_SCALA_VERSION environment and let load-spark-env scripts detect it in AppVeyor

2020-07-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165549#comment-17165549 ] Hyukjin Kwon commented on SPARK-32453: -- It will be fixed together at

  1   2   3   4   5   6   7   8   9   10   >