[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655366#comment-16655366 ] Marco Gaido commented on SPARK-25767: - I tried on current master branch but I wasn't able to

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Thomas Brugiere (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655391#comment-16655391 ] Thomas Brugiere commented on SPARK-25767: - If it can help, I have created a github repo with the

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Thomas Brugiere (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655428#comment-16655428 ] Thomas Brugiere commented on SPARK-25767: - It looks like it's not stopping the processing but I

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655440#comment-16655440 ] Marco Gaido commented on SPARK-25767: - So I tracked down the issue. The problem is that you are

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655530#comment-16655530 ] Marco Gaido commented on SPARK-25767: - Your conversion of a Java array in a Scala Seq creates a

[jira] [Updated] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21402: Fix Version/s: (was: 2.4.0) > Java encoders - switch fields on collectAsList >

[jira] [Updated] (SPARK-25768) Constant argument expecting Hive UDAFs doesn't work

2018-10-18 Thread Peter Toth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-25768: --- Description:   The following doesn't work since -SPARK-18186:(- {code:java} test("constant

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655418#comment-16655418 ] Marco Gaido commented on SPARK-25767: - It is interesting, I can reproduce with the Java API but not

[jira] [Updated] (SPARK-25768) Constant argument expecting Hive UDAFs doesn't work

2018-10-18 Thread Peter Toth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-25768: --- Description:   The following doesn't work since SPARK-18186: {code:java} test("constant argument

[jira] [Updated] (SPARK-25768) Constant argument expecting Hive UDAFs doesn't work

2018-10-18 Thread Peter Toth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-25768: --- Summary: Constant argument expecting Hive UDAFs doesn't work (was: Constant argument expecting

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Vladimir Kuriatkov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655295#comment-16655295 ] Vladimir Kuriatkov commented on SPARK-21402: I guess, only array case has been fixed yet,

[jira] [Updated] (SPARK-25758) Deprecate BisectingKMeans compute cost

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25758: -- Issue Type: Task (was: Improvement) > Deprecate BisectingKMeans compute cost >

[jira] [Resolved] (SPARK-25758) Deprecate BisectingKMeans compute cost

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25758. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22756

[jira] [Resolved] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25682. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22681

[jira] [Assigned] (SPARK-25682) Docker images generated from dev build and from dist tarball are different

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25682: -- Assignee: Marcelo Vanzin > Docker images generated from dev build and from dist

[jira] [Assigned] (SPARK-25758) Deprecate BisectingKMeans compute cost

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25758: - Assignee: Marco Gaido > Deprecate BisectingKMeans compute cost >

[jira] [Updated] (SPARK-25769) UnresolvedAttribute.sql() incorrectly escapes nested columns

2018-10-18 Thread Simeon Simeonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simeon Simeonov updated SPARK-25769: Description: {{UnresolvedAttribute.sql()}} output is incorrectly escaped for nested

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655322#comment-16655322 ] Wenchen Fan commented on SPARK-21402: - oh this ticket reports the bug for map type. I'm reopening

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Thomas Brugiere (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655374#comment-16655374 ] Thomas Brugiere commented on SPARK-25767: - Hi Marco, I have noticed the bug using the jar

[jira] [Commented] (SPARK-25665) Refactor ObjectHashAggregateExecBenchmark to use main method

2018-10-18 Thread Peter Toth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655416#comment-16655416 ] Peter Toth commented on SPARK-25665: I will submit a PR but I've run into SPARK-25768. > Refactor

[jira] [Commented] (SPARK-25767) Error reported in Spark logs when using the org.apache.spark:spark-sql_2.11:2.3.2 Java library

2018-10-18 Thread Thomas Brugiere (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655445#comment-16655445 ] Thomas Brugiere commented on SPARK-25767: - Please can you provide an example? I don't see any

[jira] [Resolved] (SPARK-25760) Set AddJarCommand return empty

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25760. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22747

[jira] [Assigned] (SPARK-25760) Set AddJarCommand return empty

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25760: - Assignee: Yuming Wang > Set AddJarCommand return empty >

[jira] [Commented] (SPARK-25768) Constant argument expecting Hive UDAFs doesn't work

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655254#comment-16655254 ] Apache Spark commented on SPARK-25768: -- User 'peter-toth' has created a pull request for this

[jira] [Assigned] (SPARK-25768) Constant argument expecting Hive UDAFs doesn't work

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25768: Assignee: Apache Spark > Constant argument expecting Hive UDAFs doesn't work >

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655314#comment-16655314 ] Wenchen Fan commented on SPARK-21402: - Can you create a ticket for the map case? Basically what we

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

2018-10-18 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655442#comment-16655442 ] Yuming Wang commented on SPARK-21725: - [~owen.omalley] [~ste...@apache.org] I found lots of related

[jira] [Updated] (SPARK-25769) UnresolvedAttribute.sql() incorrectly escapes nested columns

2018-10-18 Thread Simeon Simeonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simeon Simeonov updated SPARK-25769: Description:  This issue affects dynamic SQL generation that relies on {{sql()}}.

[jira] [Created] (SPARK-25769) UnresolvedAttribute.sql() incorrectly escapes nested columns

2018-10-18 Thread Simeon Simeonov (JIRA)
Simeon Simeonov created SPARK-25769: --- Summary: UnresolvedAttribute.sql() incorrectly escapes nested columns Key: SPARK-25769 URL: https://issues.apache.org/jira/browse/SPARK-25769 Project: Spark

[jira] [Reopened] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-21402: - Assignee: (was: Vladimir Kuriatkov) > Java encoders - switch fields on collectAsList >

[jira] [Commented] (SPARK-20760) Memory Leak of RDD blocks

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655369#comment-16655369 ] Wenchen Fan commented on SPARK-20760: - has this been fixed by

[jira] [Assigned] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21402: Assignee: (was: Apache Spark) > Java encoders - switch fields on collectAsList >

[jira] [Assigned] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21402: Assignee: Apache Spark > Java encoders - switch fields on collectAsList >

[jira] [Updated] (SPARK-25768) Constant argument expecting Hive UDAFs doesn't work

2018-10-18 Thread Peter Toth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-25768: --- Description:  The following doesn't work since SPARK-18168 so it is a regression. {code:java}

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655380#comment-16655380 ] Apache Spark commented on SPARK-21402: -- User 'vofque' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25763) Use more `@contextmanager` to ensure clean-up each test.

2018-10-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25763. -- Resolution: Fixed Assignee: Takuya Ueshin Fix Version/s: 3.0.0 fixed in

[jira] [Resolved] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24499. - Resolution: Fixed Assignee: Yuanjian Li Fix Version/s: 2.4.0 Target

[jira] [Updated] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24499: Summary: Split the page of sql-programming-guide.html to multiple separate pages (was: Documentation

[jira] [Updated] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24499: Component/s: (was: Spark Core) > Split the page of sql-programming-guide.html to multiple separate

[jira] [Commented] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655776#comment-16655776 ] Xiao Li commented on SPARK-24499: - Feel free to create the extra tasks to improve the documentation. I

[jira] [Resolved] (SPARK-25761) sparksql执行sql语句的时候,sql语句已经执行成功,但是从sparkui上看该语句还是没有执行完成,还是running状态。

2018-10-18 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-25761. - Resolution: Invalid > sparksql执行sql语句的时候,sql语句已经执行成功,但是从sparkui上看该语句还是没有执行完成,还是running状态。 >

[jira] [Resolved] (SPARK-25764) Avoid usage of deprecated methods in examples for BisectingKMeans

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25764. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22763

[jira] [Assigned] (SPARK-25764) Avoid usage of deprecated methods in examples for BisectingKMeans

2018-10-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-25764: --- Assignee: Marco Gaido > Avoid usage of deprecated methods in examples for BisectingKMeans

[jira] [Assigned] (SPARK-25493) CRLF Line Separators don't work in multiline CSVs

2018-10-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25493: Assignee: Justin Uang > CRLF Line Separators don't work in multiline CSVs >

[jira] [Resolved] (SPARK-25493) CRLF Line Separators don't work in multiline CSVs

2018-10-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25493. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22503

[jira] [Updated] (SPARK-25740) Refactor DetermineTableStats to invalidate cache when some configuration changed

2018-10-18 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25740: Description: How to reproduce: {code:sql} # spark-sql create table t1 (a int) stored as parquet;

[jira] [Created] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread liuxian (JIRA)
liuxian created SPARK-25776: --- Summary: The disk write buffer size must be greater than 12. Key: SPARK-25776 URL: https://issues.apache.org/jira/browse/SPARK-25776 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-25777) Optimize DetermineTableStats if LogicalRelation already cached

2018-10-18 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-25777: --- Summary: Optimize DetermineTableStats if LogicalRelation already cached Key: SPARK-25777 URL: https://issues.apache.org/jira/browse/SPARK-25777 Project: Spark

[jira] [Updated] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-25776: Description: In {color:#205081}{{UnsafeSorterSpillWriter.java}}{color}, when we write a record to a

[jira] [Updated] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-25776: Description: In {color:#205081}{{UnsafeSorterSpillWriter.java}}{color}, when we write a record to a

[jira] [Assigned] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25776: Assignee: (was: Apache Spark) > The disk write buffer size must be greater than 12.

[jira] [Commented] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656177#comment-16656177 ] Apache Spark commented on SPARK-25776: -- User '10110346' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25776: Assignee: Apache Spark > The disk write buffer size must be greater than 12. >

[jira] [Commented] (SPARK-25776) The disk write buffer size must be greater than 12.

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656180#comment-16656180 ] Apache Spark commented on SPARK-25776: -- User '10110346' has created a pull request for this issue:

[jira] [Commented] (SPARK-25777) Optimize DetermineTableStats if LogicalRelation already cached

2018-10-18 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656228#comment-16656228 ] Yuming Wang commented on SPARK-25777: - I'm working on. This needs to fix at least:  

[jira] [Updated] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

2018-10-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-25773: - Description: When a job finishes, there may be some zombie tasks still running due to stage

[jira] [Assigned] (SPARK-25683) Updated the log for the firstTime event Drop occurs.

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25683: -- Assignee: Shivu Sondur > Updated the log for the firstTime event Drop occurs. >

[jira] [Resolved] (SPARK-25683) Updated the log for the firstTime event Drop occurs.

2018-10-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25683. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22677

[jira] [Resolved] (SPARK-21402) Fix java array of structs deserialization

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-21402. --- Resolution: Fixed Target Version/s: 2.2.3, 2.3.3, 2.4.0 > Fix java array of

[jira] [Updated] (SPARK-21402) Fix java array of structs deserialization

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21402: -- Target Version/s: (was: 2.2.3, 2.3.3, 2.4.0) > Fix java array of structs deserialization >

[jira] [Updated] (SPARK-25775) Race between end-of-task and completion iterator read lock release

2018-10-18 Thread Ablimit A. Keskin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ablimit A. Keskin updated SPARK-25775: -- Description: The following issue comes from a production Spark job where executors

[jira] [Updated] (SPARK-25775) Race between end-of-task and completion iterator read lock release

2018-10-18 Thread Ablimit A. Keskin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ablimit A. Keskin updated SPARK-25775: -- Affects Version/s: (was: 2.2.0) 2.2.2 > Race between

[jira] [Created] (SPARK-25770) support SparkDataFrame pretty print

2018-10-18 Thread Weiqiang Zhuang (JIRA)
Weiqiang Zhuang created SPARK-25770: --- Summary: support SparkDataFrame pretty print Key: SPARK-25770 URL: https://issues.apache.org/jira/browse/SPARK-25770 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655824#comment-16655824 ] Apache Spark commented on SPARK-24499: -- User 'gatorsmile' has created a pull request for this

[jira] [Commented] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655825#comment-16655825 ] Apache Spark commented on SPARK-24499: -- User 'gatorsmile' has created a pull request for this

[jira] [Created] (SPARK-25772) Java encoders - switch fields on collectAsList

2018-10-18 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-25772: - Summary: Java encoders - switch fields on collectAsList Key: SPARK-25772 URL: https://issues.apache.org/jira/browse/SPARK-25772 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-25772) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25772: Assignee: Apache Spark > Java encoders - switch fields on collectAsList >

[jira] [Commented] (SPARK-25772) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655861#comment-16655861 ] Apache Spark commented on SPARK-25772: -- User 'vofque' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21402) Fix java array of structs deserialization

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-21402: - Assignee: Vladimir Kuriatkov > Fix java array of structs deserialization >

[jira] [Updated] (SPARK-21402) Fix java array of structs deserialization

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21402: -- Fix Version/s: 2.4.0 2.3.3 2.2.3 > Fix java array of

[jira] [Commented] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656062#comment-16656062 ] Apache Spark commented on SPARK-24499: -- User 'dilipbiswal' has created a pull request for this

[jira] [Commented] (SPARK-24499) Split the page of sql-programming-guide.html to multiple separate pages

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656065#comment-16656065 ] Apache Spark commented on SPARK-24499: -- User 'dilipbiswal' has created a pull request for this

[jira] [Assigned] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25773: Assignee: Apache Spark (was: Shixiong Zhu) > Cancel zombie tasks in a result stage when

[jira] [Assigned] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25773: Assignee: Shixiong Zhu (was: Apache Spark) > Cancel zombie tasks in a result stage when

[jira] [Commented] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655887#comment-16655887 ] Apache Spark commented on SPARK-25773: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655888#comment-16655888 ] Apache Spark commented on SPARK-25773: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Created] (SPARK-25774) Eliminate query anomalies with empty partitions - TRUNCATE, SELECT DISTINCT, etc.

2018-10-18 Thread Steven Cardella (JIRA)
Steven Cardella created SPARK-25774: --- Summary: Eliminate query anomalies with empty partitions - TRUNCATE, SELECT DISTINCT, etc. Key: SPARK-25774 URL: https://issues.apache.org/jira/browse/SPARK-25774

[jira] [Created] (SPARK-25775) Race between end-of-task and completion iterator read lock release

2018-10-18 Thread Ablimit A. Keskin (JIRA)
Ablimit A. Keskin created SPARK-25775: - Summary: Race between end-of-task and completion iterator read lock release Key: SPARK-25775 URL: https://issues.apache.org/jira/browse/SPARK-25775

[jira] [Commented] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655852#comment-16655852 ] Apache Spark commented on SPARK-25771: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655855#comment-16655855 ] Dongjoon Hyun commented on SPARK-21402: --- We already committed with SPARK JIRA ID, SPARK-21402. Too

[jira] [Comment Edited] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655845#comment-16655845 ] Dongjoon Hyun edited comment on SPARK-21402 at 10/18/18 8:47 PM: - I

[jira] [Commented] (SPARK-25769) UnresolvedAttribute.sql() incorrectly escapes nested columns

2018-10-18 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655711#comment-16655711 ] Huaxin Gao commented on SPARK-25769: I will work on this. Thanks! > UnresolvedAttribute.sql()

[jira] [Created] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

2018-10-18 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-25771: Summary: Fix improper synchronization in PythonWorkerFactory Key: SPARK-25771 URL: https://issues.apache.org/jira/browse/SPARK-25771 Project: Spark Issue

[jira] [Created] (SPARK-25773) Cancel zombie tasks in a result stage when the job finishes

2018-10-18 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-25773: Summary: Cancel zombie tasks in a result stage when the job finishes Key: SPARK-25773 URL: https://issues.apache.org/jira/browse/SPARK-25773 Project: Spark

[jira] [Commented] (SPARK-21402) Java encoders - switch fields on collectAsList

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655845#comment-16655845 ] Dongjoon Hyun commented on SPARK-21402: --- I cloned this JIRA issue. Can we do like the followings?

[jira] [Assigned] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25771: Assignee: Apache Spark (was: Shixiong Zhu) > Fix improper synchronization in

[jira] [Assigned] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25771: Assignee: Shixiong Zhu (was: Apache Spark) > Fix improper synchronization in

[jira] [Commented] (SPARK-25771) Fix improper synchronization in PythonWorkerFactory

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655850#comment-16655850 ] Apache Spark commented on SPARK-25771: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25772) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25772: Assignee: (was: Apache Spark) > Java encoders - switch fields on collectAsList >

[jira] [Commented] (SPARK-25772) Java encoders - switch fields on collectAsList

2018-10-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655862#comment-16655862 ] Apache Spark commented on SPARK-25772: -- User 'vofque' has created a pull request for this issue:

[jira] [Updated] (SPARK-21402) Fix java array of structs deserialization

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21402: -- Summary: Fix java array of structs deserialization (was: Java encoders - switch fields on

[jira] [Created] (SPARK-25778) WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access

2018-10-18 Thread Greg Senia (JIRA)
Greg Senia created SPARK-25778: -- Summary: WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access Key: SPARK-25778 URL: https://issues.apache.org/jira/browse/SPARK-25778 Project: Spark

[jira] [Commented] (SPARK-25778) WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access

2018-10-18 Thread Greg Senia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656308#comment-16656308 ] Greg Senia commented on SPARK-25778: Could we add a snippit like this.. I just tested this in my

[jira] [Updated] (SPARK-25775) Race between end-of-task and completion iterator read lock release

2018-10-18 Thread Ablimit A. Keskin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ablimit A. Keskin updated SPARK-25775: -- Description: The following issue comes from a production Spark job where executors

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK9+

2018-10-18 Thread Adrian Cole (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654705#comment-16654705 ] Adrian Cole commented on SPARK-24417: - yes please skip to 11 as in Zipkin we noticed some things

[jira] [Updated] (SPARK-25740) Refactor DetermineTableStats to invalidate cache when some configuration changed

2018-10-18 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-25740: Summary: Refactor DetermineTableStats to invalidate cache when some configuration changed (was:

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK9+

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654835#comment-16654835 ] Dongjoon Hyun commented on SPARK-24417: --- Thank you for the details, [~adriancole]! > Build and

[jira] [Commented] (SPARK-25742) Is there a way to pass the Azure blob storage credentials to the spark for k8s init-container?

2018-10-18 Thread Oscar Bonilla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654917#comment-16654917 ] Oscar Bonilla commented on SPARK-25742: --- Thanks [~hyukjin.kwon]. I have posted to StackOverflow as

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK9+

2018-10-18 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654764#comment-16654764 ] Dongjoon Hyun commented on SPARK-24417: --- [~adriancole]. Just out of curiosity, could you give us

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK9+

2018-10-18 Thread Adrian Cole (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654772#comment-16654772 ] Adrian Cole commented on SPARK-24417: - [~dongjoon] our main server is spring boot and it

  1   2   >