[jira] [Commented] (SPARK-17211) Broadcast join produces incorrect results on EMR with large driver memory

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456598#comment-15456598 ] Apache Spark commented on SPARK-17211: -- User 'davies' has created a pull request for this issue:

[jira] [Commented] (SPARK-16922) Query with Broadcast Hash join fails due to executor OOM in Spark 2.0

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456597#comment-15456597 ] Apache Spark commented on SPARK-16922: -- User 'davies' has created a pull request for this issue:

[jira] [Resolved] (SPARK-16926) Partition columns are present in columns metadata for partition but not table

2016-09-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-16926. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull

[jira] [Created] (SPARK-17366) Temp tables cached in spark - Joins performance

2016-09-01 Thread Chris Sanjiv Xavier (JIRA)
Chris Sanjiv Xavier created SPARK-17366: --- Summary: Temp tables cached in spark - Joins performance Key: SPARK-17366 URL: https://issues.apache.org/jira/browse/SPARK-17366 Project: Spark

[jira] [Commented] (SPARK-16922) Query with Broadcast Hash join fails due to executor OOM in Spark 2.0

2016-09-01 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456744#comment-15456744 ] Sital Kedia commented on SPARK-16922: - Thanks for the fix [~davies]. I will test this change with our

[jira] [Created] (SPARK-17367) Cannot define value classes in REPL

2016-09-01 Thread Jakob Odersky (JIRA)
Jakob Odersky created SPARK-17367: - Summary: Cannot define value classes in REPL Key: SPARK-17367 URL: https://issues.apache.org/jira/browse/SPARK-17367 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-17211) Broadcast join produces incorrect results on EMR with large driver memory

2016-09-01 Thread gurmukh singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456867#comment-15456867 ] gurmukh singh commented on SPARK-17211: --- Thanks Davies. Can see the issue with the offsets: ***

[jira] [Resolved] (SPARK-17263) Support binary literals in SQL

2016-09-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17263. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14832

[jira] [Assigned] (SPARK-17299) TRIM/LTRIM/RTRIM strips characters other than spaces

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17299: Assignee: (was: Apache Spark) > TRIM/LTRIM/RTRIM strips characters other than spaces

[jira] [Commented] (SPARK-17299) TRIM/LTRIM/RTRIM strips characters other than spaces

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456387#comment-15456387 ] Apache Spark commented on SPARK-17299: -- User 'techaddict' has created a pull request for this issue:

[jira] [Resolved] (SPARK-11138) Flaky pyspark test: test_add_py_file

2016-09-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-11138. Resolution: Cannot Reproduce Haven't seen this in a long while, let's close it for now. >

[jira] [Commented] (SPARK-16922) Query with Broadcast Hash join fails due to executor OOM in Spark 2.0

2016-09-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456605#comment-15456605 ] Davies Liu commented on SPARK-16922: [~sitalke...@gmail.com] I think I found the cause and fix it,

[jira] [Assigned] (SPARK-17211) Broadcast join produces incorrect results on EMR with large driver memory

2016-09-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-17211: -- Assignee: Davies Liu > Broadcast join produces incorrect results on EMR with large driver

[jira] [Updated] (SPARK-16461) Support partition batch pruning with `<=>` (EqualNullSafe) predicate in InMemoryTableScanExec

2016-09-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16461: --- Component/s: SQL > Support partition batch pruning with `<=>` (EqualNullSafe) predicate in >

[jira] [Updated] (SPARK-16461) Support partition batch pruning with `<=>` (EqualNullSafe) predicate in InMemoryTableScanExec

2016-09-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16461: --- Assignee: Hyukjin Kwon > Support partition batch pruning with `<=>` (EqualNullSafe) predicate in >

[jira] [Resolved] (SPARK-16461) Support partition batch pruning with `<=>` (EqualNullSafe) predicate in InMemoryTableScanExec

2016-09-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-16461. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14117

[jira] [Assigned] (SPARK-16922) Query with Broadcast Hash join fails due to executor OOM in Spark 2.0

2016-09-01 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-16922: -- Assignee: Davies Liu > Query with Broadcast Hash join fails due to executor OOM in Spark 2.0

[jira] [Updated] (SPARK-16732) Remove unused codes in subexpressionEliminationForWholeStageCodegen

2016-09-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-16732: --- Fix Version/s: (was: 2.0.1) > Remove unused codes in

[jira] [Commented] (SPARK-17366) Temp tables cached in spark - Joins performance

2016-09-01 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456694#comment-15456694 ] Herman van Hovell commented on SPARK-17366: --- [~chris_sanjiv] Is this a question or a bug

[jira] [Comment Edited] (SPARK-17211) Broadcast join produces incorrect results on EMR with large driver memory

2016-09-01 Thread gurmukh singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456867#comment-15456867 ] gurmukh singh edited comment on SPARK-17211 at 9/1/16 10:52 PM: Thanks

[jira] [Updated] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17356: --- Description: When using MLLib, when calling toJSON on a plan with many level of sub-queries, it may

[jira] [Commented] (SPARK-16334) SQL query on parquet table java.lang.ArrayIndexOutOfBoundsException

2016-09-01 Thread Sebastian Herold (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454644#comment-15454644 ] Sebastian Herold commented on SPARK-16334: -- Sorry for the late response. We noticed the error on

[jira] [Commented] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454553#comment-15454553 ] Sean Zhong commented on SPARK-17356: Reproducer: {code} # Trigger OOM scala> :paste -raw // Entering

[jira] [Commented] (SPARK-17358) Cached table(parquet/orc) should be shard between beelines

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454638#comment-15454638 ] Apache Spark commented on SPARK-17358: -- User 'watermen' has created a pull request for this issue:

[jira] [Commented] (SPARK-17336) Repeated calls sbin/spark-config.sh file Causes ${PYTHONPATH} Value duplicate

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454745#comment-15454745 ] Sean Owen commented on SPARK-17336: --- [~axu4apache] I assume you modified the scripts to print

[jira] [Resolved] (SPARK-17262) Spark SizeEstimator does not ignore transient fields in java classes when calculates class size

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17262. --- Resolution: Not A Problem > Spark SizeEstimator does not ignore transient fields in java classes

[jira] [Commented] (SPARK-16402) JDBC source: Implement save API

2016-09-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454500#comment-15454500 ] Xiao Li commented on SPARK-16402: - : ) Let me ping them. You still can bypass it, right? > JDBC source:

[jira] [Created] (SPARK-17358) Cached table(parquet/orc) should be shard between beelines

2016-09-01 Thread Yadong Qi (JIRA)
Yadong Qi created SPARK-17358: - Summary: Cached table(parquet/orc) should be shard between beelines Key: SPARK-17358 URL: https://issues.apache.org/jira/browse/SPARK-17358 Project: Spark Issue

[jira] [Assigned] (SPARK-17359) Use +=(A) instead of append(A) in performance critical paths

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17359: Assignee: Apache Spark > Use +=(A) instead of append(A) in performance critical paths >

[jira] [Commented] (SPARK-17359) Use +=(A) instead of append(A) in performance critical paths

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454648#comment-15454648 ] Apache Spark commented on SPARK-17359: -- User 'lw-lin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17359) Use +=(A) instead of append(A) in performance critical paths

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17359: Assignee: (was: Apache Spark) > Use +=(A) instead of append(A) in performance

[jira] [Resolved] (SPARK-17257) the physical plan of CREATE TABLE or CTAS should take CatalogTable

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17257. - Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14823

[jira] [Commented] (SPARK-15920) Using map on DataFrame

2016-09-01 Thread Matthias Egli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454430#comment-15454430 ] Matthias Egli commented on SPARK-15920: --- [~srowen] Why was this ticket closed? The documentation of

[jira] [Commented] (SPARK-15920) Using map on DataFrame

2016-09-01 Thread Piotr Milanowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454634#comment-15454634 ] Piotr Milanowski commented on SPARK-15920: -- That is why I used PySpark in components. This

[jira] [Commented] (SPARK-13525) SparkR: java.net.SocketTimeoutException: Accept timed out when running any dataframe function

2016-09-01 Thread Arihanth Jain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454700#comment-15454700 ] Arihanth Jain commented on SPARK-13525: --- Sorry, I am not sure on how to achieve this. I could only

[jira] [Issue Comment Deleted] (SPARK-7445) StringIndexer should handle binary labels properly

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7445: - Comment: was deleted (was: Yuhao on business trip from Aug 31th to Sep 2nd. Email response will be

[jira] [Updated] (SPARK-16283) Implement percentile_approx SQL function

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-16283: Assignee: Sean Zhong > Implement percentile_approx SQL function >

[jira] [Updated] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17356: --- Attachment: queryplan.txt > Out of memory when calling TreeNode.toJSON >

[jira] [Updated] (SPARK-17357) Simplified predicates should be able to pushdown through operators because of the rule order in Optimizer

2016-09-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17357: Summary: Simplified predicates should be able to pushdown through operators because of the

[jira] [Commented] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454495#comment-15454495 ] Sean Zhong commented on SPARK-17356: Root cause: 1. MLLib heavily leverage MetaData to store a lot

[jira] [Comment Edited] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454495#comment-15454495 ] Sean Zhong edited comment on SPARK-17356 at 9/1/16 6:38 AM: *Root cause:* 1.

[jira] [Created] (SPARK-17357) Simplified predicates should be able to pushdown through operators

2016-09-01 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17357: --- Summary: Simplified predicates should be able to pushdown through operators Key: SPARK-17357 URL: https://issues.apache.org/jira/browse/SPARK-17357 Project:

[jira] [Updated] (SPARK-17357) Simplified predicates can't be pushed down through operators because of the rule order in Optimizer

2016-09-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17357: Summary: Simplified predicates can't be pushed down through operators because of the rule

[jira] [Resolved] (SPARK-17329) Don't build PRs with -Pyarn unless YARN code changed

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17329. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14892

[jira] [Updated] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17356: --- Attachment: jmap.txt > Out of memory when calling TreeNode.toJSON >

[jira] [Commented] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454488#comment-15454488 ] Sean Zhong commented on SPARK-17356: *Analysis* After looking at the mmap, there is a suspicious

[jira] [Commented] (SPARK-17357) Simplified predicates can't be pushed down through operators because of the rule order in Optimizer

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454522#comment-15454522 ] Apache Spark commented on SPARK-17357: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17357) Simplified predicates can't be pushed down through operators because of the rule order in Optimizer

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17357: Assignee: (was: Apache Spark) > Simplified predicates can't be pushed down through

[jira] [Assigned] (SPARK-17357) Simplified predicates can't be pushed down through operators because of the rule order in Optimizer

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17357: Assignee: Apache Spark > Simplified predicates can't be pushed down through operators

[jira] [Updated] (SPARK-17347) Encoder in Dataset example has incorrect type

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17347: -- Priority: Trivial (was: Minor) Issue Type: Improvement (was: Bug) >From the PR it doesn't

[jira] [Assigned] (SPARK-17358) Cached table(parquet/orc) should be shard between beelines

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17358: Assignee: (was: Apache Spark) > Cached table(parquet/orc) should be shard between

[jira] [Created] (SPARK-17359) Use +=(A) instead of append(A) in performance critical paths

2016-09-01 Thread Liwei Lin (JIRA)
Liwei Lin created SPARK-17359: - Summary: Use +=(A) instead of append(A) in performance critical paths Key: SPARK-17359 URL: https://issues.apache.org/jira/browse/SPARK-17359 Project: Spark

[jira] [Assigned] (SPARK-17358) Cached table(parquet/orc) should be shard between beelines

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17358: Assignee: Apache Spark > Cached table(parquet/orc) should be shard between beelines >

[jira] [Updated] (SPARK-15920) Fix incorrect DataFrame references in Pyspark docs

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15920: -- Priority: Minor (was: Major) Component/s: Documentation Summary: Fix incorrect

[jira] [Reopened] (SPARK-15920) Using map on DataFrame

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-15920: --- Yeah let's reopen this to make it about correcting the docs. I see in the link above that {{teenagers =

[jira] [Resolved] (SPARK-16283) Implement percentile_approx SQL function

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16283. - Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14868

[jira] [Resolved] (SPARK-17353) CREATE TABLE LIKE statements when Source is a VIEW

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17353. - Resolution: Fixed Fix Version/s: 2.1.0 Target Version/s: 2.0.1, 2.1.0 > CREATE

[jira] [Commented] (SPARK-17307) Document what all access is needed on S3 bucket when trying to save a model

2016-09-01 Thread Aseem Bansal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454791#comment-15454791 ] Aseem Bansal commented on SPARK-17307: -- I would add that bit of information at

[jira] [Updated] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17356: --- Attachment: jstack.txt > Out of memory when calling TreeNode.toJSON >

[jira] [Commented] (SPARK-15920) Using map on DataFrame

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454543#comment-15454543 ] Sean Owen commented on SPARK-15920: --- It was closed because it's a question, and that should go on the

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454777#comment-15454777 ] Saisai Shao commented on SPARK-17340: - I think in your scenario, it is because you killed local

[jira] [Resolved] (SPARK-16942) CREATE TABLE LIKE generates External table when source table is an External Hive Serde table

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16942. - Resolution: Fixed Assignee: Xiao Li Fix Version/s: 2.1.0 Target

[jira] [Updated] (SPARK-17353) CREATE TABLE LIKE statements when Source is a VIEW

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-17353: Assignee: Xiao Li > CREATE TABLE LIKE statements when Source is a VIEW >

[jira] [Resolved] (SPARK-16943) CREATE TABLE LIKE generates a non-empty table when source is a data source table

2016-09-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-16943. - Resolution: Fixed Assignee: Xiao Li Fix Version/s: 2.1.0 Target

[jira] [Resolved] (SPARK-17318) Fix flaky test: o.a.s.repl.ReplSuite replicating blocks of object with class defined in repl

2016-09-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17318. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Fix flaky test:

[jira] [Commented] (SPARK-17365) Kill multiple executors together to reduce lock contention

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456507#comment-15456507 ] Apache Spark commented on SPARK-17365: -- User 'dhruve' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17365) Kill multiple executors together to reduce lock contention

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17365: Assignee: (was: Apache Spark) > Kill multiple executors together to reduce lock

[jira] [Updated] (SPARK-17309) ALTER VIEW should throw exception if view not exist

2016-09-01 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-17309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej BryƄski updated SPARK-17309: --- Fix Version/s: (was: 2.01) 2.0.1 > ALTER VIEW should throw exception

[jira] [Assigned] (SPARK-17365) Kill multiple executors together to reduce lock contention

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17365: Assignee: Apache Spark > Kill multiple executors together to reduce lock contention >

[jira] [Updated] (SPARK-17067) Revocable resource support

2016-09-01 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-17067: Description: Blocked by https://issues.apache.org/jira/browse/MESOS-4392 > Revocable

[jira] [Resolved] (SPARK-17362) fix MultivariantOnlineSummerizer.numNonZeros

2016-09-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17362. --- Resolution: Duplicate > fix MultivariantOnlineSummerizer.numNonZeros >

[jira] [Updated] (SPARK-17356) A large Metadata filed in Alias can cause OOM when calling TreeNode.toJSON

2016-09-01 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-17356: - Target Version/s: 2.0.0, 1.6.3, 2.1.0 (was: 2.0.0, 2.1.0) > A large Metadata filed in Alias can cause

[jira] [Created] (SPARK-17364) Can not query hive table starting with number

2016-09-01 Thread Egor Pahomov (JIRA)
Egor Pahomov created SPARK-17364: Summary: Can not query hive table starting with number Key: SPARK-17364 URL: https://issues.apache.org/jira/browse/SPARK-17364 Project: Spark Issue Type:

[jira] [Closed] (SPARK-5197) Support external shuffle service in fine-grained mode on mesos cluster

2016-09-01 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt closed SPARK-5197. -- Resolution: Won't Fix fine-grained is deprecated > Support external shuffle service in

[jira] [Commented] (SPARK-16533) Spark application not handling preemption messages

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456416#comment-15456416 ] Apache Spark commented on SPARK-16533: -- User 'vanzin' has created a pull request for this issue:

[jira] [Created] (SPARK-17365) Kill multiple executors together to reduce lock contention

2016-09-01 Thread Dhruve Ashar (JIRA)
Dhruve Ashar created SPARK-17365: Summary: Kill multiple executors together to reduce lock contention Key: SPARK-17365 URL: https://issues.apache.org/jira/browse/SPARK-17365 Project: Spark

[jira] [Closed] (SPARK-6679) java.lang.ClassNotFoundException on Mesos fine grained mode and input replication

2016-09-01 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt closed SPARK-6679. -- Resolution: Won't Fix fine-grained is deprecated > java.lang.ClassNotFoundException on Mesos

[jira] [Closed] (SPARK-11183) enable support for mesos 0.24+

2016-09-01 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt closed SPARK-11183. --- Resolution: Done > enable support for mesos 0.24+ > -- > >

[jira] [Resolved] (SPARK-17331) Avoid allocating 0-length arrays

2016-09-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-17331. Resolution: Fixed Fix Version/s: 2.1.0 > Avoid allocating 0-length arrays >

[jira] [Assigned] (SPARK-17299) TRIM/LTRIM/RTRIM strips characters other than spaces

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17299: Assignee: Apache Spark > TRIM/LTRIM/RTRIM strips characters other than spaces >

[jira] [Assigned] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17340: Assignee: Apache Spark > .sparkStaging not cleaned if application exited incorrectly >

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455050#comment-15455050 ] Apache Spark commented on SPARK-17340: -- User 'Devian-ua' has created a pull request for this issue:

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455055#comment-15455055 ] Saisai Shao commented on SPARK-17340: - I'm saying yarn cluster mode, I think here in my comment

[jira] [Assigned] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17142: Assignee: Apache Spark > Complex query triggers binding error in HashAggregateExec >

[jira] [Assigned] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17142: Assignee: (was: Apache Spark) > Complex query triggers binding error in

[jira] [Commented] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455069#comment-15455069 ] Apache Spark commented on SPARK-17142: -- User 'jiangxb1987' has created a pull request for this

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455086#comment-15455086 ] Artur Sukhenko commented on SPARK-17340: Yes, I can. However some users will stop app like this

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455076#comment-15455076 ] Saisai Shao commented on SPARK-17340: - You can try not kill local {{yarn#client}} process after

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455077#comment-15455077 ] Artur Sukhenko commented on SPARK-17340: Oh, now I get this. However, if you stop it in

[jira] [Commented] (SPARK-13301) PySpark Dataframe return wrong results with custom UDF

2016-09-01 Thread Tom Phillips (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455112#comment-15455112 ] Tom Phillips commented on SPARK-13301: -- I'm unable to reproduce this bug using the code above on 1.5

[jira] [Assigned] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17356: Assignee: (was: Apache Spark) > Out of memory when calling TreeNode.toJSON >

[jira] [Commented] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454841#comment-15454841 ] Apache Spark commented on SPARK-17356: -- User 'clockfly' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17356) Out of memory when calling TreeNode.toJSON

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17356: Assignee: Apache Spark > Out of memory when calling TreeNode.toJSON >

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455038#comment-15455038 ] Artur Sukhenko commented on SPARK-17340: With yarn-client it will be cleaned up. I am talking

[jira] [Comment Edited] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-01 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454837#comment-15454837 ] Jiang Xingbo edited comment on SPARK-17142 at 9/1/16 9:07 AM: -- In

[jira] [Comment Edited] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-01 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454837#comment-15454837 ] Jiang Xingbo edited comment on SPARK-17142 at 9/1/16 9:07 AM: -- In

[jira] [Commented] (SPARK-17142) Complex query triggers binding error in HashAggregateExec

2016-09-01 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454837#comment-15454837 ] Jiang Xingbo commented on SPARK-17142: -- In `ReorderAssociativeOperator` rule, we extract foldable

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455080#comment-15455080 ] Saisai Shao commented on SPARK-17340: - yarn-client and yarn-cluster has different way to handle

[jira] [Assigned] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17340: Assignee: (was: Apache Spark) > .sparkStaging not cleaned if application exited

[jira] [Comment Edited] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

2016-09-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454777#comment-15454777 ] Saisai Shao edited comment on SPARK-17340 at 9/1/16 11:02 AM: -- I think in

  1   2   >