[jira] [Created] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17651: --- Summary: Automate Spark version update for documentations Key: SPARK-17651 URL: https://issues.apache.org/jira/browse/SPARK-17651 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517626#comment-15517626 ] Apache Spark commented on SPARK-17651: -- User 'shivaram' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17651: Assignee: (was: Apache Spark) > Automate Spark version update for documentations >

[jira] [Assigned] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17651: Assignee: Apache Spark > Automate Spark version update for documentations >

[jira] [Resolved] (SPARK-17651) Automate Spark version update for documentations

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17651. - Resolution: Fixed Assignee: Shivaram Venkataraman Fix Version/s: 2.1.0

[jira] [Comment Edited] (SPARK-10713) SPARK_DIST_CLASSPATH ignored on Mesos executors

2016-09-23 Thread Michael McCarthy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517638#comment-15517638 ] Michael McCarthy edited comment on SPARK-10713 at 9/23/16 9:48 PM: --- I'm

[jira] [Assigned] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17652: Assignee: (was: Apache Spark) > Fix confusing exception message while reserving

[jira] [Created] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Kent Yao (JIRA)
Kent Yao created SPARK-17655: Summary: [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Key: SPARK-17655 URL: https://issues.apache.org/jira/browse/SPARK-17655

[jira] [Assigned] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17655: Assignee: Apache Spark > [SQL]Remove unused variables declarations and definations in a

[jira] [Assigned] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17654: Assignee: Apache Spark > Propagate bucketing information for Hive tables to / from

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517589#comment-15517589 ] Josh Rosen commented on SPARK-17647: I think that the first case is clearly a bug (and have a fix)

[jira] [Assigned] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17650: Assignee: (was: Apache Spark) > Adding a malformed URL to sc.addJar and/or sc.addFile

[jira] [Commented] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517591#comment-15517591 ] Apache Spark commented on SPARK-17650: -- User 'brkyvz' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517656#comment-15517656 ] Josh Rosen edited comment on SPARK-17647 at 9/23/16 9:52 PM: - Another piece

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-09-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-15703: - Fix Version/s: 2.0.2 > Make ListenerBus event queue size configurable >

[jira] [Created] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Sameer Agarwal (JIRA)
Sameer Agarwal created SPARK-17652: -- Summary: Fix confusing exception message while reserving capacity Key: SPARK-17652 URL: https://issues.apache.org/jira/browse/SPARK-17652 Project: Spark

[jira] [Commented] (SPARK-17649) Log how many Spark events got dropped in LiveListenerBus

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517949#comment-15517949 ] Apache Spark commented on SPARK-17649: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517656#comment-15517656 ] Josh Rosen commented on SPARK-17647: Another piece of evidence to help untangle this: In MySQL,

[jira] [Commented] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517723#comment-15517723 ] Apache Spark commented on SPARK-17652: -- User 'sameeragarwal' has created a pull request for this

[jira] [Assigned] (SPARK-17652) Fix confusing exception message while reserving capacity

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17652: Assignee: Apache Spark > Fix confusing exception message while reserving capacity >

[jira] [Updated] (SPARK-17655) Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-17655: - Summary: Remove unused variables declarations and definations in a WholeStageCodeGened stage (was:

[jira] [Commented] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518175#comment-15518175 ] Apache Spark commented on SPARK-17655: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17655) [SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17655: Assignee: (was: Apache Spark) > [SQL]Remove unused variables declarations and

[jira] [Commented] (SPARK-17634) Spark job hangs when using dapply

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517837#comment-15517837 ] Felix Cheung commented on SPARK-17634: -- I see. Do you know if the partitions are evenly distributed

[jira] [Commented] (SPARK-17634) Spark job hangs when using dapply

2016-09-23 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517839#comment-15517839 ] Felix Cheung commented on SPARK-17634: -- Also it would be great if you have a shareable example to

[jira] [Created] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Tejas Patil (JIRA)
Tejas Patil created SPARK-17654: --- Summary: Propagate bucketing information for Hive tables to / from Catalog Key: SPARK-17654 URL: https://issues.apache.org/jira/browse/SPARK-17654 Project: Spark

[jira] [Resolved] (SPARK-12221) Add CPU time metric to TaskMetrics

2016-09-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-12221. Resolution: Fixed Assignee: Jisoo Kim Fix Version/s: 2.1.0 > Add CPU time

[jira] [Assigned] (SPARK-17650) Adding a malformed URL to sc.addJar and/or sc.addFile bricks Executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17650: Assignee: Apache Spark > Adding a malformed URL to sc.addJar and/or sc.addFile bricks

[jira] [Commented] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517636#comment-15517636 ] Josh Rosen commented on SPARK-17647: On the other hand, running {code} select '' like '%\\%'

[jira] [Updated] (SPARK-17653) Optimizer should remove unnecessary distincts (in multiple unions)

2016-09-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-17653: Description: Query: {code} select 1 a union select 2 b union select 3 c {code} Explain plan:

[jira] [Created] (SPARK-17653) Optimizer should remove unnecessary distincts (in multiple unions)

2016-09-23 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-17653: --- Summary: Optimizer should remove unnecessary distincts (in multiple unions) Key: SPARK-17653 URL: https://issues.apache.org/jira/browse/SPARK-17653 Project: Spark

[jira] [Assigned] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17654: Assignee: (was: Apache Spark) > Propagate bucketing information for Hive tables to /

[jira] [Commented] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518203#comment-15518203 ] Apache Spark commented on SPARK-17654: -- User 'tejasapatil' has created a pull request for this

[jira] [Commented] (SPARK-10713) SPARK_DIST_CLASSPATH ignored on Mesos executors

2016-09-23 Thread Michael McCarthy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517638#comment-15517638 ] Michael McCarthy commented on SPARK-10713: -- I'm also seeing this issue. Executors get lost with

[jira] [Commented] (SPARK-17654) Propagate bucketing information for Hive tables to / from Catalog

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518468#comment-15518468 ] Apache Spark commented on SPARK-17654: -- User 'tejasapatil' has created a pull request for this

[jira] [Comment Edited] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1551#comment-1551 ] Hyukjin Kwon edited comment on SPARK-17636 at 9/23/16 6:23 AM: ---

[jira] [Commented] (SPARK-14709) spark.ml API for linear SVM

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515583#comment-15515583 ] Apache Spark commented on SPARK-14709: -- User 'hhbyyh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-14709) spark.ml API for linear SVM

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14709: Assignee: (was: Apache Spark) > spark.ml API for linear SVM >

[jira] [Assigned] (SPARK-14709) spark.ml API for linear SVM

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14709: Assignee: Apache Spark > spark.ml API for linear SVM > --- > >

[jira] [Created] (SPARK-17644) The failed stage never resubmitted due to abort stage in another thread

2016-09-23 Thread Fei Wang (JIRA)
Fei Wang created SPARK-17644: Summary: The failed stage never resubmitted due to abort stage in another thread Key: SPARK-17644 URL: https://issues.apache.org/jira/browse/SPARK-17644 Project: Spark

[jira] [Created] (SPARK-17645) Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)

2016-09-23 Thread Peng Meng (JIRA)
Peng Meng created SPARK-17645: - Summary: Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE) Key: SPARK-17645 URL: https://issues.apache.org/jira/browse/SPARK-17645

[jira] [Commented] (SPARK-17645) Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515732#comment-15515732 ] Apache Spark commented on SPARK-17645: -- User 'mpjlu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17645) Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17645: Assignee: (was: Apache Spark) > Add feature selector methods based on: False

[jira] [Comment Edited] (SPARK-17134) Use level 2 BLAS operations in LogisticAggregator

2016-09-23 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515529#comment-15515529 ] Seth Hendrickson edited comment on SPARK-17134 at 9/23/16 6:09 AM: ---

[jira] [Commented] (SPARK-17134) Use level 2 BLAS operations in LogisticAggregator

2016-09-23 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515529#comment-15515529 ] Seth Hendrickson commented on SPARK-17134: -- This makes sense. In my initial testing I found that

[jira] [Assigned] (SPARK-17644) The failed stage never resubmitted due to abort stage in another thread

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17644: Assignee: Apache Spark > The failed stage never resubmitted due to abort stage in another

[jira] [Commented] (SPARK-17644) The failed stage never resubmitted due to abort stage in another thread

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515747#comment-15515747 ] Apache Spark commented on SPARK-17644: -- User 'scwf' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17644) The failed stage never resubmitted due to abort stage in another thread

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17644: Assignee: (was: Apache Spark) > The failed stage never resubmitted due to abort stage

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1551#comment-1551 ] Hyukjin Kwon commented on SPARK-17636: -- Confirmation from committer -

[jira] [Resolved] (SPARK-17640) Avoid using -1 as the default batchId for FileStreamSource.FileEntry

2016-09-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17640. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 > Avoid using -1 as the

[jira] [Assigned] (SPARK-17645) Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17645: Assignee: Apache Spark > Add feature selector methods based on: False Discovery Rate

[jira] [Commented] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516179#comment-15516179 ] Sean Owen commented on SPARK-17646: --- This is not a bug. add does not modify the object but creates a

[jira] [Commented] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy

2016-09-23 Thread Jun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515811#comment-15515811 ] Jun commented on SPARK-10878: - It seems this issue is still active. Is there a good workaround? Does Spark

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515836#comment-15515836 ] Saisai Shao commented on SPARK-17637: - [~zhanzhang] would you mind sharing more details about your

[jira] [Comment Edited] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-23 Thread Gaurav Shah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515948#comment-15515948 ] Gaurav Shah edited comment on SPARK-17527 at 9/23/16 9:31 AM: -- I am unable

[jira] [Resolved] (SPARK-16861) Refactor PySpark accumulator API to be on top of AccumulatorV2 API

2016-09-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16861. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14467

[jira] [Updated] (SPARK-16861) Refactor PySpark accumulator API to be on top of AccumulatorV2 API

2016-09-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16861: -- Assignee: holdenk Priority: Minor (was: Major) > Refactor PySpark accumulator API to be on top of

[jira] [Created] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Pawel Skorupinski (JIRA)
Pawel Skorupinski created SPARK-17646: - Summary: SparkType::add method does not work in 2.0.0 (in Java) Key: SPARK-17646 URL: https://issues.apache.org/jira/browse/SPARK-17646 Project: Spark

[jira] [Comment Edited] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516190#comment-15516190 ] Hyukjin Kwon edited comment on SPARK-17621 at 9/23/16 11:35 AM: Hi

[jira] [Commented] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-23 Thread Gaurav Shah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515948#comment-15515948 ] Gaurav Shah commented on SPARK-17527: - I am unable to create a smaller script that can reproduce this

[jira] [Commented] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516190#comment-15516190 ] Hyukjin Kwon commented on SPARK-17621: -- Hi [~srowen] and [~sreelalsl], I just happened to look at

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 1:05 PM: --- [~rxin] attached a

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh commented on SPARK-17556: - [~scwf]I already submitted a PR for this. Can you also

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh commented on SPARK-17556: - [~Fei Wang] I quickly go through your design doc.

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:49 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:48 PM: --

[jira] [Commented] (SPARK-17017) Add a chiSquare Selector based on False Positive Rate (FPR) test

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516254#comment-15516254 ] Apache Spark commented on SPARK-17017: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Attachment: executor broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 1:15 PM: --- [~rxin] attached a

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:45 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:45 PM: --

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516386#comment-15516386 ] Fei Wang commented on SPARK-17556: -- [~rxin] attached a design doc for the executor based broadcast. Will

[jira] [Commented] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

2016-09-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516283#comment-15516283 ] Sean Owen commented on SPARK-17621: --- Yes, in that sense, the result is correct. The code accumulates

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516545#comment-15516545 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516546#comment-15516546 ] Fei Wang commented on SPARK-17556: -- Not correct, I just collect the broadcast ref to the driver but not

[jira] [Commented] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516307#comment-15516307 ] Hyukjin Kwon commented on SPARK-17621: -- Cool! I just found it is dependent on job stages and

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516577#comment-15516577 ] Liang-Chi Hsieh commented on SPARK-17556: - In other words, from the jira description we say "the

[jira] [Commented] (SPARK-17577) SparkR support add files to Spark job and get by executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516606#comment-15516606 ] Apache Spark commented on SPARK-17577: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang commented on SPARK-17556: -- Yes, the main different is is does not introduce overhead to driver,

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516568#comment-15516568 ] Liang-Chi Hsieh commented on SPARK-17556: - OK. You create the broadcast object on one executor.

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516639#comment-15516639 ] Fei Wang edited comment on SPARK-17556 at 9/23/16 2:50 PM: --- Yes, the main

[jira] [Commented] (SPARK-17454) Add option to specify Mesos resource offer constraints

2016-09-23 Thread Chris Bannister (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516685#comment-15516685 ] Chris Bannister commented on SPARK-17454: - The scheduler will write the to mesos disk sandbox if

[jira] [Issue Comment Deleted] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fei Wang updated SPARK-17556: - Comment: was deleted (was: Not correct, I just collect the broadcast ref to the driver but not the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516668#comment-15516668 ] Liang-Chi Hsieh commented on SPARK-17556: - No. It doesn't. I think the point is not only the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516732#comment-15516732 ] Fei Wang commented on SPARK-17556: -- That's a good point! In your solution, the broadcast rdd must

[jira] [Commented] (SPARK-17577) SparkR support add files to Spark job and get by executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516658#comment-15516658 ] Apache Spark commented on SPARK-17577: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Fei Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516772#comment-15516772 ] Fei Wang commented on SPARK-17556: -- [~viirya] in this case how about notify driver to re-persist the

[jira] [Assigned] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17637: Assignee: (was: Apache Spark) > Packed scheduling for Spark tasks across executors >

[jira] [Assigned] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17637: Assignee: Apache Spark > Packed scheduling for Spark tasks across executors >

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516881#comment-15516881 ] Apache Spark commented on SPARK-17637: -- User 'zhzhan' has created a pull request for this issue:

[jira] [Commented] (SPARK-17619) To add support for pattern matching in ArrayContains Expression.

2016-09-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516905#comment-15516905 ] Hyukjin Kwon commented on SPARK-17619: -- Do you mind if I ask you mean {{array_contains}} function?

[jira] [Updated] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-17647: -- Description: Try the following in SQL shell: {code} select '' like '%\\%'; select ''

[jira] [Created] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-17647: - Summary: SQL LIKE/RLIKE do not handle backslashes correctly Key: SPARK-17647 URL: https://issues.apache.org/jira/browse/SPARK-17647 Project: Spark Issue

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-09-23 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516851#comment-15516851 ] Zhan Zhang commented on SPARK-17637: [~jerryshao] The idea is straightforward. Instead of doing round

[jira] [Commented] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Pawel Skorupinski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516876#comment-15516876 ] Pawel Skorupinski commented on SPARK-17646: --- [~srowen] Correct. If this is an assumed

[jira] [Commented] (SPARK-17634) Spark job hangs when using dapply

2016-09-23 Thread Thomas Powell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516975#comment-15516975 ] Thomas Powell commented on SPARK-17634: --- Several hours. I modified {{worker.R}} so that it logs the

[jira] [Updated] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-17647: -- Description: Try the following in SQL shell: {code} select '' like '%\\%'; select ''

[jira] [Updated] (SPARK-17647) SQL LIKE/RLIKE do not handle backslashes correctly

2016-09-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-17647: -- Labels: correctness (was: ) > SQL LIKE/RLIKE do not handle backslashes correctly >

[jira] [Comment Edited] (SPARK-17646) SparkType::add method does not work in 2.0.0 (in Java)

2016-09-23 Thread Pawel Skorupinski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516876#comment-15516876 ] Pawel Skorupinski edited comment on SPARK-17646 at 9/23/16 4:17 PM:

[jira] [Commented] (SPARK-17454) Add option to specify Mesos resource offer constraints

2016-09-23 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517007#comment-15517007 ] Michael Gummelt commented on SPARK-17454: - So you're trying to only launch executors on nodes

  1   2   >