[jira] [Resolved] (SPARK-23799) [CBO] FilterEstimation.evaluateInSet produces devision by zero in a case of empty table with analyzed statistics

2018-04-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23799. - Resolution: Fixed Fix Version/s: 2.4.0 2.3.1 > [CBO]

[jira] [Assigned] (SPARK-24044) Explicitly print out skipped tests from unittest module

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24044: Assignee: (was: Apache Spark) > Explicitly print out skipped tests from unittest

[jira] [Assigned] (SPARK-24044) Explicitly print out skipped tests from unittest module

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24044: Assignee: Apache Spark > Explicitly print out skipped tests from unittest module >

[jira] [Commented] (SPARK-24044) Explicitly print out skipped tests from unittest module

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447169#comment-16447169 ] Apache Spark commented on SPARK-24044: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-23799) [CBO] FilterEstimation.evaluateInSet produces devision by zero in a case of empty table with analyzed statistics

2018-04-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-23799: --- Assignee: Michael Shtelma > [CBO] FilterEstimation.evaluateInSet produces devision by zero in a

[jira] [Updated] (SPARK-24044) Explicitly print out skipped tests from unittest module

2018-04-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24044: - Description: There was an actual issue, SPARK-23300, and we fixed this by manually checking if

[jira] [Created] (SPARK-24044) Explicitly print out skipped tests from unittest module

2018-04-22 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-24044: Summary: Explicitly print out skipped tests from unittest module Key: SPARK-24044 URL: https://issues.apache.org/jira/browse/SPARK-24044 Project: Spark

[jira] [Commented] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447349#comment-16447349 ] Gerard Maas commented on SPARK-24046: - In graphical form, this is what is happening when  rampUpTime

[jira] [Updated] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerard Maas updated SPARK-24046: Attachment: image-2018-04-22-22-06-49-202.png > Rate Source doesn't gradually increase rate when

[jira] [Created] (SPARK-24047) use spark package to load csv file

2018-04-22 Thread Jijiao Zeng (JIRA)
Jijiao Zeng created SPARK-24047: --- Summary: use spark package to load csv file Key: SPARK-24047 URL: https://issues.apache.org/jira/browse/SPARK-24047 Project: Spark Issue Type: IT Help

[jira] [Created] (SPARK-24048) load a local csv file

2018-04-22 Thread Jijiao Zeng (JIRA)
Jijiao Zeng created SPARK-24048: --- Summary: load a local csv file Key: SPARK-24048 URL: https://issues.apache.org/jira/browse/SPARK-24048 Project: Spark Issue Type: IT Help

[jira] [Updated] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerard Maas updated SPARK-24046: Attachment: image-2018-04-22-22-03-03-945.png > Rate Source doesn't gradually increase rate when

[jira] [Commented] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447440#comment-16447440 ] Tathagata Das commented on SPARK-23004: --- [~joshrosen] hit the issue as well, and thanks to him I

[jira] [Commented] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread secfree (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447441#comment-16447441 ] secfree commented on SPARK-23004: - secfree 赞了您的邮件。 Spark by Readdle > Structured Streaming raise

[jira] [Resolved] (SPARK-24031) the method of postTaskEnd should write once in handleTaskCompletion

2018-04-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24031. --- Resolution: Invalid > the method of postTaskEnd should write once in handleTaskCompletion >

[jira] [Resolved] (SPARK-24048) load a local csv file

2018-04-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24048. -- Resolution: Invalid Sounds more like a question. Please ask it to the mailing list and I think

[jira] [Created] (SPARK-24050) StreamingQuery does not calculate input / processing rates in some cases

2018-04-22 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24050: - Summary: StreamingQuery does not calculate input / processing rates in some cases Key: SPARK-24050 URL: https://issues.apache.org/jira/browse/SPARK-24050 Project:

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Description:         {code:java} java.lang.IllegalStateException: Cannot remove after

[jira] [Assigned] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23004: Assignee: Apache Spark (was: Tathagata Das) > Structured Streaming raise

[jira] [Assigned] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23004: Assignee: Tathagata Das (was: Apache Spark) > Structured Streaming raise

[jira] [Commented] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447449#comment-16447449 ] Apache Spark commented on SPARK-23004: -- User 'tdas' has created a pull request for this issue:

[jira] [Resolved] (SPARK-24047) use spark package to load csv file

2018-04-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24047. -- Resolution: Invalid Sounds more like a question. Please ask it to the mailing list. You could

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Target Version/s: 2.3.1, 2.4.0 (was: 2.3.1) > Structured Streaming raise

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Component/s: (was: Input/Output) Structured Streaming > Structured

[jira] [Comment Edited] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447440#comment-16447440 ] Tathagata Das edited comment on SPARK-23004 at 4/23/18 1:16 AM:

[jira] [Commented] (SPARK-24024) Fix deviance calculations in GLM to handle corner cases

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447478#comment-16447478 ] Apache Spark commented on SPARK-24024: -- User 'tengpeng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24024) Fix deviance calculations in GLM to handle corner cases

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24024: Assignee: Apache Spark > Fix deviance calculations in GLM to handle corner cases >

[jira] [Assigned] (SPARK-24024) Fix deviance calculations in GLM to handle corner cases

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24024: Assignee: (was: Apache Spark) > Fix deviance calculations in GLM to handle corner

[jira] [Updated] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerard Maas updated SPARK-24046: Summary: Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond (was: Rate

[jira] [Commented] (SPARK-24046) Rate Source doesn't gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447347#comment-16447347 ] Gerard Maas commented on SPARK-24046: - The problem seems to come from this integer division:

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Description: {{A structured streaming query with streaming aggregations can throw the

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Description: A structured streaming query with a streaming aggregation can throw the

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Target Version/s: 2.3.1 > Structured Streaming raise "llegalStateException: Cannot remove

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Affects Version/s: 2.1.0 2.1.1 2.1.2

[jira] [Assigned] (SPARK-22372) Make YARN client extend SparkApplication

2018-04-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22372: --- Assignee: Marcelo Vanzin > Make YARN client extend SparkApplication >

[jira] [Assigned] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-23004: - Assignee: Tathagata Das > Structured Streaming raise "llegalStateException: Cannot

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

2018-04-22 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-23004: -- Description: A structured streaming query with streaming aggregations can throw the following

[jira] [Updated] (SPARK-24049) Add a feature to not start speculative tasks when average task duration is less than a configurable absolute number

2018-04-22 Thread Lars Francke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke updated SPARK-24049: - Summary: Add a feature to not start speculative tasks when average task duration is less than a

[jira] [Created] (SPARK-24049) Add a feature to not start speculative tasks before an absolute minimum amount of time has passed

2018-04-22 Thread Lars Francke (JIRA)
Lars Francke created SPARK-24049: Summary: Add a feature to not start speculative tasks before an absolute minimum amount of time has passed Key: SPARK-24049 URL: https://issues.apache.org/jira/browse/SPARK-24049

[jira] [Comment Edited] (SPARK-21476) RandomForest classification model not using broadcast in transform

2018-04-22 Thread Tamilselvan Veeramani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447545#comment-16447545 ] Tamilselvan Veeramani edited comment on SPARK-21476 at 4/23/18 4:10 AM:

[jira] [Comment Edited] (SPARK-21476) RandomForest classification model not using broadcast in transform

2018-04-22 Thread Tamilselvan Veeramani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447545#comment-16447545 ] Tamilselvan Veeramani edited comment on SPARK-21476 at 4/23/18 4:10 AM:

[jira] [Commented] (SPARK-21476) RandomForest classification model not using broadcast in transform

2018-04-22 Thread Tamilselvan Veeramani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447545#comment-16447545 ] Tamilselvan Veeramani commented on SPARK-21476: --- I am facing the same issue - high task

[jira] [Created] (SPARK-24046) rampUpTime in Rate Source does not work for rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
Gerard Maas created SPARK-24046: --- Summary: rampUpTime in Rate Source does not work for rampUpTime>=RowsPerSecond Key: SPARK-24046 URL: https://issues.apache.org/jira/browse/SPARK-24046 Project: Spark

[jira] [Updated] (SPARK-24046) Rate Source does gradually increase rate when rampUpTime>=RowsPerSecond

2018-04-22 Thread Gerard Maas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerard Maas updated SPARK-24046: Summary: Rate Source does gradually increase rate when rampUpTime>=RowsPerSecond (was: rampUpTime

[jira] [Resolved] (SPARK-24034) StopIteration in pyspark mapper gives partial results

2018-04-22 Thread Emilio Dorigatti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emilio Dorigatti resolved SPARK-24034. -- Resolution: Duplicate > StopIteration in pyspark mapper gives partial results >

[jira] [Created] (SPARK-24045) Create base class for file data source v2

2018-04-22 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-24045: -- Summary: Create base class for file data source v2 Key: SPARK-24045 URL: https://issues.apache.org/jira/browse/SPARK-24045 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24045) Create base class for file data source v2

2018-04-22 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-24045: --- Description: >From https://github.com/apache/spark/pull/20933, we can see that during File

[jira] [Commented] (SPARK-24045) Create base class for file data source v2

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447239#comment-16447239 ] Apache Spark commented on SPARK-24045: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-24045) Create base class for file data source v2

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24045: Assignee: Apache Spark > Create base class for file data source v2 >

[jira] [Assigned] (SPARK-24045) Create base class for file data source v2

2018-04-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24045: Assignee: (was: Apache Spark) > Create base class for file data source v2 >