[jira] [Commented] (SPARK-19634) Feature parity for descriptive statistics in MLlib

2017-12-04 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278148#comment-16278148 ] zhengruifeng commented on SPARK-19634: -- I think we can now use the new summarizer in the algs. >

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2017-12-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278142#comment-16278142 ] Hyukjin Kwon commented on SPARK-22674: -- If that deduplication brings performance regression or is

[jira] [Assigned] (SPARK-22690) Imputer inherit HasOutputCols

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22690: Assignee: (was: Apache Spark) > Imputer inherit HasOutputCols >

[jira] [Assigned] (SPARK-22690) Imputer inherit HasOutputCols

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22690: Assignee: Apache Spark > Imputer inherit HasOutputCols > - >

[jira] [Commented] (SPARK-22690) Imputer inherit HasOutputCols

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278128#comment-16278128 ] Apache Spark commented on SPARK-22690: -- User 'zhengruifeng' has created a pull request for this

[jira] [Updated] (SPARK-22689) Could not resolve dependencies for project

2017-12-04 Thread Puja Mudaliar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Puja Mudaliar updated SPARK-22689: -- Description: Hello team, Spark code compile operation fails on few machines whereas the same

[jira] [Created] (SPARK-22689) Could not resolve dependencies for project

2017-12-04 Thread Puja Mudaliar (JIRA)
Puja Mudaliar created SPARK-22689: - Summary: Could not resolve dependencies for project Key: SPARK-22689 URL: https://issues.apache.org/jira/browse/SPARK-22689 Project: Spark Issue Type:

[jira] [Created] (SPARK-22690) Imputer inherit HasOutputCols

2017-12-04 Thread zhengruifeng (JIRA)
zhengruifeng created SPARK-22690: Summary: Imputer inherit HasOutputCols Key: SPARK-22690 URL: https://issues.apache.org/jira/browse/SPARK-22690 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2017-12-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278121#comment-16278121 ] Hyukjin Kwon commented on SPARK-22674: -- Oh, sorry, I overlooked at {{ that regular pickle won't be

[jira] [Updated] (SPARK-22688) Upgrade Janino version 3.0.8

2017-12-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22688: - Description: [Janino 3.0.8|https://janino-compiler.github.io/janino/changelog.html]

[jira] [Updated] (SPARK-22688) Upgrade Janino version 3.0.8

2017-12-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22688: - Summary: Upgrade Janino version 3.0.8 (was: Upgrade Janino version 0.3.8) > Upgrade

[jira] [Created] (SPARK-22688) Upgrade Janino version 0.3.8

2017-12-04 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-22688: Summary: Upgrade Janino version 0.3.8 Key: SPARK-22688 URL: https://issues.apache.org/jira/browse/SPARK-22688 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-22660) Compile with scala-2.12 and JDK9

2017-12-04 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278064#comment-16278064 ] liyunzhang commented on SPARK-22660: Ok,create SPARK-22687 to record the problem about runtime.

[jira] [Created] (SPARK-22687) Run spark-sql in scala-2.12 and JDK9

2017-12-04 Thread liyunzhang (JIRA)
liyunzhang created SPARK-22687: -- Summary: Run spark-sql in scala-2.12 and JDK9 Key: SPARK-22687 URL: https://issues.apache.org/jira/browse/SPARK-22687 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-8971) Support balanced class labels when splitting train/cross validation sets

2017-12-04 Thread Ashish Chopra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278060#comment-16278060 ] Ashish Chopra commented on SPARK-8971: -- When can we expect this in Dataframe API? > Support balanced

[jira] [Assigned] (SPARK-22686) DROP TABLE IF EXISTS should not throw AnalysisException

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22686: Assignee: Apache Spark > DROP TABLE IF EXISTS should not throw AnalysisException >

[jira] [Assigned] (SPARK-22686) DROP TABLE IF EXISTS should not throw AnalysisException

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22686: Assignee: (was: Apache Spark) > DROP TABLE IF EXISTS should not throw

[jira] [Commented] (SPARK-22686) DROP TABLE IF EXISTS should not throw AnalysisException

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278022#comment-16278022 ] Apache Spark commented on SPARK-22686: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-22686) DROP TABLE IF EXISTS should not throw AnalysisException

2017-12-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22686: -- Summary: DROP TABLE IF EXISTS should not throw AnalysisException (was: DROP TABLE IF NOT

[jira] [Created] (SPARK-22686) DROP TABLE IF NOT EXISTS should not throw AnalysisException

2017-12-04 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-22686: - Summary: DROP TABLE IF NOT EXISTS should not throw AnalysisException Key: SPARK-22686 URL: https://issues.apache.org/jira/browse/SPARK-22686 Project: Spark

[jira] [Resolved] (SPARK-22682) HashExpression does not need to create global variables

2017-12-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22682. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19878

[jira] [Resolved] (SPARK-22677) cleanup whole stage codegen for hash aggregate

2017-12-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22677. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19869

[jira] [Comment Edited] (SPARK-22365) Spark UI executors empty list with 500 error

2017-12-04 Thread bruce xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276618#comment-16276618 ] bruce xu edited comment on SPARK-22365 at 12/5/17 4:31 AM: --- Hi [~dubovsky].

[jira] [Comment Edited] (SPARK-22365) Spark UI executors empty list with 500 error

2017-12-04 Thread bruce xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276618#comment-16276618 ] bruce xu edited comment on SPARK-22365 at 12/5/17 3:47 AM: --- Hi [~dubovsky].

[jira] [Comment Edited] (SPARK-22365) Spark UI executors empty list with 500 error

2017-12-04 Thread bruce xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276618#comment-16276618 ] bruce xu edited comment on SPARK-22365 at 12/5/17 3:46 AM: --- Hi [~dubovsky].

[jira] [Updated] (SPARK-18801) Support resolve a nested view

2017-12-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-18801: -- Fix Version/s: 2.2.0 > Support resolve a nested view > - > >

[jira] [Commented] (SPARK-21168) KafkaRDD should always set kafka clientId.

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277939#comment-16277939 ] Apache Spark commented on SPARK-21168: -- User 'liu-zhaokun' has created a pull request for this

[jira] [Created] (SPARK-22685) Spark Streaming using Kinesis doesn't work if shard checkpoints exist in DynamoDB

2017-12-04 Thread Grega Kespret (JIRA)
Grega Kespret created SPARK-22685: - Summary: Spark Streaming using Kinesis doesn't work if shard checkpoints exist in DynamoDB Key: SPARK-22685 URL: https://issues.apache.org/jira/browse/SPARK-22685

[jira] [Commented] (SPARK-21168) KafkaRDD should always set kafka clientId.

2017-12-04 Thread liuzhaokun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277875#comment-16277875 ] liuzhaokun commented on SPARK-21168: [~dixingx...@yeah.net] Hi,as your PR are not in progess,can I

[jira] [Resolved] (SPARK-22656) Upgrade Arrow to 0.8.0

2017-12-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22656. -- Resolution: Duplicate > Upgrade Arrow to 0.8.0 > -- > >

[jira] [Assigned] (SPARK-22665) Dataset API: .repartition() inconsistency / issue

2017-12-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-22665: --- Assignee: Marco Gaido > Dataset API: .repartition() inconsistency / issue >

[jira] [Resolved] (SPARK-22665) Dataset API: .repartition() inconsistency / issue

2017-12-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22665. - Resolution: Fixed Fix Version/s: 2.3.0 > Dataset API: .repartition() inconsistency / issue >

[jira] [Commented] (SPARK-22162) Executors and the driver use inconsistent Job IDs during the new RDD commit protocol

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277795#comment-16277795 ] Apache Spark commented on SPARK-22162: -- User 'rezasafi' has created a pull request for this issue:

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277787#comment-16277787 ] Apache Spark commented on SPARK-22587: -- User 'merlintang' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-12-04 Thread Mingjie Tang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273745#comment-16273745 ] Mingjie Tang edited comment on SPARK-22587 at 12/5/17 12:01 AM: we can

[jira] [Assigned] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22587: Assignee: (was: Apache Spark) > Spark job fails if fs.defaultFS and application jar

[jira] [Assigned] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22587: Assignee: Apache Spark > Spark job fails if fs.defaultFS and application jar are

[jira] [Assigned] (SPARK-22324) Upgrade Arrow to version 0.8.0

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22324: Assignee: Apache Spark > Upgrade Arrow to version 0.8.0 > --

[jira] [Commented] (SPARK-22324) Upgrade Arrow to version 0.8.0

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277755#comment-16277755 ] Apache Spark commented on SPARK-22324: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-22324) Upgrade Arrow to version 0.8.0

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22324: Assignee: (was: Apache Spark) > Upgrade Arrow to version 0.8.0 >

[jira] [Commented] (SPARK-22599) Avoid extra reading for cached table

2017-12-04 Thread Rajesh Balamohan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277754#comment-16277754 ] Rajesh Balamohan commented on SPARK-22599: -- [~CodingCat] - Thanks for sharing results. Results

[jira] [Commented] (SPARK-20368) Support Sentry on PySpark workers

2017-12-04 Thread Taylor Edmiston (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277710#comment-16277710 ] Taylor Edmiston commented on SPARK-20368: - I also posted this on the PR linked in the comment

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-12-04 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277615#comment-16277615 ] Li Jin commented on SPARK-21187: Gotcha. Thanks! > Complete support for remaining Spark data types in

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2017-12-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277577#comment-16277577 ] Hyukjin Kwon commented on SPARK-22674: -- Basically yes, for now. I think we should avoid having a

[jira] [Assigned] (SPARK-22684) Avoid the generation of useless mutable states by datetime functions

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22684: Assignee: Apache Spark > Avoid the generation of useless mutable states by datetime

[jira] [Assigned] (SPARK-22684) Avoid the generation of useless mutable states by datetime functions

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22684: Assignee: (was: Apache Spark) > Avoid the generation of useless mutable states by

[jira] [Commented] (SPARK-22684) Avoid the generation of useless mutable states by datetime functions

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277559#comment-16277559 ] Apache Spark commented on SPARK-22684: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-22672) Refactor ORC Tests

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277553#comment-16277553 ] Apache Spark commented on SPARK-22672: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-12-04 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277549#comment-16277549 ] Bryan Cutler commented on SPARK-21187: -- Hi [~icexelloss], StructType has been added on the Java

[jira] [Created] (SPARK-22684) Avoid the generation of useless mutable states by datetime functions

2017-12-04 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-22684: --- Summary: Avoid the generation of useless mutable states by datetime functions Key: SPARK-22684 URL: https://issues.apache.org/jira/browse/SPARK-22684 Project: Spark

[jira] [Updated] (SPARK-22672) Refactor ORC Tests

2017-12-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22672: -- Description: Since SPARK-20682, we have two `OrcFileFormat`s. This issue refactor ORC tests.

[jira] [Updated] (SPARK-22672) Refactor ORC Tests

2017-12-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22672: -- Summary: Refactor ORC Tests (was: Move OrcTest to `sql/core`) > Refactor ORC Tests >

[jira] [Updated] (SPARK-22672) Refactor ORC Tests

2017-12-04 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22672: -- Priority: Major (was: Trivial) > Refactor ORC Tests > -- > >

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

2017-12-04 Thread Jonas Amrich (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277424#comment-16277424 ] Jonas Amrich commented on SPARK-22674: -- Sure, you're right that pickle won't unpickle it without

[jira] [Resolved] (SPARK-22372) Make YARN client extend SparkApplication

2017-12-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22372. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19631

[jira] [Commented] (SPARK-21187) Complete support for remaining Spark data types in Arrow Converters

2017-12-04 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277271#comment-16277271 ] Li Jin commented on SPARK-21187: [~bryanc] Thanks for the update! Is there any thing particular needs to

[jira] [Updated] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Julien Cuquemelle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Cuquemelle updated SPARK-22683: -- Labels: pull-request-available (was: ) Description: let's say an executor

[jira] [Commented] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277187#comment-16277187 ] Apache Spark commented on SPARK-22683: -- User 'jcuquemelle' has created a pull request for this

[jira] [Assigned] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22683: Assignee: (was: Apache Spark) > Allow tuning the number of dynamically allocated

[jira] [Assigned] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22683: Assignee: Apache Spark > Allow tuning the number of dynamically allocated executors wrt

[jira] [Updated] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Julien Cuquemelle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Cuquemelle updated SPARK-22683: -- Priority: Major (was: Minor) > Allow tuning the number of dynamically allocated

[jira] [Updated] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22683: -- Target Version/s: (was: 2.1.1, 2.2.0) The overhead of small tasks doesn't change if you over-commit

[jira] [Created] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-04 Thread Julien Cuquemelle (JIRA)
Julien Cuquemelle created SPARK-22683: - Summary: Allow tuning the number of dynamically allocated executors wrt task number Key: SPARK-22683 URL: https://issues.apache.org/jira/browse/SPARK-22683

[jira] [Resolved] (SPARK-22162) Executors and the driver use inconsistent Job IDs during the new RDD commit protocol

2017-12-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22162. Resolution: Fixed Assignee: Reza Safi Fix Version/s: 2.3.0 > Executors and

[jira] [Updated] (SPARK-22162) Executors and the driver use inconsistent Job IDs during the new RDD commit protocol

2017-12-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-22162: --- Affects Version/s: (was: 2.3.0) > Executors and the driver use inconsistent Job IDs

[jira] [Commented] (SPARK-22626) Wrong Hive table statistics may trigger OOM if enables CBO

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276988#comment-16276988 ] Apache Spark commented on SPARK-22626: -- User 'wzhfy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20706) Spark-shell not overriding method/variable definition

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20706: Assignee: (was: Apache Spark) > Spark-shell not overriding method/variable definition

[jira] [Commented] (SPARK-20706) Spark-shell not overriding method/variable definition

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276948#comment-16276948 ] Apache Spark commented on SPARK-20706: -- User 'mpetruska' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20706) Spark-shell not overriding method/variable definition

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20706: Assignee: Apache Spark > Spark-shell not overriding method/variable definition >

[jira] [Assigned] (SPARK-22682) HashExpression does not need to create global variables

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22682: Assignee: Wenchen Fan (was: Apache Spark) > HashExpression does not need to create

[jira] [Commented] (SPARK-22682) HashExpression does not need to create global variables

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276945#comment-16276945 ] Apache Spark commented on SPARK-22682: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22682) HashExpression does not need to create global variables

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22682: Assignee: Apache Spark (was: Wenchen Fan) > HashExpression does not need to create

[jira] [Created] (SPARK-22682) HashExpression does not need to create global variables

2017-12-04 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22682: --- Summary: HashExpression does not need to create global variables Key: SPARK-22682 URL: https://issues.apache.org/jira/browse/SPARK-22682 Project: Spark Issue

[jira] [Commented] (SPARK-20706) Spark-shell not overriding method/variable definition

2017-12-04 Thread Mark Petruska (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276921#comment-16276921 ] Mark Petruska commented on SPARK-20706: --- This is a Scala repl bug, see:

[jira] [Commented] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2017-12-04 Thread Sasaki Toru (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276892#comment-16276892 ] Sasaki Toru commented on SPARK-20050: - Thank you comment. I think this patch can be backported to

[jira] [Comment Edited] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2017-12-04 Thread Sasaki Toru (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276892#comment-16276892 ] Sasaki Toru edited comment on SPARK-20050 at 12/4/17 2:54 PM: -- Thank you

[jira] [Commented] (SPARK-1940) Enable rolling of executor logs (stdout / stderr)

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276759#comment-16276759 ] Apache Spark commented on SPARK-1940: - User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22681) Accumulator should only be updated once for each task in result stage

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22681: Assignee: Apache Spark > Accumulator should only be updated once for each task in result

[jira] [Commented] (SPARK-22681) Accumulator should only be updated once for each task in result stage

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276718#comment-16276718 ] Apache Spark commented on SPARK-22681: -- User 'carsonwang' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22681) Accumulator should only be updated once for each task in result stage

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22681: Assignee: (was: Apache Spark) > Accumulator should only be updated once for each task

[jira] [Created] (SPARK-22681) Accumulator should only be updated once for each task in result stage

2017-12-04 Thread Carson Wang (JIRA)
Carson Wang created SPARK-22681: --- Summary: Accumulator should only be updated once for each task in result stage Key: SPARK-22681 URL: https://issues.apache.org/jira/browse/SPARK-22681 Project: Spark

[jira] [Updated] (SPARK-22680) SparkSQL scan all partitions when the specified partitions are not exists in parquet formatted table

2017-12-04 Thread Xiaochen Ouyang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaochen Ouyang updated SPARK-22680: Summary: SparkSQL scan all partitions when the specified partitions are not exists in

[jira] [Created] (SPARK-22680) SparkSQL scan all partitions when specified partition is not exists in parquet formatted table

2017-12-04 Thread Xiaochen Ouyang (JIRA)
Xiaochen Ouyang created SPARK-22680: --- Summary: SparkSQL scan all partitions when specified partition is not exists in parquet formatted table Key: SPARK-22680 URL:

[jira] [Assigned] (SPARK-11239) PMML export for ML linear regression

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11239: Assignee: Apache Spark > PMML export for ML linear regression >

[jira] [Commented] (SPARK-11239) PMML export for ML linear regression

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276643#comment-16276643 ] Apache Spark commented on SPARK-11239: -- User 'holdenk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-11239) PMML export for ML linear regression

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11239: Assignee: (was: Apache Spark) > PMML export for ML linear regression >

[jira] [Commented] (SPARK-11171) PMML for Pipelines API

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276640#comment-16276640 ] Apache Spark commented on SPARK-11171: -- User 'holdenk' has created a pull request for this issue:

[jira] [Commented] (SPARK-22473) Replace deprecated AsyncAssertions.Waiter and methods of java.sql.Date

2017-12-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276629#comment-16276629 ] Apache Spark commented on SPARK-22473: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Commented] (SPARK-22365) Spark UI executors empty list with 500 error

2017-12-04 Thread bruce xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276618#comment-16276618 ] bruce xu commented on SPARK-22365: -- Hi [~dubovsky]. Glad to have your response. I met this issue by

[jira] [Commented] (SPARK-22660) Compile with scala-2.12 and JDK9

2017-12-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276583#comment-16276583 ] Sean Owen commented on SPARK-22660: --- You keep changing what this JIRA is about . There are too many JDK

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2017-12-04 Thread Omer van Kloeten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276569#comment-16276569 ] Omer van Kloeten commented on SPARK-22634: -- Understandable, but since Bouncy Castle may be used

[jira] [Commented] (SPARK-22365) Spark UI executors empty list with 500 error

2017-12-04 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276564#comment-16276564 ] Jakub Dubovsky commented on SPARK-22365: In my instance it looks like it is a result of some

[jira] [Resolved] (SPARK-22670) Not able to create table in HIve with SparkSession when JavaSparkContext is already initialized.

2017-12-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22670. --- Resolution: Not A Problem That's an issue with the design of your app then. > Not able to create

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2017-12-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276556#comment-16276556 ] Sean Owen commented on SPARK-22634: --- I'm hesitant to do that in a maintenance branch because it's a

[jira] [Commented] (SPARK-7953) Spark should cleanup output dir if job fails

2017-12-04 Thread Nandor Kollar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276535#comment-16276535 ] Nandor Kollar commented on SPARK-7953: -- [~joshrosen] could you please help me with this issue, is

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2017-12-04 Thread Omer van Kloeten (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276492#comment-16276492 ] Omer van Kloeten commented on SPARK-22634: -- [~srowen], thanks for taking this up. However, this

[jira] [Updated] (SPARK-22286) OutOfMemoryError caused by memory leak and large serializer batch size in ExternalAppendOnlyMap

2017-12-04 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22286: - Description: *[Abstract]* I recently encountered an OOM error in a simple _groupByKey_ application.

[jira] [Updated] (SPARK-22286) OutOfMemoryError caused by memory leak and large serializer batch size in ExternalAppendOnlyMap

2017-12-04 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22286: - Description: *[Abstract]* I recently encountered an OOM error in a simple _groupByKey_ application.

[jira] [Updated] (SPARK-22675) Refactoring PropagateTypes in TypeCoercion

2017-12-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22675: Description: PropagateTypes are called at the beginning of TypeCocercion and then called at the end of