[jira] [Updated] (SPARK-23705) dataframe.groupBy() may inadvertently receive sequence of non-distinct strings

2018-03-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23705: - Target Version/s: (was: 2.3.0) Let's avoid to set a target version which is usually reserved

[jira] [Commented] (SPARK-23696) StructType.fromString swallows exceptions from DataType.fromJson

2018-03-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401479#comment-16401479 ] Hyukjin Kwon commented on SPARK-23696: -- Why don't we just directly use {{DataType.fromJson}} if you

[jira] [Assigned] (SPARK-23162) PySpark ML LinearRegressionSummary missing r2adj

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23162: Assignee: Apache Spark > PySpark ML LinearRegressionSummary missing r2adj >

[jira] [Assigned] (SPARK-23162) PySpark ML LinearRegressionSummary missing r2adj

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23162: Assignee: (was: Apache Spark) > PySpark ML LinearRegressionSummary missing r2adj >

[jira] [Commented] (SPARK-23162) PySpark ML LinearRegressionSummary missing r2adj

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401458#comment-16401458 ] Apache Spark commented on SPARK-23162: -- User 'kevinyu98' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23706) spark.conf.get(value, default=None) should produce None in PySpark

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23706: Assignee: Apache Spark > spark.conf.get(value, default=None) should produce None in

[jira] [Commented] (SPARK-23706) spark.conf.get(value, default=None) should produce None in PySpark

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401451#comment-16401451 ] Apache Spark commented on SPARK-23706: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-23706) spark.conf.get(value, default=None) should produce None in PySpark

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23706: Assignee: (was: Apache Spark) > spark.conf.get(value, default=None) should produce

[jira] [Created] (SPARK-23706) spark.conf.get(value, default=None) should produce None in PySpark

2018-03-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-23706: Summary: spark.conf.get(value, default=None) should produce None in PySpark Key: SPARK-23706 URL: https://issues.apache.org/jira/browse/SPARK-23706 Project: Spark

[jira] [Created] (SPARK-23705) dataframe.groupBy() may inadvertently receive sequence of non-distinct strings

2018-03-15 Thread Khoa Tran (JIRA)
Khoa Tran created SPARK-23705: - Summary: dataframe.groupBy() may inadvertently receive sequence of non-distinct strings Key: SPARK-23705 URL: https://issues.apache.org/jira/browse/SPARK-23705 Project:

[jira] [Comment Edited] (SPARK-23685) Spark Structured Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2018-03-15 Thread sirisha (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401428#comment-16401428 ] sirisha edited comment on SPARK-23685 at 3/16/18 3:47 AM: -- [~apachespark] Can

[jira] [Commented] (SPARK-23685) Spark Structured Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2018-03-15 Thread sirisha (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401428#comment-16401428 ] sirisha commented on SPARK-23685: - [~apachespark] Can anyone please guide me on how to assign this pull

[jira] [Commented] (SPARK-23673) PySpark dayofweek does not conform with ISO 8601

2018-03-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401392#comment-16401392 ] Kazuaki Ishizaki commented on SPARK-23673: -- In Spark, `dayofweek` comes from SQL. The result

[jira] [Comment Edited] (SPARK-22390) Aggregate push down

2018-03-15 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401378#comment-16401378 ] Huaxin Gao edited comment on SPARK-22390 at 3/16/18 1:56 AM: - [~cloud_fan], I

[jira] [Commented] (SPARK-22390) Aggregate push down

2018-03-15 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401378#comment-16401378 ] Huaxin Gao commented on SPARK-22390: [~cloud_fan], I am working on Aggregate push down design doc and

[jira] [Resolved] (SPARK-23651) Add a check for host name

2018-03-15 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian resolved SPARK-23651. - Resolution: Fixed > Add a check for host name > -- > > Key:

[jira] [Created] (SPARK-23704) PySpark access of individual trees in random forest is slow

2018-03-15 Thread Julian King (JIRA)
Julian King created SPARK-23704: --- Summary: PySpark access of individual trees in random forest is slow Key: SPARK-23704 URL: https://issues.apache.org/jira/browse/SPARK-23704 Project: Spark

[jira] [Assigned] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23670: -- Assignee: Myroslav Lisniak > Memory leak of SparkPlanGraphWrapper in sparkUI >

[jira] [Resolved] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23670. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Assigned] (SPARK-23608) SHS needs synchronization between attachSparkUI and detachSparkUI functions

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23608: -- Assignee: Ye Zhou > SHS needs synchronization between attachSparkUI and detachSparkUI

[jira] [Resolved] (SPARK-23608) SHS needs synchronization between attachSparkUI and detachSparkUI functions

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23608. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Assigned] (SPARK-23671) SHS is ignoring number of replay threads

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23671: -- Assignee: Marcelo Vanzin > SHS is ignoring number of replay threads >

[jira] [Resolved] (SPARK-23671) SHS is ignoring number of replay threads

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23671. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Commented] (SPARK-8008) JDBC data source can overload the external database system due to high concurrency

2018-03-15 Thread Jo Desmet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401300#comment-16401300 ] Jo Desmet commented on SPARK-8008: -- Too bad that this issue is not considered high priority. Too many

[jira] [Created] (SPARK-23703) Collapse sequential watermarks

2018-03-15 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23703: --- Summary: Collapse sequential watermarks Key: SPARK-23703 URL: https://issues.apache.org/jira/browse/SPARK-23703 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-23702) Forbid watermarks on both sides of a streaming aggregate

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23702: Assignee: (was: Apache Spark) > Forbid watermarks on both sides of a streaming

[jira] [Commented] (SPARK-23702) Forbid watermarks on both sides of a streaming aggregate

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401295#comment-16401295 ] Apache Spark commented on SPARK-23702: -- User 'jose-torres' has created a pull request for this

[jira] [Assigned] (SPARK-23702) Forbid watermarks on both sides of a streaming aggregate

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23702: Assignee: Apache Spark > Forbid watermarks on both sides of a streaming aggregate >

[jira] [Resolved] (SPARK-23658) InProcessAppHandle uses the wrong class in getLogger

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23658. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Assigned] (SPARK-23658) InProcessAppHandle uses the wrong class in getLogger

2018-03-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23658: -- Assignee: Sahil Takiar > InProcessAppHandle uses the wrong class in getLogger >

[jira] [Created] (SPARK-23702) Forbid watermarks on both sides of a streaming aggregate

2018-03-15 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23702: --- Summary: Forbid watermarks on both sides of a streaming aggregate Key: SPARK-23702 URL: https://issues.apache.org/jira/browse/SPARK-23702 Project: Spark Issue

[jira] [Created] (SPARK-23701) Multiple sequential watermarks are not supported

2018-03-15 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23701: --- Summary: Multiple sequential watermarks are not supported Key: SPARK-23701 URL: https://issues.apache.org/jira/browse/SPARK-23701 Project: Spark Issue Type:

[jira] [Created] (SPARK-23700) Cleanup unused imports

2018-03-15 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-23700: Summary: Cleanup unused imports Key: SPARK-23700 URL: https://issues.apache.org/jira/browse/SPARK-23700 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-03-15 Thread cclauss (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401252#comment-16401252 ] cclauss commented on SPARK-23698: - A PR to fix for 17 of the 20 issues is at

[jira] [Issue Comment Deleted] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-03-15 Thread cclauss (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cclauss updated SPARK-23698: Comment: was deleted (was: A PR to fix for 17 of the 20 issues is at

[jira] [Assigned] (SPARK-23699) PySpark should raise same Error when Arrow fallback is disabled

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23699: Assignee: Apache Spark > PySpark should raise same Error when Arrow fallback is disabled

[jira] [Assigned] (SPARK-23699) PySpark should raise same Error when Arrow fallback is disabled

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23699: Assignee: (was: Apache Spark) > PySpark should raise same Error when Arrow fallback

[jira] [Commented] (SPARK-23699) PySpark should raise same Error when Arrow fallback is disabled

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401250#comment-16401250 ] Apache Spark commented on SPARK-23699: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23698: Assignee: Apache Spark > Spark code contains numerous undefined names in Python 3 >

[jira] [Assigned] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23698: Assignee: (was: Apache Spark) > Spark code contains numerous undefined names in

[jira] [Commented] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401247#comment-16401247 ] Apache Spark commented on SPARK-23698: -- User 'cclauss' has created a pull request for this issue:

[jira] [Created] (SPARK-23699) PySpark should raise same Error when Arrow fallback is disabled

2018-03-15 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-23699: Summary: PySpark should raise same Error when Arrow fallback is disabled Key: SPARK-23699 URL: https://issues.apache.org/jira/browse/SPARK-23699 Project: Spark

[jira] [Commented] (SPARK-23632) sparkR.session() error with spark packages - JVM is not ready after 10 seconds

2018-03-15 Thread Jaehyeon Kim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401210#comment-16401210 ] Jaehyeon Kim commented on SPARK-23632: -- I've looked into further and found it'd be better to have

[jira] [Created] (SPARK-23698) Spark code contains numerous undefined names in Python 3

2018-03-15 Thread cclauss (JIRA)
cclauss created SPARK-23698: --- Summary: Spark code contains numerous undefined names in Python 3 Key: SPARK-23698 URL: https://issues.apache.org/jira/browse/SPARK-23698 Project: Spark Issue Type:

[jira] [Updated] (SPARK-23697) Accumulators of Spark 1.x no longer work with Spark 2.x

2018-03-15 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-23697: --- Description: I've noticed that accumulators of Spark 1.x no longer work with Spark

[jira] [Updated] (SPARK-23697) Accumulators of Spark 1.x no longer work with Spark 2.x

2018-03-15 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-23697: --- Description: I've noticed that accumulators of Spark 1.x no longer work with Spark

[jira] [Updated] (SPARK-23697) Accumulators of Spark 1.x no longer work with Spark 2.x

2018-03-15 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-23697: --- Description: I've noticed that accumulators of Spark 1.x no longer work with Spark

[jira] [Updated] (SPARK-23697) Accumulators of Spark 1.x no longer work with Spark 2.x

2018-03-15 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-23697: --- Description: I've noticed that accumulators of Spark 1.x no longer work with Spark

[jira] [Created] (SPARK-23697) Accumulators of Spark 1.x no longer work with Spark 2.x

2018-03-15 Thread Sergey Zhemzhitsky (JIRA)
Sergey Zhemzhitsky created SPARK-23697: -- Summary: Accumulators of Spark 1.x no longer work with Spark 2.x Key: SPARK-23697 URL: https://issues.apache.org/jira/browse/SPARK-23697 Project: Spark

[jira] [Resolved] (SPARK-23684) mode append function not working

2018-03-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23684. -- Resolution: Duplicate > mode append function not working > -

[jira] [Commented] (SPARK-7131) Move tree,forest implementation from spark.mllib to spark.ml

2018-03-15 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401071#comment-16401071 ] Joseph K. Bradley commented on SPARK-7131: -- CCing people watching this JIRA about

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2018-03-15 Thread Dylan Guedes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401029#comment-16401029 ] Dylan Guedes commented on SPARK-20169: -- Hi, I also reproduced it in v2.3 and master. I think that

[jira] [Assigned] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23686: Assignee: (was: Apache Spark) > Make better usage of

[jira] [Commented] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400903#comment-16400903 ] Apache Spark commented on SPARK-23686: -- User 'MrBago' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23686: Assignee: Apache Spark > Make better usage of org.apache.spark.ml.util.Instrumentation >

[jira] [Resolved] (SPARK-23695) Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

2018-03-15 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23695. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.1 Issue resolved by

[jira] [Assigned] (SPARK-23695) Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

2018-03-15 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23695: - Assignee: Hyukjin Kwon > Confusing error message for PySpark's Kinesis tests when its

[jira] [Commented] (SPARK-4038) Outlier Detection Algorithm for MLlib

2018-03-15 Thread Gustavo Orair (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400839#comment-16400839 ] Gustavo Orair commented on SPARK-4038: -- There is a paper that discuss multiple different strategies

[jira] [Commented] (SPARK-23684) mode append function not working

2018-03-15 Thread Evan Zamir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400823#comment-16400823 ] Evan Zamir commented on SPARK-23684: Yes, you're right. Feel free to close this. > mode append

[jira] [Commented] (SPARK-23685) Spark Structured Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400755#comment-16400755 ] Apache Spark commented on SPARK-23685: -- User 'sirishaSindri' has created a pull request for this

[jira] [Assigned] (SPARK-23685) Spark Structured Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23685: Assignee: Apache Spark > Spark Structured Streaming Kafka 0.10 Consumer Can't Handle

[jira] [Assigned] (SPARK-23685) Spark Structured Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets (i.e. Log Compaction)

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23685: Assignee: (was: Apache Spark) > Spark Structured Streaming Kafka 0.10 Consumer Can't

[jira] [Created] (SPARK-23696) StructType.fromString swallows exceptions from DataType.fromJson

2018-03-15 Thread Simeon H.K. Fitch (JIRA)
Simeon H.K. Fitch created SPARK-23696: - Summary: StructType.fromString swallows exceptions from DataType.fromJson Key: SPARK-23696 URL: https://issues.apache.org/jira/browse/SPARK-23696 Project:

[jira] [Updated] (SPARK-23693) SQL function uuid()

2018-03-15 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-23693: - Description: Add function uuid() to org.apache.spark.sql.functions that returns

[jira] [Updated] (SPARK-23693) SQL function uuid()

2018-03-15 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-23693: - Description: Add function uuid() to org.apache.spark.sql.functions that returns

[jira] [Commented] (SPARK-23695) Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400325#comment-16400325 ] Apache Spark commented on SPARK-23695: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-23695) Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23695: Assignee: Apache Spark > Confusing error message for PySpark's Kinesis tests when its jar

[jira] [Assigned] (SPARK-23695) Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23695: Assignee: (was: Apache Spark) > Confusing error message for PySpark's Kinesis tests

[jira] [Created] (SPARK-23695) Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled

2018-03-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-23695: Summary: Confusing error message for PySpark's Kinesis tests when its jar is missing but enabled Key: SPARK-23695 URL: https://issues.apache.org/jira/browse/SPARK-23695

[jira] [Created] (SPARK-23694) The staging directory should under hive.exec.stagingdir if we set hive.exec.stagingdir but not under the table directory

2018-03-15 Thread Yifeng Dong (JIRA)
Yifeng Dong created SPARK-23694: --- Summary: The staging directory should under hive.exec.stagingdir if we set hive.exec.stagingdir but not under the table directory Key: SPARK-23694 URL:

[jira] [Updated] (SPARK-23693) SQL function uuid()

2018-03-15 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-23693: - Description: Add function uuid() to org.apache.spark.sql.functions that returns

[jira] [Updated] (SPARK-23693) SQL function uuid()

2018-03-15 Thread Arseniy Tashoyan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arseniy Tashoyan updated SPARK-23693: - Description: Add function uuid() to org.apache.spark.sql.functions that returns

[jira] [Created] (SPARK-23693) SQL function uuid()

2018-03-15 Thread Arseniy Tashoyan (JIRA)
Arseniy Tashoyan created SPARK-23693: Summary: SQL function uuid() Key: SPARK-23693 URL: https://issues.apache.org/jira/browse/SPARK-23693 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23683) FileCommitProtocol.instantiate to require 3-arg constructor for dynamic partition overwrite

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400177#comment-16400177 ] Apache Spark commented on SPARK-23683: -- User 'steveloughran' has created a pull request for this

[jira] [Assigned] (SPARK-23683) FileCommitProtocol.instantiate to require 3-arg constructor for dynamic partition overwrite

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23683: Assignee: Apache Spark > FileCommitProtocol.instantiate to require 3-arg constructor for

[jira] [Assigned] (SPARK-23683) FileCommitProtocol.instantiate to require 3-arg constructor for dynamic partition overwrite

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23683: Assignee: (was: Apache Spark) > FileCommitProtocol.instantiate to require 3-arg

[jira] [Commented] (SPARK-23692) Print metadata of files when infer schema failed

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400106#comment-16400106 ] Apache Spark commented on SPARK-23692: -- User 'caneGuy' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23692) Print metadata of files when infer schema failed

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23692: Assignee: (was: Apache Spark) > Print metadata of files when infer schema failed >

[jira] [Created] (SPARK-23692) Print metadata of files when infer schema failed

2018-03-15 Thread zhoukang (JIRA)
zhoukang created SPARK-23692: Summary: Print metadata of files when infer schema failed Key: SPARK-23692 URL: https://issues.apache.org/jira/browse/SPARK-23692 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-23692) Print metadata of files when infer schema failed

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23692: Assignee: Apache Spark > Print metadata of files when infer schema failed >

[jira] [Commented] (SPARK-20536) Extend ColumnName to create StructFields with explicit nullable

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16400021#comment-16400021 ] Apache Spark commented on SPARK-20536: -- User 'efimpoberezkin' has created a pull request for this

[jira] [Assigned] (SPARK-20536) Extend ColumnName to create StructFields with explicit nullable

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20536: Assignee: (was: Apache Spark) > Extend ColumnName to create StructFields with

[jira] [Assigned] (SPARK-20536) Extend ColumnName to create StructFields with explicit nullable

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20536: Assignee: Apache Spark > Extend ColumnName to create StructFields with explicit nullable

[jira] [Assigned] (SPARK-23533) Add support for changing ContinuousDataReader's startOffset

2018-03-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-23533: Assignee: Li Yuanjian > Add support for changing ContinuousDataReader's startOffset >

[jira] [Resolved] (SPARK-23533) Add support for changing ContinuousDataReader's startOffset

2018-03-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23533. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20689

[jira] [Updated] (SPARK-23614) Union produces incorrect results when caching is used

2018-03-15 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-23614: Component/s: (was: Spark Core) SQL > Union produces incorrect results

[jira] [Assigned] (SPARK-23614) Union produces incorrect results when caching is used

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23614: Assignee: (was: Apache Spark) > Union produces incorrect results when caching is used

[jira] [Assigned] (SPARK-23614) Union produces incorrect results when caching is used

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23614: Assignee: Apache Spark > Union produces incorrect results when caching is used >

[jira] [Commented] (SPARK-23614) Union produces incorrect results when caching is used

2018-03-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399985#comment-16399985 ] Apache Spark commented on SPARK-23614: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-23677) Selecting columns from joined DataFrames with the same origin yields wrong results

2018-03-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399969#comment-16399969 ] Takeshi Yamamuro commented on SPARK-23677: -- You mean this ticket? SPARK-14948. I think this is a