[jira] [Resolved] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-06-06 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-24279. - Resolution: Invalid > Incompatible byte code errors, when using test-jar of spark sql.

[jira] [Assigned] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24477: Assignee: (was: Apache Spark) > Import submodules under pyspark.ml by default >

[jira] [Assigned] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24477: Assignee: Apache Spark > Import submodules under pyspark.ml by default >

[jira] [Commented] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504247#comment-16504247 ] Apache Spark commented on SPARK-24477: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-06-06 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504246#comment-16504246 ] Prashant Sharma commented on SPARK-24279: - Thanks a lot, that was the mistake.  > Incompatible

[jira] [Resolved] (SPARK-24475) Nested JSON count() Exception

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24475. -- Resolution: Duplicate > Nested JSON count() Exception > - > >

[jira] [Commented] (SPARK-24475) Nested JSON count() Exception

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504235#comment-16504235 ] Hyukjin Kwon commented on SPARK-24475: -- I don't think Spark currently support Java 9 and 10 yet.

[jira] [Comment Edited] (SPARK-24447) Pyspark RowMatrix.columnSimilarities() loses spark context

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504213#comment-16504213 ] Liang-Chi Hsieh edited comment on SPARK-24447 at 6/7/18 4:09 AM: - I just

[jira] [Commented] (SPARK-24447) Pyspark RowMatrix.columnSimilarities() loses spark context

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504213#comment-16504213 ] Liang-Chi Hsieh commented on SPARK-24447: - I just build Spark from current 2.3 branch. The above

[jira] [Commented] (SPARK-24431) wrong areaUnderPR calculation in BinaryClassificationEvaluator

2018-06-06 Thread Xinyong Tian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504202#comment-16504202 ] Xinyong Tian commented on SPARK-24431: -- I also feel it is reasonable to set first point as (0,p).

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-06-06 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504171#comment-16504171 ] Li Yuanjian commented on SPARK-24375: - Got it, great thanks for your detailed explanation. > Design

[jira] [Commented] (SPARK-24475) Nested JSON count() Exception

2018-06-06 Thread Joseph Toth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504163#comment-16504163 ] Joseph Toth commented on SPARK-24475: - It looks like it was my version of java on this machine. It

[jira] [Commented] (SPARK-24447) Pyspark RowMatrix.columnSimilarities() loses spark context

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504152#comment-16504152 ] Liang-Chi Hsieh commented on SPARK-24447: - Yes, I can run the example code on a build from

[jira] [Commented] (SPARK-24475) Nested JSON count() Exception

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504134#comment-16504134 ] Hyukjin Kwon commented on SPARK-24475: -- I can't reproduce this. What's your version and env? does

[jira] [Updated] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24479: - Affects Version/s: 2.4.0 > Register StreamingQueryListener in Spark Conf >

[jira] [Commented] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504121#comment-16504121 ] Hyukjin Kwon commented on SPARK-24480: -- (let's avoid to set the fix version which is usually set

[jira] [Updated] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24480: - Fix Version/s: (was: 2.4.0) > Add a config to register custom StreamingQueryListeners >

[jira] [Commented] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-06 Thread Arun Mahadevan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504120#comment-16504120 ] Arun Mahadevan commented on SPARK-24479: PR raised - https://github.com/apache/spark/pull/21504

[jira] [Assigned] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24479: Assignee: Apache Spark > Register StreamingQueryListener in Spark Conf >

[jira] [Commented] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504119#comment-16504119 ] Apache Spark commented on SPARK-24479: -- User 'arunmahadevan' has created a pull request for this

[jira] [Assigned] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24479: Assignee: (was: Apache Spark) > Register StreamingQueryListener in Spark Conf >

[jira] [Resolved] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Arun Mahadevan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Mahadevan resolved SPARK-24480. Resolution: Duplicate Issue is already raised. Duplicate of SPARK-24479 > Add a config

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Environment: Emr 5.13.0 and Databricks Cloud 4.0 (was: Emr 5.13.0) >

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Description: Similar to other "grows beyond 64 KB" errors.  Happens with large case

[jira] [Updated] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Conegliano updated SPARK-24481: -- Attachment: log4j-active(1).log > GeneratedIteratorForCodegenStage1 grows beyond 64

[jira] [Assigned] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24480: Assignee: (was: Apache Spark) > Add a config to register custom

[jira] [Commented] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504113#comment-16504113 ] Apache Spark commented on SPARK-24480: -- User 'arunmahadevan' has created a pull request for this

[jira] [Assigned] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24480: Assignee: Apache Spark > Add a config to register custom StreamingQueryListeners >

[jira] [Created] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-06 Thread Andrew Conegliano (JIRA)
Andrew Conegliano created SPARK-24481: - Summary: GeneratedIteratorForCodegenStage1 grows beyond 64 KB Key: SPARK-24481 URL: https://issues.apache.org/jira/browse/SPARK-24481 Project: Spark

[jira] [Created] (SPARK-24480) Add a config to register custom StreamingQueryListeners

2018-06-06 Thread Arun Mahadevan (JIRA)
Arun Mahadevan created SPARK-24480: -- Summary: Add a config to register custom StreamingQueryListeners Key: SPARK-24480 URL: https://issues.apache.org/jira/browse/SPARK-24480 Project: Spark

[jira] [Created] (SPARK-24479) Register StreamingQueryListener in Spark Conf

2018-06-06 Thread Mingjie Tang (JIRA)
Mingjie Tang created SPARK-24479: Summary: Register StreamingQueryListener in Spark Conf Key: SPARK-24479 URL: https://issues.apache.org/jira/browse/SPARK-24479 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-24478) DataSourceV2 should push filters and projection at physical plan conversion

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24478: Assignee: (was: Apache Spark) > DataSourceV2 should push filters and projection at

[jira] [Commented] (SPARK-24478) DataSourceV2 should push filters and projection at physical plan conversion

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503874#comment-16503874 ] Apache Spark commented on SPARK-24478: -- User 'rdblue' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24478) DataSourceV2 should push filters and projection at physical plan conversion

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24478: Assignee: Apache Spark > DataSourceV2 should push filters and projection at physical

[jira] [Commented] (SPARK-24357) createDataFrame in Python infers large integers as long type and then fails silently when converting them

2018-06-06 Thread Joel Croteau (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503835#comment-16503835 ] Joel Croteau commented on SPARK-24357: -- [~viirya], yes that's what I said. What I am saying is that

[jira] [Created] (SPARK-24478) DataSourceV2 should push filters and projection at physical plan conversion

2018-06-06 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-24478: - Summary: DataSourceV2 should push filters and projection at physical plan conversion Key: SPARK-24478 URL: https://issues.apache.org/jira/browse/SPARK-24478 Project: Spark

[jira] [Commented] (SPARK-24469) Support collations in Spark SQL

2018-06-06 Thread Alexander Shkapsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503723#comment-16503723 ] Alexander Shkapsky commented on SPARK-24469: *first* or *min* on a StringType column gets

[jira] [Commented] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503710#comment-16503710 ] Hyukjin Kwon commented on SPARK-24477: -- Thanks for cc'ing me, [~mengxr]. Will do this too tomorrow.

[jira] [Updated] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24477: -- Description: Right now, we do not import submodules under pyspark.ml by default. So users

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Issue Type: Improvement (was: Bug) > ml.image doesn't have __all__ explicitly defined >

[jira] [Commented] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503700#comment-16503700 ] Xiangrui Meng commented on SPARK-24454: --- Updated this JIRA and created SPARK-24477. > ml.image

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Priority: Minor (was: Major) > ml.image doesn't have __all__ explicitly defined >

[jira] [Created] (SPARK-24477) Import submodules under pyspark.ml by default

2018-06-06 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24477: - Summary: Import submodules under pyspark.ml by default Key: SPARK-24477 URL: https://issues.apache.org/jira/browse/SPARK-24477 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Description: ml/image.py doesn't have __all__ explicitly defined. It will import all global

[jira] [Updated] (SPARK-24454) ml.image doesn't have __all__ explicitly defined

2018-06-06 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24454: -- Summary: ml.image doesn't have __all__ explicitly defined (was: ml.image doesn't have

[jira] [Commented] (SPARK-24469) Support collations in Spark SQL

2018-06-06 Thread Eric Maynard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503687#comment-16503687 ] Eric Maynard commented on SPARK-24469: -- Ah, I see, I was wrongly thinking of the second case where

[jira] [Commented] (SPARK-24447) Pyspark RowMatrix.columnSimilarities() loses spark context

2018-06-06 Thread Perry Chu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503678#comment-16503678 ] Perry Chu commented on SPARK-24447: --- Do you mean building from source code? I haven't done that

[jira] [Updated] (SPARK-24466) TextSocketMicroBatchReader no longer works with nc utility

2018-06-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-24466: - Target Version/s: 2.4.0 > TextSocketMicroBatchReader no longer works with nc utility >

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-06-06 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503647#comment-16503647 ] Jiang Xingbo commented on SPARK-24375: -- The major problem is that tasks in the same stage of a MPI

[jira] [Commented] (SPARK-24279) Incompatible byte code errors, when using test-jar of spark sql.

2018-06-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503630#comment-16503630 ] Shixiong Zhu commented on SPARK-24279: -- Just did a quick look at your pom.xml. I think it's missing

[jira] [Commented] (SPARK-24469) Support collations in Spark SQL

2018-06-06 Thread Alexander Shkapsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503621#comment-16503621 ] Alexander Shkapsky commented on SPARK-24469: A simple case with a single input value "George

[jira] [Commented] (SPARK-24469) Support collations in Spark SQL

2018-06-06 Thread Eric Maynard (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503586#comment-16503586 ] Eric Maynard commented on SPARK-24469: -- bq. SELECT UPPER(text)GROUP BY UPPER(text) bq.

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0

2018-06-06 Thread bharath kumar avusherla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath kumar avusherla updated SPARK-24476: Description: We are working on spark streaming application using spark

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0

2018-06-06 Thread bharath kumar avusherla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath kumar avusherla updated SPARK-24476: Description: We are working on spark streaming application using spark

[jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0

2018-06-06 Thread bharath kumar avusherla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath kumar avusherla updated SPARK-24476: Attachment: socket-timeout-exception > java.net.SocketTimeoutException:

[jira] [Created] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0

2018-06-06 Thread bharath kumar avusherla (JIRA)
bharath kumar avusherla created SPARK-24476: --- Summary: java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0 Key: SPARK-24476 URL:

[jira] [Created] (SPARK-24475) Nested JSON count() Exception

2018-06-06 Thread Joseph Toth (JIRA)
Joseph Toth created SPARK-24475: --- Summary: Nested JSON count() Exception Key: SPARK-24475 URL: https://issues.apache.org/jira/browse/SPARK-24475 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-06-06 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503486#comment-16503486 ] Li Yuanjian edited comment on SPARK-24375 at 6/6/18 3:55 PM: - Hi

[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark

2018-06-06 Thread Li Yuanjian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503486#comment-16503486 ] Li Yuanjian commented on SPARK-24375: - Hi [~cloud_fan] and [~jiangxb1987], just I tiny question

[jira] [Commented] (SPARK-24431) wrong areaUnderPR calculation in BinaryClassificationEvaluator

2018-06-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503466#comment-16503466 ] Sean Owen commented on SPARK-24431: --- So, the model makes the same prediction p for all examples? In

[jira] [Commented] (SPARK-24472) Orc RecordReaderFactory throws IndexOutOfBoundsException

2018-06-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503445#comment-16503445 ] Dongjoon Hyun commented on SPARK-24472: --- Thank you for pinging me, [~zsxwing] > Orc

[jira] [Updated] (SPARK-24472) Orc RecordReaderFactory throws IndexOutOfBoundsException

2018-06-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24472: -- Affects Version/s: 1.6.3 2.0.2 2.1.2

[jira] [Commented] (SPARK-22575) Making Spark Thrift Server clean up its cache

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503394#comment-16503394 ] Apache Spark commented on SPARK-22575: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22575) Making Spark Thrift Server clean up its cache

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22575: Assignee: (was: Apache Spark) > Making Spark Thrift Server clean up its cache >

[jira] [Assigned] (SPARK-22575) Making Spark Thrift Server clean up its cache

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22575: Assignee: Apache Spark > Making Spark Thrift Server clean up its cache >

[jira] [Assigned] (SPARK-23803) Support bucket pruning to optimize filtering on a bucketed column

2018-06-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23803: --- Assignee: Asher Saban > Support bucket pruning to optimize filtering on a bucketed column

[jira] [Resolved] (SPARK-23803) Support bucket pruning to optimize filtering on a bucketed column

2018-06-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23803. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20915

[jira] [Comment Edited] (SPARK-15882) Discuss distributed linear algebra in spark.ml package

2018-06-06 Thread Kyle Prifogle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503256#comment-16503256 ] Kyle Prifogle edited comment on SPARK-15882 at 6/6/18 1:00 PM: ---

[jira] [Commented] (SPARK-15882) Discuss distributed linear algebra in spark.ml package

2018-06-06 Thread Kyle Prifogle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503256#comment-16503256 ] Kyle Prifogle commented on SPARK-15882: --- Noticing this is almost 2 years old now which gives me

[jira] [Resolved] (SPARK-24471) MlLib distributed plans

2018-06-06 Thread Kyle Prifogle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Prifogle resolved SPARK-24471. --- Resolution: Duplicate > MlLib distributed plans > --- > >

[jira] [Commented] (SPARK-24471) MlLib distributed plans

2018-06-06 Thread Kyle Prifogle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503251#comment-16503251 ] Kyle Prifogle commented on SPARK-24471: --- This is what I was looking for:

[jira] [Updated] (SPARK-24474) Cores are left idle when there are a lot of stages to run

2018-06-06 Thread Al M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Al M updated SPARK-24474: - Description: I've observed an issue happening consistently when: * A job contains a join of two datasets *

[jira] [Commented] (SPARK-24474) Cores are left idle when there are a lot of stages to run

2018-06-06 Thread Al M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503225#comment-16503225 ] Al M commented on SPARK-24474: -- I appreciate that 2.2.0 is slightly old but I couldn't see any scheduler

[jira] [Updated] (SPARK-24474) Cores are left idle when there are a lot of stages to run

2018-06-06 Thread Al M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Al M updated SPARK-24474: - Description: I've observed an issue happening consistently when: * A job contains a join of two datasets *

[jira] [Commented] (SPARK-24431) wrong areaUnderPR calculation in BinaryClassificationEvaluator

2018-06-06 Thread Teng Peng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503217#comment-16503217 ] Teng Peng commented on SPARK-24431: --- [~Ben2018] The article makes sense to me. It seems the current

[jira] [Created] (SPARK-24474) Cores are left idle when there are a lot of stages to run

2018-06-06 Thread Al M (JIRA)
Al M created SPARK-24474: Summary: Cores are left idle when there are a lot of stages to run Key: SPARK-24474 URL: https://issues.apache.org/jira/browse/SPARK-24474 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-24474) Cores are left idle when there are a lot of stages to run

2018-06-06 Thread Al M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Al M updated SPARK-24474: - Description: I've observed an issue happening consistently when: * A job contains a join of two datasets *

[jira] [Commented] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503150#comment-16503150 ] Apache Spark commented on SPARK-21687: -- User 'debugger87' has created a pull request for this

[jira] [Assigned] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21687: Assignee: (was: Apache Spark) > Spark SQL should set createTime for Hive partition >

[jira] [Assigned] (SPARK-21687) Spark SQL should set createTime for Hive partition

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21687: Assignee: Apache Spark > Spark SQL should set createTime for Hive partition >

[jira] [Comment Edited] (SPARK-24357) createDataFrame in Python infers large integers as long type and then fails silently when converting them

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503065#comment-16503065 ] Liang-Chi Hsieh edited comment on SPARK-24357 at 6/6/18 9:50 AM: - I

[jira] [Commented] (SPARK-24357) createDataFrame in Python infers large integers as long type and then fails silently when converting them

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503065#comment-16503065 ] Liang-Chi Hsieh commented on SPARK-24357: - I think this is because this number {{1 << 65}}

[jira] [Created] (SPARK-24473) It is no need to clip the predictive value by maxValue and minValue when computing gradient on SVDplusplus model

2018-06-06 Thread caijianming (JIRA)
caijianming created SPARK-24473: --- Summary: It is no need to clip the predictive value by maxValue and minValue when computing gradient on SVDplusplus model Key: SPARK-24473 URL:

[jira] [Commented] (SPARK-24467) VectorAssemblerEstimator

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503002#comment-16503002 ] Liang-Chi Hsieh commented on SPARK-24467: - [~josephkb] Does that mean {{VectorAssembler}} will

[jira] [Commented] (SPARK-15064) Locale support in StopWordsRemover

2018-06-06 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502929#comment-16502929 ] yuhao yang commented on SPARK-15064: Yuhao will be OOF from May 29th to June 6th (annual leave and

[jira] [Commented] (SPARK-15064) Locale support in StopWordsRemover

2018-06-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502928#comment-16502928 ] Apache Spark commented on SPARK-15064: -- User 'dongjinleekr' has created a pull request for this

[jira] [Commented] (SPARK-24472) Orc RecordReaderFactory throws IndexOutOfBoundsException

2018-06-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502914#comment-16502914 ] Shixiong Zhu commented on SPARK-24472: -- cc [~dongjoon] > Orc RecordReaderFactory throws

[jira] [Closed] (SPARK-24455) fix typo in TaskSchedulerImpl's comments

2018-06-06 Thread xueyu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xueyu closed SPARK-24455. - > fix typo in TaskSchedulerImpl's comments > > > Key:

[jira] [Commented] (SPARK-24447) Pyspark RowMatrix.columnSimilarities() loses spark context

2018-06-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502876#comment-16502876 ] Liang-Chi Hsieh commented on SPARK-24447: - I can't reproduce this in current master branch. Can