[jira] [Commented] (SPARK-15505) Explode nested Array in DF Column into Multiple Columns

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839386#comment-15839386 ] Hyukjin Kwon commented on SPARK-15505: -- Ah, then, we should calculate the maximum length of that

[jira] [Assigned] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15463: Assignee: (was: Apache Spark) > Support for creating a dataframe from CSV in

[jira] [Assigned] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15463: Assignee: Apache Spark > Support for creating a dataframe from CSV in Dataset[String] >

[jira] [Reopened] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-15463: -- I am reopening this as I feel we need this per the issues

[jira] [Commented] (SPARK-15505) Explode nested Array in DF Column into Multiple Columns

2017-01-25 Thread Jorge Machado (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839365#comment-15839365 ] Jorge Machado commented on SPARK-15505: --- [~hyukjin.kwon] you don't really always know if you have

[jira] [Commented] (SPARK-19333) Files out of compliance with ASF policy

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839349#comment-15839349 ] Apache Spark commented on SPARK-19333: -- User 'felixcheung' has created a pull request for this

[jira] [Assigned] (SPARK-19333) Files out of compliance with ASF policy

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19333: Assignee: Apache Spark (was: Felix Cheung) > Files out of compliance with ASF policy >

[jira] [Assigned] (SPARK-19333) Files out of compliance with ASF policy

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19333: Assignee: Felix Cheung (was: Apache Spark) > Files out of compliance with ASF policy >

[jira] [Commented] (SPARK-15809) PySpark SQL UDF default returnType

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839348#comment-15839348 ] Hyukjin Kwon commented on SPARK-15809: -- I don't think it is worth to do this with breaking the API

[jira] [Closed] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2017-01-25 Thread Xin Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Wu closed SPARK-15463. -- Resolution: Later > Support for creating a dataframe from CSV in Dataset[String] >

[jira] [Updated] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-19366: - Component/s: (was: Spark Core) SQL > Dataset should have getNumPartitions

[jira] [Assigned] (SPARK-19338) Always Identical Name for UDF in the EXPLAIN output

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19338: Assignee: (was: Apache Spark) > Always Identical Name for UDF in the EXPLAIN output

[jira] [Commented] (SPARK-19338) Always Identical Name for UDF in the EXPLAIN output

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839309#comment-15839309 ] Apache Spark commented on SPARK-19338: -- User 'maropu' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19338) Always Identical Name for UDF in the EXPLAIN output

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19338: Assignee: Apache Spark > Always Identical Name for UDF in the EXPLAIN output >

[jira] [Assigned] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19366: Assignee: Felix Cheung (was: Apache Spark) > Dataset should have getNumPartitions method

[jira] [Commented] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839308#comment-15839308 ] Apache Spark commented on SPARK-19366: -- User 'felixcheung' has created a pull request for this

[jira] [Assigned] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19366: Assignee: Apache Spark (was: Felix Cheung) > Dataset should have getNumPartitions method

[jira] [Commented] (SPARK-11620) parquet.hadoop.ParquetOutputCommitter.commitJob() throws parquet.io.ParquetEncodingException

2017-01-25 Thread Swaranga Sarma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839306#comment-15839306 ] Swaranga Sarma commented on SPARK-11620: I encountered this issue in Spark 2.0.2 >

[jira] [Assigned] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung reassigned SPARK-19366: Assignee: Felix Cheung > Dataset should have getNumPartitions method >

[jira] [Updated] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-19366: - Description: This would avoid inefficiency in converting Dataset/DataFrame into RDD in non-JVM

[jira] [Created] (SPARK-19366) Dataset should have getNumPartitions method

2017-01-25 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-19366: Summary: Dataset should have getNumPartitions method Key: SPARK-19366 URL: https://issues.apache.org/jira/browse/SPARK-19366 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-15505) Explode nested Array in DF Column into Multiple Columns

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839280#comment-15839280 ] Hyukjin Kwon commented on SPARK-15505: -- Can we just {{df.selectExpr("Col1", "Col2[0]", "Col2[1]",

[jira] [Updated] (SPARK-19365) Optimize RequestMessage serialization

2017-01-25 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19365: - Description: Right now Netty PRC serializes RequestMessage using Java serialization, and the

[jira] [Commented] (SPARK-19365) Optimize RequestMessage serialization

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839030#comment-15839030 ] Apache Spark commented on SPARK-19365: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19365) Optimize RequestMessage serialization

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19365: Assignee: Apache Spark (was: Shixiong Zhu) > Optimize RequestMessage serialization >

[jira] [Assigned] (SPARK-19365) Optimize RequestMessage serialization

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19365: Assignee: Shixiong Zhu (was: Apache Spark) > Optimize RequestMessage serialization >

[jira] [Created] (SPARK-19365) Optimize RequestMessage serialization

2017-01-25 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-19365: Summary: Optimize RequestMessage serialization Key: SPARK-19365 URL: https://issues.apache.org/jira/browse/SPARK-19365 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2017-01-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18020: Assignee: Takeshi Yamamuro > Kinesis receiver does not snapshot when shard completes >

[jira] [Updated] (SPARK-14804) Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:

2017-01-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-14804: -- Assignee: Tathagata Das > Graph vertexRDD/EdgeRDD checkpoint results

[jira] [Commented] (SPARK-17975) EMLDAOptimizer fails with ClassCastException on YARN

2017-01-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838995#comment-15838995 ] Joseph K. Bradley commented on SPARK-17975: --- [SPARK-14804] was just fixed. [~jvstein], do you

[jira] [Resolved] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2017-01-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-18020. - Resolution: Fixed Fix Version/s: 2.2.0 Resolved by

[jira] [Commented] (SPARK-17265) EdgeRDD Difference throws an exception

2017-01-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838990#comment-15838990 ] Joseph K. Bradley commented on SPARK-17265: --- [SPARK-14804] was just fixed. [~shishir167], are

[jira] [Comment Edited] (SPARK-17265) EdgeRDD Difference throws an exception

2017-01-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838990#comment-15838990 ] Joseph K. Bradley edited comment on SPARK-17265 at 1/26/17 1:41 AM:

[jira] [Commented] (SPARK-17877) Can not checkpoint connectedComponents resulting graph

2017-01-25 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838993#comment-15838993 ] Joseph K. Bradley commented on SPARK-17877: --- [SPARK-14804] was just fixed. [~apivovarov], are

[jira] [Resolved] (SPARK-18495) Web UI should document meaning of green dot in DAG visualization

2017-01-25 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-18495. - Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0 > Web UI should

[jira] [Resolved] (SPARK-14804) Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:

2017-01-25 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-14804. --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.3

[jira] [Commented] (SPARK-19338) Always Identical Name for UDF in the EXPLAIN output

2017-01-25 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838893#comment-15838893 ] Takeshi Yamamuro commented on SPARK-19338: -- okay, I'll do! > Always Identical Name for UDF in

[jira] [Commented] (SPARK-7768) Make user-defined type (UDT) API public

2017-01-25 Thread Randall Whitman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838812#comment-15838812 ] Randall Whitman commented on SPARK-7768: OK, UserDefinedType with UdtRegistration should support

[jira] [Updated] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow

2017-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-18750: --- Fix Version/s: 2.0.3 > spark should be able to control the number of executor and should not

[jira] [Commented] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%

2017-01-25 Thread Sven Krasser (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838681#comment-15838681 ] Sven Krasser commented on SPARK-4049: - [~srowen], see my comment from before: {quote} As a user, when

[jira] [Commented] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%

2017-01-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838649#comment-15838649 ] Sean Owen commented on SPARK-4049: -- Data can be cached in multiple locations, so it makes some sense to

[jira] [Assigned] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19354: Assignee: Apache Spark > Killed tasks are getting marked as FAILED >

[jira] [Assigned] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19354: Assignee: (was: Apache Spark) > Killed tasks are getting marked as FAILED >

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838642#comment-15838642 ] Apache Spark commented on SPARK-19354: -- User 'devaraj-kavali' has created a pull request for this

[jira] [Commented] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%

2017-01-25 Thread Barry Becker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838617#comment-15838617 ] Barry Becker commented on SPARK-4049: - I read the comments, but I'm still not really sure what over

[jira] [Resolved] (SPARK-19307) SPARK-17387 caused ignorance of conf object passed to SparkContext:

2017-01-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-19307. Resolution: Fixed Assignee: Marcelo Vanzin Target Version/s: 2.1.1,

[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-01-25 Thread Devaraj K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838396#comment-15838396 ] Devaraj K commented on SPARK-19354: --- bq. The question is, why the error during shutdown? The shutdown

[jira] [Commented] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838390#comment-15838390 ] Apache Spark commented on SPARK-18750: -- User 'vanzin' has created a pull request for this issue:

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.1.0

2017-01-25 Thread Helena Edelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838274#comment-15838274 ] Helena Edelson commented on SPARK-18057: Thanks [~c...@koeninger.org]. Chatting with [~tdas]

[jira] [Commented] (SPARK-19363) order by cannot be parsed when group by is missing

2017-01-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838257#comment-15838257 ] Dongjoon Hyun commented on SPARK-19363: --- Thank you for closing, [~mkolev] > order by cannot be

[jira] [Comment Edited] (SPARK-19363) order by cannot be parsed when group by is missing

2017-01-25 Thread Mitko Kolev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838252#comment-15838252 ] Mitko Kolev edited comment on SPARK-19363 at 1/25/17 6:03 PM: -- Hi Dongjoon,

[jira] [Closed] (SPARK-19363) order by cannot be parsed when group by is missing

2017-01-25 Thread Mitko Kolev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitko Kolev closed SPARK-19363. --- Resolution: Not A Problem Hi Hyun, thanks, it is my mistake, sorry for opening an issue. Best

[jira] [Updated] (SPARK-19364) Some Stream Blocks in Storage Persists Forever

2017-01-25 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Summary: Some Stream Blocks in Storage Persists Forever (was: Some Blocks in Storage

[jira] [Created] (SPARK-19364) Some Blocks in Storage Persists Forever

2017-01-25 Thread Andrew Milkowski (JIRA)
Andrew Milkowski created SPARK-19364: Summary: Some Blocks in Storage Persists Forever Key: SPARK-19364 URL: https://issues.apache.org/jira/browse/SPARK-19364 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19364) Some Blocks in Storage Persists Forever

2017-01-25 Thread Andrew Milkowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Milkowski updated SPARK-19364: - Environment: ubuntu unix spark 2.0.2 application is java was: ubuntu unix spar 2.0.2

[jira] [Commented] (SPARK-19363) order by cannot be parsed when group by is missing

2017-01-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838222#comment-15838222 ] Dongjoon Hyun commented on SPARK-19363: --- Hi, [~me2stk]. For me, it works. Could you provide a more

[jira] [Created] (SPARK-19363) order by cannot be parsed when group by is missing

2017-01-25 Thread Mitko Kolev (JIRA)
Mitko Kolev created SPARK-19363: --- Summary: order by cannot be parsed when group by is missing Key: SPARK-19363 URL: https://issues.apache.org/jira/browse/SPARK-19363 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-14955) JDBCRelation should report an IllegalArgumentException if stride equals 0

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14955. -- Resolution: Duplicate > JDBCRelation should report an IllegalArgumentException if stride

[jira] [Updated] (SPARK-19311) UDFs disregard UDT type hierarchy

2017-01-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19311: Assignee: Gregor Moehler > UDFs disregard UDT type hierarchy > - > >

[jira] [Resolved] (SPARK-19311) UDFs disregard UDT type hierarchy

2017-01-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-19311. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > UDFs disregard UDT type

[jira] [Commented] (SPARK-19338) Always Identical Name for UDF in the EXPLAIN output

2017-01-25 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838033#comment-15838033 ] Xiao Li commented on SPARK-19338: - Yes. I am not working on this. Please submit it. Thanks! > Always

[jira] [Resolved] (SPARK-18863) Output non-aggregate expressions without GROUP BY in a subquery does not yield an error

2017-01-25 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-18863. --- Resolution: Fixed Assignee: Nattavut Sutyanyong Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-14165) NoSuchElementException: None.get when joining DataFrames with Seq of fields of different case

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14165. -- Resolution: Fixed Ah, thanks. Let me then resolve it. > NoSuchElementException: None.get

[jira] [Commented] (SPARK-13637) use more information to simplify the code in Expand builder

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837913#comment-15837913 ] Hyukjin Kwon commented on SPARK-13637: -- ([~cloud_fan] it seems this one is mistakenly not resolved)

[jira] [Resolved] (SPARK-13316) "SparkException: DStream has not been initialized" when restoring StreamingContext from checkpoint and the dstream is created afterwards

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13316. -- Resolution: Not A Problem I tried to reproduce this as below: {code} nc -lk {code}

[jira] [Resolved] (SPARK-19313) GaussianMixture throws cryptic error when number of features is too high

2017-01-25 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-19313. - Resolution: Fixed Assignee: Seth Hendrickson Fix Version/s: 2.2.0 >

[jira] [Commented] (SPARK-12970) Error in documentation on creating rows with schemas defined by structs

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837871#comment-15837871 ] Apache Spark commented on SPARK-12970: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-12970) Error in documentation on creating rows with schemas defined by structs

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12970: Assignee: (was: Apache Spark) > Error in documentation on creating rows with schemas

[jira] [Assigned] (SPARK-12970) Error in documentation on creating rows with schemas defined by structs

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-12970: Assignee: Apache Spark > Error in documentation on creating rows with schemas defined by

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.1.0

2017-01-25 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837844#comment-15837844 ] Cody Koeninger commented on SPARK-18057: If you can get commiter agreement on the outstanding

[jira] [Resolved] (SPARK-10924) Failed to update accumulators for ShuffleMapTask: Broken pipe

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-10924. -- Resolution: Cannot Reproduce Thank you so much for checking this. Let me then resolve this as

[jira] [Resolved] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow

2017-01-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18750. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.1.0

2017-01-25 Thread Helena Edelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837791#comment-15837791 ] Helena Edelson commented on SPARK-18057: I'd tried this upgrade just cursory attempt with version

[jira] [Comment Edited] (SPARK-13407) TaskMetrics.fromAccumulatorUpdates can crash when trying to access garbage-collected accumulators

2017-01-25 Thread EE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1583#comment-1583 ] EE edited comment on SPARK-13407 at 1/25/17 2:09 PM: - We recreated this issue on

[jira] [Comment Edited] (SPARK-13407) TaskMetrics.fromAccumulatorUpdates can crash when trying to access garbage-collected accumulators

2017-01-25 Thread EE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1583#comment-1583 ] EE edited comment on SPARK-13407 at 1/25/17 2:07 PM: - We recreated this issue on

[jira] [Commented] (SPARK-10924) Failed to update accumulators for ShuffleMapTask: Broken pipe

2017-01-25 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837778#comment-15837778 ] Pau Tallada CrespĂ­ commented on SPARK-10924: That would be difficult now :/ We have since

[jira] [Commented] (SPARK-13407) TaskMetrics.fromAccumulatorUpdates can crash when trying to access garbage-collected accumulators

2017-01-25 Thread EE (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1583#comment-1583 ] EE commented on SPARK-13407: We recreated this case on spark 1.6.2 as well. on spark-streaming application

[jira] [Resolved] (SPARK-12827) Configurable bind address for WebUI

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12827. -- Resolution: Duplicate > Configurable bind address for WebUI >

[jira] [Commented] (SPARK-17360) PySpark can create dataframe from a Python generator

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837681#comment-15837681 ] Hyukjin Kwon commented on SPARK-17360: -- Hi [~holdenk], could we resolve this given the discussion in

[jira] [Closed] (SPARK-19362) master UI kill link stops spark context but leave it active

2017-01-25 Thread Artem Aliev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Aliev closed SPARK-19362. --- Resolution: Duplicate > master UI kill link stops spark context but leave it active >

[jira] [Updated] (SPARK-19362) master UI kill link stops spark context but leave it active

2017-01-25 Thread Artem Aliev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Aliev updated SPARK-19362: Affects Version/s: (was: 2.1.0) > master UI kill link stops spark context but leave it active

[jira] [Created] (SPARK-19362) master UI kill link stops spark context but leave it active

2017-01-25 Thread Artem Aliev (JIRA)
Artem Aliev created SPARK-19362: --- Summary: master UI kill link stops spark context but leave it active Key: SPARK-19362 URL: https://issues.apache.org/jira/browse/SPARK-19362 Project: Spark

[jira] [Commented] (SPARK-19356) Number of active tasks is negative even when there is no failed executor

2017-01-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837621#comment-15837621 ] Sean Owen commented on SPARK-19356: --- Maybe; why do you think that change relates to negative counts?

[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED

2017-01-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19354: -- Issue Type: Improvement (was: Bug) I think the status here is correct: it was killed, but failed

[jira] [Created] (SPARK-19361) kafka.maxRatePerPartition for compacted topic cause exception

2017-01-25 Thread Natalia Gorchakova (JIRA)
Natalia Gorchakova created SPARK-19361: -- Summary: kafka.maxRatePerPartition for compacted topic cause exception Key: SPARK-19361 URL: https://issues.apache.org/jira/browse/SPARK-19361 Project:

[jira] [Updated] (SPARK-19360) Spark 2.X does not support stored by cluase

2017-01-25 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19360: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) > Spark 2.X does not support

[jira] [Commented] (SPARK-18495) Web UI should document meaning of green dot in DAG visualization

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837515#comment-15837515 ] Apache Spark commented on SPARK-18495: -- User 'uncleGen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-18495) Web UI should document meaning of green dot in DAG visualization

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18495: Assignee: Apache Spark > Web UI should document meaning of green dot in DAG visualization

[jira] [Assigned] (SPARK-18495) Web UI should document meaning of green dot in DAG visualization

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18495: Assignee: (was: Apache Spark) > Web UI should document meaning of green dot in DAG

[jira] [Commented] (SPARK-19360) Spark 2.X does not support stored by cluase

2017-01-25 Thread Ran Haim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837503#comment-15837503 ] Ran Haim commented on SPARK-19360: -- Hi, I added my own storage handler actually - and now I cannot use

[jira] [Comment Edited] (SPARK-19360) Spark 2.X does not support stored by cluase

2017-01-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837457#comment-15837457 ] Dongjoon Hyun edited comment on SPARK-19360 at 1/25/17 9:44 AM: Hi,

[jira] [Comment Edited] (SPARK-19360) Spark 2.X does not support stored by cluase

2017-01-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837457#comment-15837457 ] Dongjoon Hyun edited comment on SPARK-19360 at 1/25/17 9:43 AM: Hi,

[jira] [Commented] (SPARK-19360) Spark 2.X does not support stored by cluase

2017-01-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837457#comment-15837457 ] Dongjoon Hyun commented on SPARK-19360: --- Hi, [~ran.h...@optimalplus.com]. Which storage handle do

[jira] [Commented] (SPARK-18909) The error message in `ExpressionEncoder.toRow` and `fromRow` is too verbose

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837438#comment-15837438 ] Apache Spark commented on SPARK-18909: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-18909) The error message in `ExpressionEncoder.toRow` and `fromRow` is too verbose

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18909: Assignee: Apache Spark > The error message in `ExpressionEncoder.toRow` and `fromRow` is

[jira] [Assigned] (SPARK-18909) The error message in `ExpressionEncoder.toRow` and `fromRow` is too verbose

2017-01-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-18909: Assignee: (was: Apache Spark) > The error message in `ExpressionEncoder.toRow` and

[jira] [Comment Edited] (SPARK-19340) Opening a file in CSV format will result in an exception if the filename contains special characters

2017-01-25 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837424#comment-15837424 ] Reza Safi edited comment on SPARK-19340 at 1/25/17 9:24 AM: As I mentioned in

[jira] [Comment Edited] (SPARK-19340) Opening a file in CSV format will result in an exception if the filename contains special characters

2017-01-25 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837424#comment-15837424 ] Reza Safi edited comment on SPARK-19340 at 1/25/17 9:23 AM: As I mentioned in

[jira] [Commented] (SPARK-19340) Opening a file in CSV format will result in an exception if the filename contains special characters

2017-01-25 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837424#comment-15837424 ] Reza Safi commented on SPARK-19340: --- As I mentioned in an earlier comment the exception only occurs if

[jira] [Created] (SPARK-19360) Spark 2.X does not support stored by cluase

2017-01-25 Thread Ran Haim (JIRA)
Ran Haim created SPARK-19360: Summary: Spark 2.X does not support stored by cluase Key: SPARK-19360 URL: https://issues.apache.org/jira/browse/SPARK-19360 Project: Spark Issue Type: Bug

  1   2   >