[jira] [Updated] (SPARK-17851) Make sure all test sqls in catalyst pass checkAnalysis

2016-10-15 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-17851: - Description: Currently we have several tens of test sqls in catalyst will fail at

[jira] [Commented] (SPARK-17930) The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-15 Thread Guoqiang Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577506#comment-15577506 ] Guoqiang Li commented on SPARK-17930: - If a stage contains a lot of tasks, eg one million tasks, the

[jira] [Created] (SPARK-17952) Java SparkSession createDataFrame doesn't work with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
Amit Baghel created SPARK-17952: --- Summary: Java SparkSession createDataFrame doesn't work with nested Javabean Key: SPARK-17952 URL: https://issues.apache.org/jira/browse/SPARK-17952 Project: Spark

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame doesn't work with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation at

[jira] [Created] (SPARK-17953) Fix typo in SparkSession scaladoc

2016-10-15 Thread Tae Jun Kim (JIRA)
Tae Jun Kim created SPARK-17953: --- Summary: Fix typo in SparkSession scaladoc Key: SPARK-17953 URL: https://issues.apache.org/jira/browse/SPARK-17953 Project: Spark Issue Type: Documentation

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame doesn't work with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation for Java at

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame doesn't work with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation at

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame doesn't work with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation at

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame doesn't work with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation at

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame method throws exception for nested JavaBeans

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Summary: Java SparkSession createDataFrame method throws exception for nested JavaBeans (was:

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame method throws exception with nested Javabean

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Summary: Java SparkSession createDataFrame method throws exception with nested Javabean (was:

[jira] [Commented] (SPARK-17953) Fix typo in SparkSession scaladoc

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577582#comment-15577582 ] Apache Spark commented on SPARK-17953: -- User 'tae-jun' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17953) Fix typo in SparkSession scaladoc

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17953: Assignee: (was: Apache Spark) > Fix typo in SparkSession scaladoc >

[jira] [Assigned] (SPARK-17953) Fix typo in SparkSession scaladoc

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17953: Assignee: Apache Spark > Fix typo in SparkSession scaladoc >

[jira] [Commented] (SPARK-17945) Writing to S3 should allow setting object metadata

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577593#comment-15577593 ] Sean Owen commented on SPARK-17945: --- You can just set this with the S3 APIs directly? this borders or

[jira] [Resolved] (SPARK-17953) Fix typo in SparkSession scaladoc

2016-10-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17953. - Resolution: Fixed Assignee: Tae Jun Kim Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-17950) Match SparseVector behavior with DenseVector

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577596#comment-15577596 ] Sean Owen commented on SPARK-17950: --- Doesn't that make a potentially huge array every time this is

[jira] [Updated] (SPARK-17953) Fix typo in SparkSession scaladoc

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17953: -- Priority: Trivial (was: Minor) > Fix typo in SparkSession scaladoc >

[jira] [Updated] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2016-10-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-16845: Target Version/s: 2.1.0 >

[jira] [Updated] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17951: -- Description: The following code demonstrates the issue: {code} def main(args: Array[String]): Unit =

[jira] [Commented] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577637#comment-15577637 ] Sean Owen commented on SPARK-17951: --- I'm not sure this suggests a problem with Spark. It's a very small

[jira] [Commented] (SPARK-17944) sbin/start-* scripts use of `hostname -f` fail with Solaris

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577826#comment-15577826 ] Sean Owen commented on SPARK-17944: --- Yes, for example `hostname` differs on OS X / Linux but the

[jira] [Resolved] (SPARK-17936) "CodeGenerator - failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of" method Error

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17936. --- Resolution: Fixed Assignee: Takuya Ueshin Fix Version/s: 2.1.0 OK, if you're pretty

[jira] [Commented] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577832#comment-15577832 ] Sean Owen commented on SPARK-14887: --- Possibly resolved by

[jira] [Commented] (SPARK-17930) The SerializerInstance instance used when deserializing a TaskResult is not reused

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577852#comment-15577852 ] Sean Owen commented on SPARK-17930: --- Maybe so; I think the question is whether this would cause it to

[jira] [Created] (SPARK-17954) FetchFailedException executor cannot connect to another worker executor

2016-10-15 Thread Vitaly Gerasimov (JIRA)
Vitaly Gerasimov created SPARK-17954: Summary: FetchFailedException executor cannot connect to another worker executor Key: SPARK-17954 URL: https://issues.apache.org/jira/browse/SPARK-17954

[jira] [Updated] (SPARK-17954) FetchFailedException executor cannot connect to another worker executor

2016-10-15 Thread Vitaly Gerasimov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitaly Gerasimov updated SPARK-17954: - Issue Type: Bug (was: Question) > FetchFailedException executor cannot connect to

[jira] [Commented] (SPARK-14428) [SQL] Allow more flexibility when parsing dates and timestamps in json datasources

2016-10-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577994#comment-15577994 ] Hyukjin Kwon commented on SPARK-14428: -- For 1. I guess this was fixed in

[jira] [Created] (SPARK-17955) Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc")

2016-10-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-17955: Summary: Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc") Key: SPARK-17955 URL: https://issues.apache.org/jira/browse/SPARK-17955

[jira] [Commented] (SPARK-17955) Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc")

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578038#comment-15578038 ] Apache Spark commented on SPARK-17955: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-17955) Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc")

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17955: Assignee: Apache Spark > Use the same read path in DataFrameReader.jdbc and >

[jira] [Assigned] (SPARK-17955) Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc")

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17955: Assignee: (was: Apache Spark) > Use the same read path in DataFrameReader.jdbc and >

[jira] [Commented] (SPARK-17945) Writing to S3 should allow setting object metadata

2016-10-15 Thread Jeff Schobelock (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578064#comment-15578064 ] Jeff Schobelock commented on SPARK-17945: - That's true enough. The use case on my end is that we

[jira] [Comment Edited] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-15 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578987#comment-15578987 ] ding edited comment on SPARK-17951 at 10/16/16 12:22 AM: - I tried to call

[jira] [Updated] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-17637: Affects Version/s: 2.1.0 > Packed scheduling for Spark tasks across executors >

[jira] [Resolved] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-17637. - Resolution: Fixed Assignee: Zhan Zhang Target Version/s:

[jira] [Commented] (SPARK-17865) R API for global temp view

2016-10-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578835#comment-15578835 ] Reynold Xin commented on SPARK-17865: - Alright in this case I don't think we need R API for now. >

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame method throws exception for nested JavaBeans

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation for Java at

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame method throws exception for nested JavaBeans

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation for Java at

[jira] [Updated] (SPARK-17952) Java SparkSession createDataFrame method throws exception for nested JavaBeans

2016-10-15 Thread Amit Baghel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Baghel updated SPARK-17952: Description: As per latest spark documentation for Java at

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-15 Thread Ofir Manor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578699#comment-15578699 ] Ofir Manor commented on SPARK-17812: I think Michael suggest that if you use {{startingOffsets}}

[jira] [Commented] (SPARK-17951) BlockFetch with multiple threads slows down after spark 1.6

2016-10-15 Thread ding (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578987#comment-15578987 ] ding commented on SPARK-17951: -- I tried to call rdd.collect which internally called bm.getRemoteBytes in

[jira] [Updated] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-17637: Fix Version/s: 2.1.0 > Packed scheduling for Spark tasks across executors >

[jira] [Closed] (SPARK-17865) R API for global temp view

2016-10-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-17865. --- Resolution: Not A Problem > R API for global temp view > -- > >

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Lars Francke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578875#comment-15578875 ] Lars Francke commented on SPARK-650: I also have to disagree with this being a duplicate or obsolete.

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-15 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578894#comment-15578894 ] Cody Koeninger commented on SPARK-17812: As you just said yourself, assign doesn't mean you

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Lars Francke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578914#comment-15578914 ] Lars Francke commented on SPARK-650: I can only come up with three reasons at the moment. I hope they

[jira] [Commented] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Linbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579142#comment-15579142 ] Linbo commented on SPARK-17957: --- cc [~smilegator] > Calling outer join and na.fill(0) and then inner join

[jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579206#comment-15579206 ] Apache Spark commented on SPARK-17812: -- User 'koeninger' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17812: Assignee: Cody Koeninger (was: Apache Spark) > More granular control of starting offsets

[jira] [Assigned] (SPARK-17812) More granular control of starting offsets (assign)

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17812: Assignee: Apache Spark (was: Cody Koeninger) > More granular control of starting offsets

[jira] [Reopened] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-650: - As you wish, but, I disagree with this type of reasoning about JIRAs. I dont think anyone has addressed why a

[jira] [Comment Edited] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Linbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579142#comment-15579142 ] Linbo edited comment on SPARK-17957 at 10/16/16 2:26 AM: - cc [~smilegator] and

[jira] [Updated] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Linbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Linbo updated SPARK-17957: -- Description: I reported a similar bug two months ago and it's fixed in Spark 2.0.1:

[jira] [Created] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Linbo (JIRA)
Linbo created SPARK-17957: - Summary: Calling outer join and na.fill(0) and then inner join will miss rows Key: SPARK-17957 URL: https://issues.apache.org/jira/browse/SPARK-17957 Project: Spark

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

2016-10-15 Thread Saikat Kanjilal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578723#comment-15578723 ] Saikat Kanjilal commented on SPARK-9487: Synched the code, am familiarizing myself first with how

[jira] [Comment Edited] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Lars Francke (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578914#comment-15578914 ] Lars Francke edited comment on SPARK-650 at 10/15/16 11:22 PM: --- I can only

[jira] [Reopened] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin reopened SPARK-17637: - > Packed scheduling for Spark tasks across executors >

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579382#comment-15579382 ] Reynold Xin commented on SPARK-17637: - Note: I reverted the commit due to quality issues. We can

[jira] [Commented] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Linbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579294#comment-15579294 ] Linbo commented on SPARK-17957: --- Thank you! > Calling outer join and na.fill(0) and then inner join will

[jira] [Commented] (SPARK-17954) FetchFailedException executor cannot connect to another worker executor

2016-10-15 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579262#comment-15579262 ] Tejas Patil commented on SPARK-17954: - I agree with [~srowen]'s comment about this being more of a

[jira] [Commented] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579260#comment-15579260 ] Xiao Li commented on SPARK-17957: - You can see the plan. The optimized plan is still full outer join. : )

[jira] [Created] (SPARK-17958) Why I ran into issue " accumulator, copyandreset must be zero error"

2016-10-15 Thread dianwei Han (JIRA)
dianwei Han created SPARK-17958: --- Summary: Why I ran into issue " accumulator, copyandreset must be zero error" Key: SPARK-17958 URL: https://issues.apache.org/jira/browse/SPARK-17958 Project: Spark

[jira] [Assigned] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17637: Assignee: Zhan Zhang (was: Apache Spark) > Packed scheduling for Spark tasks across

[jira] [Assigned] (SPARK-17637) Packed scheduling for Spark tasks across executors

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17637: Assignee: Apache Spark (was: Zhan Zhang) > Packed scheduling for Spark tasks across

[jira] [Commented] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579252#comment-15579252 ] Xiao Li commented on SPARK-17957: - Thank you for reporting it. Let me do a quick check. > Calling outer

[jira] [Updated] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-17957: Priority: Critical (was: Major) > Calling outer join and na.fill(0) and then inner join will miss rows >

[jira] [Updated] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-17957: Target Version/s: 2.0.2, 2.1.0 (was: 2.0.2) > Calling outer join and na.fill(0) and then inner join will

[jira] [Updated] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-17957: Labels: correctness (was: joins na.fill) > Calling outer join and na.fill(0) and then inner join will

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

2016-10-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579284#comment-15579284 ] Shixiong Zhu commented on SPARK-13747: -- [~chinwei] could you post the stack trace here? >

[jira] [Commented] (SPARK-17957) Calling outer join and na.fill(0) and then inner join will miss rows

2016-10-15 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579282#comment-15579282 ] Xiao Li commented on SPARK-17957: - Found the bug. {noformat} Project [a#29, b#30, c#31, d#48] +- Join

[jira] [Commented] (SPARK-17954) FetchFailedException executor cannot connect to another worker executor

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578542#comment-15578542 ] Sean Owen commented on SPARK-17954: --- Lots of things changed. I think you need to first confirm

[jira] [Commented] (SPARK-17938) Backpressure rate not adjusting

2016-10-15 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578133#comment-15578133 ] Cody Koeninger commented on SPARK-17938: There was pretty extensive discussion of this on list,

[jira] [Created] (SPARK-17956) ProjectExec has incorrect outputOrdering property

2016-10-15 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17956: --- Summary: ProjectExec has incorrect outputOrdering property Key: SPARK-17956 URL: https://issues.apache.org/jira/browse/SPARK-17956 Project: Spark

[jira] [Commented] (SPARK-17954) FetchFailedException executor cannot connect to another worker executor

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578155#comment-15578155 ] Sean Owen commented on SPARK-17954: --- Is this not just a networking or IP/hostname problem? it says it

[jira] [Commented] (SPARK-17935) Add KafkaForeachWriter in external kafka-0.8.0 for structured streaming module

2016-10-15 Thread zhangxinyu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578176#comment-15578176 ] zhangxinyu commented on SPARK-17935: I will try it on 0.10 in the future. But as I know most of users

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Olivier Armand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578487#comment-15578487 ] Olivier Armand commented on SPARK-650: -- Sean, a singleton is not the best option in our case. The

[jira] [Commented] (SPARK-17865) R API for global temp view

2016-10-15 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578639#comment-15578639 ] Felix Cheung commented on SPARK-17865: -- I see. So then we could either omit the support for global

[jira] [Commented] (SPARK-17838) Strict type checking for arguments with a better messages across APIs.

2016-10-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578101#comment-15578101 ] Hyukjin Kwon commented on SPARK-17838: -- I am trying to work on this but I don't mind if anyone takes

[jira] [Commented] (SPARK-636) Add mechanism to run system management/configuration tasks on all workers

2016-10-15 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578294#comment-15578294 ] Michael Schmeißer commented on SPARK-636: - I agree, that's why I also feel that these issues are no

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578535#comment-15578535 ] Sean Owen commented on SPARK-650: - If you need init to happen ASAP when the driver starts, isn't any

[jira] [Commented] (SPARK-17878) Support for multiple null values when reading CSV data

2016-10-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578098#comment-15578098 ] Hyukjin Kwon commented on SPARK-17878: -- Yes, this might not be a reason to block this feature. It

[jira] [Assigned] (SPARK-17956) ProjectExec has incorrect outputOrdering property

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17956: Assignee: (was: Apache Spark) > ProjectExec has incorrect outputOrdering property >

[jira] [Commented] (SPARK-17956) ProjectExec has incorrect outputOrdering property

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578136#comment-15578136 ] Apache Spark commented on SPARK-17956: -- User 'viirya' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17956) ProjectExec has incorrect outputOrdering property

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17956: Assignee: Apache Spark > ProjectExec has incorrect outputOrdering property >

[jira] [Commented] (SPARK-11524) Support SparkR with Mesos cluster

2016-10-15 Thread Susan X. Huynh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578137#comment-15578137 ] Susan X. Huynh commented on SPARK-11524: I would like to work on this. What are the missing

[jira] [Commented] (SPARK-17935) Add KafkaForeachWriter in external kafka-0.8.0 for structured streaming module

2016-10-15 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578141#comment-15578141 ] Cody Koeninger commented on SPARK-17935: Why is this in kafka-0-8, when we haven't resolved (for

[jira] [Assigned] (SPARK-17892) Query in CTAS is Optimized Twice (branch-2.0)

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17892: Assignee: Xiao Li (was: Apache Spark) > Query in CTAS is Optimized Twice (branch-2.0) >

[jira] [Commented] (SPARK-17892) Query in CTAS is Optimized Twice (branch-2.0)

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578300#comment-15578300 ] Apache Spark commented on SPARK-17892: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17892) Query in CTAS is Optimized Twice (branch-2.0)

2016-10-15 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17892: Assignee: Apache Spark (was: Xiao Li) > Query in CTAS is Optimized Twice (branch-2.0) >

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578338#comment-15578338 ] Sean Owen commented on SPARK-650: - In practice, these should probably all be WontFix as it hasn't mattered

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578361#comment-15578361 ] Michael Schmeißer commented on SPARK-650: - Then somebody should please explain to me, how this

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578378#comment-15578378 ] Sean Owen commented on SPARK-650: - Sorry, I mean the _status_ doesn't matter. Most issues this old are

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2016-10-15 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578289#comment-15578289 ] Michael Schmeißer commented on SPARK-650: - I disagree that those issues are duplicates. Spark-636

[jira] [Commented] (SPARK-17878) Support for multiple null values when reading CSV data

2016-10-15 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578607#comment-15578607 ] Hossein Falaki commented on SPARK-17878: I think moving it to another ticket is a good idea. One

[jira] [Commented] (SPARK-17709) spark 2.0 join - column resolution error

2016-10-15 Thread Ashish Shrowty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578200#comment-15578200 ] Ashish Shrowty commented on SPARK-17709: Oh .. sorry .. I misread. Will try with 2.0.1 later >

[jira] [Commented] (SPARK-17954) FetchFailedException executor cannot connect to another worker executor

2016-10-15 Thread Vitaly Gerasimov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578539#comment-15578539 ] Vitaly Gerasimov commented on SPARK-17954: -- I don't think so. Spark 1.6 works fine in this case.