[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203095#comment-16203095 ] Takeshi Yamamuro commented on SPARK-22271: -- More->Attach Files? btw, text file (csv or

[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Shafique Jamal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203094#comment-16203094 ] Shafique Jamal commented on SPARK-22271: I'm happy to share the parquet file - how can I upload

[jira] [Assigned] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22266: Assignee: (was: Apache Spark) > The same aggregate function was evaluated multiple

[jira] [Assigned] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22266: Assignee: Apache Spark > The same aggregate function was evaluated multiple times >

[jira] [Commented] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203091#comment-16203091 ] Apache Spark commented on SPARK-22266: -- User 'maryannxue' has created a pull request for this issue:

[jira] [Resolved] (SPARK-22257) Reserve all non-deterministic expressions in ExpressionSet.

2017-10-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22257. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.3.0 > Reserve all

[jira] [Commented] (SPARK-22270) Renaming DF column breaks sparkPlan.outputOrdering

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203081#comment-16203081 ] Takeshi Yamamuro commented on SPARK-22270: -- Probably, this is duplicate to

[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203080#comment-16203080 ] Takeshi Yamamuro commented on SPARK-22271: -- You need to give us the data and the schema, too? >

[jira] [Comment Edited] (SPARK-16060) Vectorized Orc reader

2017-10-12 Thread Rajiv Chodisetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203064#comment-16203064 ] Rajiv Chodisetti edited comment on SPARK-16060 at 10/13/17 5:19 AM:

[jira] [Updated] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Shafique Jamal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shafique Jamal updated SPARK-22271: --- Description: Please excuse me if this issue was addressed already - I was unable to find it.

[jira] [Comment Edited] (SPARK-16060) Vectorized Orc reader

2017-10-12 Thread Rajiv Chodisetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203064#comment-16203064 ] Rajiv Chodisetti edited comment on SPARK-16060 at 10/13/17 5:18 AM:

[jira] [Commented] (SPARK-16060) Vectorized Orc reader

2017-10-12 Thread Rajiv Chodisetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203064#comment-16203064 ] Rajiv Chodisetti commented on SPARK-16060: -- Will this fix, this issue the below issue as well ,

[jira] [Created] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Shafique Jamal (JIRA)
Shafique Jamal created SPARK-22271: -- Summary: Describe results in "null" for the value of "mean" of a numeric variable Key: SPARK-22271 URL: https://issues.apache.org/jira/browse/SPARK-22271

[jira] [Updated] (SPARK-21165) Fail to write into partitioned hive table due to attribute reference not working with cast on partition column

2017-10-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21165: Fix Version/s: 2.3.0 > Fail to write into partitioned hive table due to attribute reference not >

[jira] [Updated] (SPARK-22252) FileFormatWriter should respect the input query schema

2017-10-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22252: Fix Version/s: 2.2.1 > FileFormatWriter should respect the input query schema >

[jira] [Commented] (SPARK-16628) OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203042#comment-16203042 ] Apache Spark commented on SPARK-16628: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-22270) Renaming DF column breaks sparkPlan.outputOrdering

2017-10-12 Thread Yuri Bogomolov (JIRA)
Yuri Bogomolov created SPARK-22270: -- Summary: Renaming DF column breaks sparkPlan.outputOrdering Key: SPARK-22270 URL: https://issues.apache.org/jira/browse/SPARK-22270 Project: Spark Issue

[jira] [Updated] (SPARK-22263) Refactor deterministic as lazy value

2017-10-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22263: Priority: Major (was: Critical) > Refactor deterministic as lazy value >

[jira] [Resolved] (SPARK-22263) Refactor deterministic as lazy value

2017-10-12 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22263. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.3.0 > Refactor deterministic

[jira] [Comment Edited] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202924#comment-16202924 ] Reynold Xin edited comment on SPARK-20928 at 10/13/17 1:40 AM: --- OK got it -

[jira] [Commented] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202924#comment-16202924 ] Reynold Xin commented on SPARK-20928: - OK got it - you are basically saying if we can send the offset

[jira] [Commented] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202923#comment-16202923 ] Cody Koeninger commented on SPARK-20928: If a given sink is handling a result, why does handling

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202900#comment-16202900 ] Saisai Shao commented on SPARK-9: - {quote} I don't think that limited familiarity with a new

[jira] [Commented] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202906#comment-16202906 ] Reynold Xin commented on SPARK-20928: - Isn't there an issue with the overhead of tracking in the

[jira] [Commented] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202905#comment-16202905 ] Cody Koeninger commented on SPARK-20928: I was talking about the specific case of jobs with only

[jira] [Updated] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22217: - Fix Version/s: 2.2.1 > ParquetFileFormat to support arbitrary OutputCommitters >

[jira] [Comment Edited] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202871#comment-16202871 ] Takeshi Yamamuro edited comment on SPARK-22266 at 10/13/17 12:39 AM: -

[jira] [Updated] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-22266: - Issue Type: Improvement (was: Bug) > The same aggregate function was evaluated multiple

[jira] [Commented] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202871#comment-16202871 ] Takeshi Yamamuro commented on SPARK-22266: -- This is not a bug, so I changed to improvement. >

[jira] [Commented] (SPARK-21549) Spark fails to complete job correctly in case of OutputFormat which do not write into hdfs

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202865#comment-16202865 ] Apache Spark commented on SPARK-21549: -- User 'mridulm' has created a pull request for this issue:

[jira] [Commented] (SPARK-22211) LimitPushDown optimization for FullOuterJoin generates wrong results

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202863#comment-16202863 ] Takeshi Yamamuro commented on SPARK-22211: -- Aha, I misunderstood. But, I think the case 3 is not

[jira] [Assigned] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22217: Assignee: Steve Loughran > ParquetFileFormat to support arbitrary OutputCommitters >

[jira] [Resolved] (SPARK-22217) ParquetFileFormat to support arbitrary OutputCommitters

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22217. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19448

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202795#comment-16202795 ] Andrew Ash commented on SPARK-22269: [~sowen] you closed this as a duplicate. What issue is it a

[jira] [Commented] (SPARK-22268) Fix java style errors

2017-10-12 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202793#comment-16202793 ] Andrew Ash commented on SPARK-22268: Any time {{./dev/run-tests}} is failing I consider that a bug.

[jira] [Resolved] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22269. --- Resolution: Duplicate > Java style checks should be run in Jenkins >

[jira] [Updated] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22269: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) Also not a bug. We've tried

[jira] [Updated] (SPARK-22268) Fix java style errors

2017-10-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22268: -- Priority: Trivial (was: Major) Issue Type: Improvement (was: Bug) OK, don't make JIRAs for

[jira] [Commented] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202710#comment-16202710 ] Marcelo Vanzin commented on SPARK-22269: I've heard in the past that this wasn't desired because

[jira] [Created] (SPARK-22269) Java style checks should be run in Jenkins

2017-10-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22269: -- Summary: Java style checks should be run in Jenkins Key: SPARK-22269 URL: https://issues.apache.org/jira/browse/SPARK-22269 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-22268) Fix java style errors

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22268: Assignee: Apache Spark > Fix java style errors > - > >

[jira] [Assigned] (SPARK-22268) Fix java style errors

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22268: Assignee: (was: Apache Spark) > Fix java style errors > - > >

[jira] [Commented] (SPARK-22268) Fix java style errors

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202697#comment-16202697 ] Apache Spark commented on SPARK-22268: -- User 'ash211' has created a pull request for this issue:

[jira] [Created] (SPARK-22268) Fix java style errors

2017-10-12 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22268: -- Summary: Fix java style errors Key: SPARK-22268 URL: https://issues.apache.org/jira/browse/SPARK-22268 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-12 Thread Arthur Baudry (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202691#comment-16202691 ] Arthur Baudry commented on SPARK-22240: --- [~hyukjin.kwon] Yes it is a single file so even counting

[jira] [Commented] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202621#comment-16202621 ] Reynold Xin commented on SPARK-20928: - [~c...@koeninger.org] can you write down your thoughts on how

[jira] [Updated] (SPARK-20928) Continuous Processing Mode for Structured Streaming

2017-10-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-20928: Labels: SPIP (was: ) > Continuous Processing Mode for Structured Streaming >

[jira] [Commented] (SPARK-21907) NullPointerException in UnsafeExternalSorter.spill()

2017-10-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202547#comment-16202547 ] Dongjoon Hyun commented on SPARK-21907: --- Hi, [~hvanhovell] and [~eyalfa]. I added `2.2.1` in fixed

[jira] [Updated] (SPARK-21907) NullPointerException in UnsafeExternalSorter.spill()

2017-10-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21907: -- Fix Version/s: 2.2.1 > NullPointerException in UnsafeExternalSorter.spill() >

[jira] [Created] (SPARK-22267) Spark SQL incorrectly reads ORC file when column order is different

2017-10-12 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-22267: - Summary: Spark SQL incorrectly reads ORC file when column order is different Key: SPARK-22267 URL: https://issues.apache.org/jira/browse/SPARK-22267 Project: Spark

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-10-12 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202385#comment-16202385 ] Hossein Falaki commented on SPARK-15799: Congrats everyone. Thanks for the hard work on this. >

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-12 Thread Yuval Degani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202368#comment-16202368 ] Yuval Degani commented on SPARK-9: -- Good point [~jerryshao]. Regarding testing on a machines

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202360#comment-16202360 ] Hyukjin Kwon commented on SPARK-15799: -- Yay! > Release SparkR on CRAN > -- > >

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-10-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202344#comment-16202344 ] Dongjoon Hyun commented on SPARK-15799: --- Great news, [~shivaram]! > Release SparkR on CRAN >

[jira] [Created] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-22266: --- Summary: The same aggregate function was evaluated multiple times Key: SPARK-22266 URL: https://issues.apache.org/jira/browse/SPARK-22266 Project: Spark Issue

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

2017-10-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202330#comment-16202330 ] Kazuaki Ishizaki commented on SPARK-16845: -- [This PR|https://github.com/apache/spark/pull/18972]

[jira] [Comment Edited] (SPARK-18350) Support session local timezone

2017-10-12 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202322#comment-16202322 ] Alexandre Dupriez edited comment on SPARK-18350 at 10/12/17 5:30 PM: -

[jira] [Comment Edited] (SPARK-18350) Support session local timezone

2017-10-12 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202322#comment-16202322 ] Alexandre Dupriez edited comment on SPARK-18350 at 10/12/17 5:30 PM: -

[jira] [Comment Edited] (SPARK-18350) Support session local timezone

2017-10-12 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202322#comment-16202322 ] Alexandre Dupriez edited comment on SPARK-18350 at 10/12/17 5:29 PM: -

[jira] [Commented] (SPARK-18350) Support session local timezone

2017-10-12 Thread Alexandre Dupriez (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202322#comment-16202322 ] Alexandre Dupriez commented on SPARK-18350: --- Hello all, I have a use case where a {{Dataset}}

[jira] [Commented] (SPARK-22211) LimitPushDown optimization for FullOuterJoin generates wrong results

2017-10-12 Thread Benyi Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202278#comment-16202278 ] Benyi Wang commented on SPARK-22211: I think my suggestion solution is correct. || Case || Left join

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-10-12 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202260#comment-16202260 ] Shivaram Venkataraman commented on SPARK-15799: --- This is now live !

[jira] [Resolved] (SPARK-15799) Release SparkR on CRAN

2017-10-12 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-15799. --- Resolution: Fixed Assignee: Shivaram Venkataraman Fix

[jira] [Commented] (SPARK-20055) Documentation for CSV datasets in SQL programming guide

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202174#comment-16202174 ] Apache Spark commented on SPARK-20055: -- User 'jomach' has created a pull request for this issue:

[jira] [Resolved] (SPARK-22265) pyspark can't erialization object

2017-10-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22265. --- Resolution: Invalid Please ask questions on the mailing list or StackOverflow, though you'll need

[jira] [Created] (SPARK-22265) pyspark can't erialization object

2017-10-12 Thread bianxiaokun (JIRA)
bianxiaokun created SPARK-22265: --- Summary: pyspark can't erialization object Key: SPARK-22265 URL: https://issues.apache.org/jira/browse/SPARK-22265 Project: Spark Issue Type: IT Help

[jira] [Commented] (SPARK-22248) spark marks all columns as null when its unable to parse single column

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202116#comment-16202116 ] Hyukjin Kwon commented on SPARK-22248: -- I think either way breaks existing support. JSON was started

[jira] [Comment Edited] (SPARK-22164) support histogram in estimating the cardinality of aggregate (or group-by) operator

2017-10-12 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202081#comment-16202081 ] Zhenhua Wang edited comment on SPARK-22164 at 10/12/17 3:21 PM: [~ron8hu]

[jira] [Resolved] (SPARK-22164) support histogram in estimating the cardinality of aggregate (or group-by) operator

2017-10-12 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang resolved SPARK-22164. -- Resolution: Won't Fix Target Version/s: (was: 2.3.0) > support histogram in

[jira] [Closed] (SPARK-22164) support histogram in estimating the cardinality of aggregate (or group-by) operator

2017-10-12 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang closed SPARK-22164. > support histogram in estimating the cardinality of aggregate (or group-by) > operator >

[jira] [Commented] (SPARK-22164) support histogram in estimating the cardinality of aggregate (or group-by) operator

2017-10-12 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202081#comment-16202081 ] Zhenhua Wang commented on SPARK-22164: -- [~ron8hu] I don't think histogram can help with group-by,

[jira] [Resolved] (SPARK-22251) Metric "aggregate time" is incorrect when codegen is off

2017-10-12 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-22251. --- Resolution: Fixed Assignee: Ala Luszczak Fix Version/s: 2.3.0 >

[jira] [Commented] (SPARK-22259) hdfs://HdfsHA/logrep/1/sspstatistic/_metadata is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [5, 28, 21, 12]

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202071#comment-16202071 ] Hyukjin Kwon commented on SPARK-22259: -- Looks a dupe of SPARK-22260 BTW. >

[jira] [Resolved] (SPARK-22259) hdfs://HdfsHA/logrep/1/sspstatistic/_metadata is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [5, 28, 21, 12]

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22259. -- Resolution: Duplicate > hdfs://HdfsHA/logrep/1/sspstatistic/_metadata is not a Parquet file.

[jira] [Commented] (SPARK-22260) java.lang.RuntimeException: hdfs://HdfsHA/logrep/1/sspstatistic/_metadata is not a Parquet file (too small)

2017-10-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202070#comment-16202070 ] Hyukjin Kwon commented on SPARK-22260: -- Does this happen randomly? Can you provide a reproducer

[jira] [Commented] (SPARK-22259) hdfs://HdfsHA/logrep/1/sspstatistic/_metadata is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [5, 28, 21, 12]

2017-10-12 Thread Kaushal Prajapati (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202052#comment-16202052 ] Kaushal Prajapati commented on SPARK-22259: --- Seem like file which you are using is not parquet.

[jira] [Commented] (SPARK-20712) [SPARK 2.1 REGRESSION][SQL] Spark can't read Hive table when column type has length greater than 4000 bytes

2017-10-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202016#comment-16202016 ] Maciej Bryński commented on SPARK-20712: After ALTER I needed to recreate all type strings in

[jira] [Updated] (SPARK-20712) [SPARK 2.1 REGRESSION][SQL] Spark can't read Hive table when column type has length greater than 4000 bytes

2017-10-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-20712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-20712: --- Affects Version/s: 2.2.0 > [SPARK 2.1 REGRESSION][SQL] Spark can't read Hive table when

[jira] [Commented] (SPARK-22252) FileFormatWriter should respect the input query schema

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202004#comment-16202004 ] Apache Spark commented on SPARK-22252: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-21165) Fail to write into partitioned hive table due to attribute reference not working with cast on partition column

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201988#comment-16201988 ] Apache Spark commented on SPARK-21165: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-22247) Hive partition filter very slow

2017-10-12 Thread Noam Asor (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201971#comment-16201971 ] Noam Asor commented on SPARK-22247: --- Maybe this issue is related SPARK-17992 > Hive partition filter

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

2017-10-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201950#comment-16201950 ] Saisai Shao commented on SPARK-9: - My concern is about how to maintain this code in the

[jira] [Commented] (SPARK-21797) spark cannot read partitioned data in S3 that are partly in glacier

2017-10-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201917#comment-16201917 ] Steve Loughran commented on SPARK-21797: Update, in HADOOP-14874 I've noted we could use the

[jira] [Resolved] (SPARK-22197) push down operators to data source before planning

2017-10-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22197. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19424

[jira] [Updated] (SPARK-22097) Request an accurate memory after we unrolled the block

2017-10-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-22097: Affects Version/s: (was: 2.2.0) 2.3.0 > Request an accurate memory

[jira] [Resolved] (SPARK-22097) Request an accurate memory after we unrolled the block

2017-10-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22097. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19316

[jira] [Commented] (SPARK-22242) streaming job failed to restart from checkpoint

2017-10-12 Thread StephenZou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201868#comment-16201868 ] StephenZou commented on SPARK-22242: Perhaps, plz not to forget to add spark.yarn.jars if fixed

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

2017-10-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201863#comment-16201863 ] Steve Loughran commented on SPARK-22240: thanks. Now for a question which is probably obvious to

[jira] [Resolved] (SPARK-22252) FileFormatWriter should respect the input query schema

2017-10-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22252. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19474

[jira] [Updated] (SPARK-21646) Add new type coercion rules to compatible with Hive

2017-10-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21646: Attachment: Type_coercion_rules_to_compatible_with_Hive.pdf > Add new type coercion rules to

[jira] [Assigned] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22264: Assignee: Apache Spark > History server will be unavailable if there is an event log file

[jira] [Assigned] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22264: Assignee: (was: Apache Spark) > History server will be unavailable if there is an

[jira] [Commented] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201741#comment-16201741 ] Apache Spark commented on SPARK-22264: -- User 'caneGuy' has created a pull request for this issue:

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Description: History server will be unavailable if there is an event log file with large size. Large

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Description: History server will be unavailable if there is an event log file with large size. Large

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Description: History server will be unavailable if there is an event log file with large size. Large

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Description: History server will be unavailable if there is an event log file with large size. Large

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Description: History server will be unavailable if there is an event log file with large size. Large

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Attachment: not-found.png > History server will be unavailable if there is an event log file with large

[jira] [Updated] (SPARK-22264) History server will be unavailable if there is an event log file with large size

2017-10-12 Thread zhoukang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhoukang updated SPARK-22264: - Description: History server will be unavailable if there is an event log file with large size. Large

  1   2   >