[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430096#comment-15430096 ] Dongjoon Hyun commented on SPARK-15285: --- Hi, [~cloud_fan] Could you resolve this issue please? >

[jira] [Assigned] (SPARK-17180) Unable to Alter the Temporary View Using ALTER VIEW command

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17180: Assignee: Apache Spark > Unable to Alter the Temporary View Using ALTER VIEW command >

[jira] [Assigned] (SPARK-17180) Unable to Alter the Temporary View Using ALTER VIEW command

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17180: Assignee: (was: Apache Spark) > Unable to Alter the Temporary View Using ALTER VIEW

[jira] [Commented] (SPARK-17180) Unable to Alter the Temporary View Using ALTER VIEW command

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430200#comment-15430200 ] Apache Spark commented on SPARK-17180: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-17180) Unable to Alter the Temporary View Using ALTER VIEW command

2016-08-22 Thread Xiao Li (JIRA)
Xiao Li created SPARK-17180: --- Summary: Unable to Alter the Temporary View Using ALTER VIEW command Key: SPARK-17180 URL: https://issues.apache.org/jira/browse/SPARK-17180 Project: Spark Issue

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2016-08-22 Thread DjvuLee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430215#comment-15430215 ] DjvuLee commented on SPARK-3630: How much data do you test? we encounter this error in our production.

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2016-08-22 Thread DjvuLee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430217#comment-15430217 ] DjvuLee commented on SPARK-3630: How much data do you test? we encounter this error in our production.

[jira] [Comment Edited] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2016-08-22 Thread DjvuLee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430215#comment-15430215 ] DjvuLee edited comment on SPARK-3630 at 8/22/16 7:10 AM: - Can I know how much data

[jira] [Issue Comment Deleted] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2016-08-22 Thread DjvuLee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DjvuLee updated SPARK-3630: --- Comment: was deleted (was: How much data do you test? we encounter this error in our production. Our data

[jira] [Commented] (SPARK-5770) Use addJar() to upload a new jar file to executor, it can't be added to classloader

2016-08-22 Thread marymwu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430243#comment-15430243 ] marymwu commented on SPARK-5770: Hey, we have ran into the same issue too. We try to fix this but failed.

[jira] [Resolved] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-15285. - Resolution: Fixed Fix Version/s: (was: 2.0.0) 2.1.0

[jira] [Commented] (SPARK-17168) CSV with header is incorrectly read if file is partitioned

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430258#comment-15430258 ] Sean Owen commented on SPARK-17168: --- It's a tough call. I can imagine for example a process ingesting

[jira] [Commented] (SPARK-17090) Make tree aggregation level in linear/logistic regression configurable

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430266#comment-15430266 ] Apache Spark commented on SPARK-17090: -- User 'hqzizania' has created a pull request for this issue:

[jira] [Resolved] (SPARK-17127) Include AArch64 in the check of cached unaligned-access capability

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17127. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14700

[jira] [Updated] (SPARK-17127) Include AArch64 in the check of cached unaligned-access capability

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17127: -- Assignee: Richael Zhuang > Include AArch64 in the check of cached unaligned-access capability >

[jira] [Commented] (SPARK-17086) QuantileDiscretizer throws InvalidArgumentException (parameter splits given invalid value) on valid data

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430270#comment-15430270 ] Apache Spark commented on SPARK-17086: -- User 'VinceShieh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17086) QuantileDiscretizer throws InvalidArgumentException (parameter splits given invalid value) on valid data

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17086: Assignee: (was: Apache Spark) > QuantileDiscretizer throws InvalidArgumentException

[jira] [Assigned] (SPARK-17086) QuantileDiscretizer throws InvalidArgumentException (parameter splits given invalid value) on valid data

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17086: Assignee: Apache Spark > QuantileDiscretizer throws InvalidArgumentException (parameter

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430273#comment-15430273 ] Dongjoon Hyun commented on SPARK-15285: --- Thank you! > Generated SpecificSafeProjection.apply

[jira] [Resolved] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17172. --- Resolution: Duplicate That sounds like exactly the same issue. You're missing /tmp or can't see it

[jira] [Commented] (SPARK-17143) pyspark unable to create UDF: java.lang.RuntimeException: org.apache.hadoop.fs.FileAlreadyExistsException: Parent path is not a directory: /tmp tmp

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430278#comment-15430278 ] Sean Owen commented on SPARK-17143: --- This sounds like an HDFS environment problem then. This dir would

[jira] [Resolved] (SPARK-17115) Improve the performance of UnsafeProjection for wide table

2016-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-17115. - Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull

[jira] [Commented] (SPARK-17169) To use scala macros to update code when SharedParamsCodeGen.scala changed

2016-08-22 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430282#comment-15430282 ] Yanbo Liang commented on SPARK-17169: - Meanwhile, it's better we can do compile time code-gen for

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2016-08-22 Thread Alexander Bij (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430290#comment-15430290 ] Alexander Bij commented on SPARK-10925: --- We also encountered this issue using with (HDP 2.4.2.0)

[jira] [Comment Edited] (SPARK-10925) Exception when joining DataFrames

2016-08-22 Thread Alexander Bij (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430290#comment-15430290 ] Alexander Bij edited comment on SPARK-10925 at 8/22/16 8:29 AM: We also

[jira] [Updated] (SPARK-17085) Documentation and actual code differs - Unsupported Operations

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17085: -- Assignee: Jagadeesan A S > Documentation and actual code differs - Unsupported Operations >

[jira] [Resolved] (SPARK-17085) Documentation and actual code differs - Unsupported Operations

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-17085. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Resolved by

[jira] [Commented] (SPARK-10925) Exception when joining DataFrames

2016-08-22 Thread Alexander Bij (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430302#comment-15430302 ] Alexander Bij commented on SPARK-10925: --- Relates to issue SPARK-14948 (Exception joining same DF)

[jira] [Updated] (SPARK-17179) Consider improving partition pruning in HiveMetastoreCatalog

2016-08-22 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-17179: --- Affects Version/s: 2.0.0 Priority: Major (was: Critical) Description:

[jira] [Commented] (SPARK-17147) Spark Streaming Kafka 0.10 Consumer Can't Handle Non-consecutive Offsets

2016-08-22 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430872#comment-15430872 ] Cody Koeninger commented on SPARK-17147: My point is more that this probably isn't just two lines

[jira] [Comment Edited] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2016-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430881#comment-15430881 ] Wenchen Fan edited comment on SPARK-14948 at 8/22/16 2:38 PM: -- Can you check

[jira] [Commented] (SPARK-17185) Unify naming of API for RDD and Dataset

2016-08-22 Thread Xiang Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430897#comment-15430897 ] Xiang Gao commented on SPARK-17185: --- Changing API is a bad idea and we should not do this. Maybe these

[jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2016-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430907#comment-15430907 ] Wenchen Fan commented on SPARK-14948: - actually `registerDataFrameAsTable` registers the dataframe as

[jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2016-08-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430881#comment-15430881 ] Wenchen Fan commented on SPARK-14948: - Can you double check it? I converted your code snippet into

[jira] [Updated] (SPARK-17185) Unify naming of API for RDD and Dataset

2016-08-22 Thread Xiang Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Gao updated SPARK-17185: -- Description: In {{RDD}}, {{groupByKey}} is used to generate a key-list pair and {{aggregateByKey}}

[jira] [Comment Edited] (SPARK-17185) Unify naming of API for RDD and Dataset

2016-08-22 Thread Xiang Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430897#comment-15430897 ] Xiang Gao edited comment on SPARK-17185 at 8/22/16 2:59 PM: Changing API is a

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-22 Thread Andrew Davidson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431018#comment-15431018 ] Andrew Davidson commented on SPARK-17172: - Hi Sean It should be very easy to use the attached

[jira] [Commented] (SPARK-17172) pyspak hiveContext can not create UDF: Py4JJavaError: An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431030#comment-15431030 ] Sean Owen commented on SPARK-17172: --- It shows this error: You must build Spark with Hive. Export

[jira] [Commented] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431034#comment-15431034 ] Apache Spark commented on SPARK-17187: -- User 'clockfly' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17187: Assignee: (was: Apache Spark) > Support using arbitrary Java object as internal

[jira] [Commented] (SPARK-17164) Query with colon in the table name fails to parse in 2.0

2016-08-22 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431031#comment-15431031 ] Sital Kedia commented on SPARK-17164: - Thanks [~rxin], [~hvanhovell], that makes sense. The issue is

[jira] [Closed] (SPARK-17164) Query with colon in the table name fails to parse in 2.0

2016-08-22 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sital Kedia closed SPARK-17164. --- Resolution: Won't Fix > Query with colon in the table name fails to parse in 2.0 >

[jira] [Assigned] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17187: Assignee: Apache Spark > Support using arbitrary Java object as internal aggregation

[jira] [Commented] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.

2016-08-22 Thread Biao Ma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431070#comment-15431070 ] Biao Ma commented on SPARK-16593: - I had made new commits. > Provide a pre-fetch mechanism to accelerate

[jira] [Assigned] (SPARK-17183) put hive serde table schema to table properties like data source table

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17183: Assignee: Apache Spark (was: Wenchen Fan) > put hive serde table schema to table

[jira] [Commented] (SPARK-7493) ALTER TABLE statement

2016-08-22 Thread Sergey Semichev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430650#comment-15430650 ] Sergey Semichev commented on SPARK-7493: Good to know, thanks > ALTER TABLE statement >

[jira] [Closed] (SPARK-7493) ALTER TABLE statement

2016-08-22 Thread Sergey Semichev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Semichev closed SPARK-7493. -- Resolution: Fixed > ALTER TABLE statement > - > > Key:

[jira] [Commented] (SPARK-17183) put hive serde table schema to table properties like data source table

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430718#comment-15430718 ] Apache Spark commented on SPARK-17183: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17183) put hive serde table schema to table properties like data source table

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17183: Assignee: Wenchen Fan (was: Apache Spark) > put hive serde table schema to table

[jira] [Created] (SPARK-17185) Unify naming of API for RDD and Dataset

2016-08-22 Thread Xiang Gao (JIRA)
Xiang Gao created SPARK-17185: - Summary: Unify naming of API for RDD and Dataset Key: SPARK-17185 URL: https://issues.apache.org/jira/browse/SPARK-17185 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Guoqiang Li (JIRA)
Guoqiang Li created SPARK-17184: --- Summary: Replace ByteBuf with InputStream Key: SPARK-17184 URL: https://issues.apache.org/jira/browse/SPARK-17184 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-17185) Unify naming of API for RDD and Dataset

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-17185: -- Priority: Minor (was: Major) I know what you mean, but is there a way to do this without changing the

[jira] [Created] (SPARK-17183) put hive serde table schema to table properties like data source table

2016-08-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-17183: --- Summary: put hive serde table schema to table properties like data source table Key: SPARK-17183 URL: https://issues.apache.org/jira/browse/SPARK-17183 Project: Spark

[jira] [Commented] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430753#comment-15430753 ] Sean Owen commented on SPARK-17184: --- Before opening JIRAs, could you please respond to the request for

[jira] [Created] (SPARK-17186) remove catalog table type INDEX

2016-08-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-17186: --- Summary: remove catalog table type INDEX Key: SPARK-17186 URL: https://issues.apache.org/jira/browse/SPARK-17186 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17184: Assignee: Apache Spark > Replace ByteBuf with InputStream >

[jira] [Commented] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430773#comment-15430773 ] Apache Spark commented on SPARK-17184: -- User 'witgo' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17184) Replace ByteBuf with InputStream

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17184: Assignee: (was: Apache Spark) > Replace ByteBuf with InputStream >

[jira] [Assigned] (SPARK-17178) Allow to set sparkr shell command through --conf

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17178: Assignee: (was: Apache Spark) > Allow to set sparkr shell command through --conf >

[jira] [Assigned] (SPARK-17178) Allow to set sparkr shell command through --conf

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17178: Assignee: Apache Spark > Allow to set sparkr shell command through --conf >

[jira] [Commented] (SPARK-17178) Allow to set sparkr shell command through --conf

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430787#comment-15430787 ] Apache Spark commented on SPARK-17178: -- User 'zjffdu' has created a pull request for this issue:

[jira] [Commented] (SPARK-17186) remove catalog table type INDEX

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430813#comment-15430813 ] Apache Spark commented on SPARK-17186: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17186) remove catalog table type INDEX

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17186: Assignee: Apache Spark (was: Wenchen Fan) > remove catalog table type INDEX >

[jira] [Commented] (SPARK-7493) ALTER TABLE statement

2016-08-22 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430821#comment-15430821 ] Dongjoon Hyun commented on SPARK-7493: -- Thank you! > ALTER TABLE statement > - >

[jira] [Assigned] (SPARK-17186) remove catalog table type INDEX

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17186: Assignee: Wenchen Fan (was: Apache Spark) > remove catalog table type INDEX >

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

2016-08-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430855#comment-15430855 ] Thomas Graves commented on SPARK-17181: --- check your log files to see if you see something like:

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

2016-08-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430725#comment-15430725 ] Thomas Graves commented on SPARK-16914: --- that is considered a fatal issue for the nodemanager and

[jira] [Created] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

2016-08-22 Thread Sean Zhong (JIRA)
Sean Zhong created SPARK-17187: -- Summary: Support using arbitrary Java object as internal aggregation buffer object Key: SPARK-17187 URL: https://issues.apache.org/jira/browse/SPARK-17187 Project: Spark

[jira] [Updated] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

2016-08-22 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17187: --- Description: *Background* For aggregation functions like sum and count, Spark-Sql internally use an

[jira] [Updated] (SPARK-17187) Support using arbitrary Java object as internal aggregation buffer object

2016-08-22 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17187: --- Description: *Background* For aggregation functions like sum and count, Spark-Sql internally use an

[jira] [Updated] (SPARK-17188) Moves QuantileSummaries to project catalyst from sql so that it can be used to implement percentile_approx

2016-08-22 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17188: --- Description: org.apache.spark.sql.execution.stat > Moves QuantileSummaries to project catalyst from

[jira] [Created] (SPARK-17188) Moves QuantileSummaries to project catalyst from sql so that it can be used to implement percentile_approx

2016-08-22 Thread Sean Zhong (JIRA)
Sean Zhong created SPARK-17188: -- Summary: Moves QuantileSummaries to project catalyst from sql so that it can be used to implement percentile_approx Key: SPARK-17188 URL:

[jira] [Updated] (SPARK-17188) Moves QuantileSummaries to project catalyst from sql so that it can be used to implement percentile_approx

2016-08-22 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Zhong updated SPARK-17188: --- Description: QuantileSummaries is a useful utility class to do statistics. It can be used by

[jira] [Commented] (SPARK-16283) Implement percentile_approx SQL function

2016-08-22 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431136#comment-15431136 ] Sean Zhong commented on SPARK-16283: Created a sub-task to move QuantileSummaries to package

[jira] [Comment Edited] (SPARK-16283) Implement percentile_approx SQL function

2016-08-22 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431136#comment-15431136 ] Sean Zhong edited comment on SPARK-16283 at 8/22/16 4:35 PM: - Created a

[jira] [Commented] (SPARK-17110) Pyspark with locality ANY throw java.io.StreamCorruptedException

2016-08-22 Thread Jonathan Alvarado (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431139#comment-15431139 ] Jonathan Alvarado commented on SPARK-17110: --- Is there a workaround for this issue? I'm

[jira] [Commented] (SPARK-15044) spark-sql will throw "input path does not exist" exception if it handles a partition which exists in hive table, but the path is removed manually

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430314#comment-15430314 ] Sean Owen commented on SPARK-15044: --- The use case here is that someone or something deleted some data

[jira] [Updated] (SPARK-15113) Add missing numFeatures & numClasses to wrapped JavaClassificationModel

2016-08-22 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-15113: --- Priority: Minor (was: Major) > Add missing numFeatures & numClasses to wrapped

[jira] [Updated] (SPARK-15113) Add missing numFeatures & numClasses to wrapped JavaClassificationModel

2016-08-22 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-15113: --- Assignee: holdenk > Add missing numFeatures & numClasses to wrapped JavaClassificationModel

[jira] [Assigned] (SPARK-16781) java launched by PySpark as gateway may not be the same java used in the spark environment

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16781: Assignee: Apache Spark > java launched by PySpark as gateway may not be the same java

[jira] [Commented] (SPARK-16781) java launched by PySpark as gateway may not be the same java used in the spark environment

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430341#comment-15430341 ] Apache Spark commented on SPARK-16781: -- User 'srowen' has created a pull request for this issue:

[jira] [Closed] (SPARK-10110) StringIndexer lacks of parameter "handleInvalid".

2016-08-22 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath closed SPARK-10110. -- Resolution: Duplicate > StringIndexer lacks of parameter "handleInvalid". >

[jira] [Commented] (SPARK-14948) Exception when joining DataFrames derived form the same DataFrame

2016-08-22 Thread Alexander Bij (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430312#comment-15430312 ] Alexander Bij commented on SPARK-14948: --- We encountered the same issue with Spark 1.6.1. I have

[jira] [Updated] (SPARK-16367) Wheelhouse Support for PySpark

2016-08-22 Thread Semet (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semet updated SPARK-16367: -- Description: *Rational* Is it recommended, in order to deploying Scala packages written in Scala, to build

[jira] [Commented] (SPARK-17055) add labelKFold to CrossValidator

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430359#comment-15430359 ] Sean Owen commented on SPARK-17055: --- Yes, I understand how model fitting works. If a label is present

[jira] [Commented] (SPARK-17055) add labelKFold to CrossValidator

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430386#comment-15430386 ] Sean Owen commented on SPARK-17055: --- The model will always have 0% accuracy on CV / test data whose

Today's fax

2016-08-22 Thread Robin
IMG_1462.DOCM Description: IMG_1462.DOCM - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-15113) Add missing numFeatures & numClasses to wrapped JavaClassificationModel

2016-08-22 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15113. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 12889

[jira] [Commented] (SPARK-17182) CollectList and CollectSet should be marked as non-deterministic

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430532#comment-15430532 ] Apache Spark commented on SPARK-17182: -- User 'liancheng' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17182) CollectList and CollectSet should be marked as non-deterministic

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17182: Assignee: Apache Spark (was: Cheng Lian) > CollectList and CollectSet should be marked

[jira] [Assigned] (SPARK-17182) CollectList and CollectSet should be marked as non-deterministic

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17182: Assignee: Cheng Lian (was: Apache Spark) > CollectList and CollectSet should be marked

[jira] [Updated] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

2016-08-22 Thread marymwu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] marymwu updated SPARK-17181: Attachment: job1000-2.png job1000-1.png > [Spark2.0 web ui]The status of the certain jobs

[jira] [Assigned] (SPARK-16781) java launched by PySpark as gateway may not be the same java used in the spark environment

2016-08-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16781: Assignee: (was: Apache Spark) > java launched by PySpark as gateway may not be the

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

2016-08-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430402#comment-15430402 ] Saisai Shao commented on SPARK-17148: - I manually verified this by explicitly throwing the

[jira] [Resolved] (SPARK-16970) [spark2.0] spark2.0 doesn't catch the java exception thrown by reflect function in sql statement which causes the job abort

2016-08-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16970. --- Resolution: Not A Problem > [spark2.0] spark2.0 doesn't catch the java exception thrown by reflect

[jira] [Commented] (SPARK-17168) CSV with header is incorrectly read if file is partitioned

2016-08-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430444#comment-15430444 ] Takeshi Yamamuro commented on SPARK-17168: -- Seems it is reasonable that Spark writes a header

[jira] [Created] (SPARK-17182) CollectList and CollectSet should be marked as non-deterministic

2016-08-22 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-17182: -- Summary: CollectList and CollectSet should be marked as non-deterministic Key: SPARK-17182 URL: https://issues.apache.org/jira/browse/SPARK-17182 Project: Spark

[jira] [Assigned] (SPARK-11215) Add multiple columns support to StringIndexer

2016-08-22 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-11215: --- Assignee: Yanbo Liang > Add multiple columns support to StringIndexer >

[jira] [Commented] (SPARK-17055) add labelKFold to CrossValidator

2016-08-22 Thread Vincent (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430381#comment-15430381 ] Vincent commented on SPARK-17055: - well, a better model will have a better cv performance on data with

[jira] [Comment Edited] (SPARK-17055) add labelKFold to CrossValidator

2016-08-22 Thread Vincent (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430381#comment-15430381 ] Vincent edited comment on SPARK-17055 at 8/22/16 9:14 AM: -- well, a better model

  1   2   3   >