[jira] [Commented] (SPARK-4782) Add inferSchema support for RDD[Map[String, Any]]

2015-06-01 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14567279#comment-14567279 ] Jianshi Huang commented on SPARK-4782: -- Thanks Luca for the clever fix! I also

[jira] [Created] (SPARK-8012) ArrayIndexOutOfBoundsException in SerializationDebugger

2015-06-01 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-8012: Summary: ArrayIndexOutOfBoundsException in SerializationDebugger Key: SPARK-8012 URL: https://issues.apache.org/jira/browse/SPARK-8012 Project: Spark Issue

[jira] [Created] (SPARK-8014) DataFrame.write.mode(error).save(...) should not scan the output folder

2015-06-01 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-8014: Summary: DataFrame.write.mode(error).save(...) should not scan the output folder Key: SPARK-8014 URL: https://issues.apache.org/jira/browse/SPARK-8014 Project: Spark

[jira] [Commented] (SPARK-8012) ArrayIndexOutOfBoundsException in SerializationDebugger

2015-06-01 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14568350#comment-14568350 ] Jianshi Huang commented on SPARK-8012: -- Yeah, it's from pretty big code base. I'm

[jira] [Commented] (SPARK-6297) EventLog permissions are always set to 770 which causes problems

2015-05-31 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566907#comment-14566907 ] Jianshi Huang commented on SPARK-6297: -- What about YARN? I have a problem to share

[jira] [Commented] (SPARK-7939) Make URL partition recognition return String by default for all partition column types and values

2015-05-29 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564412#comment-14564412 ] Jianshi Huang commented on SPARK-7939: -- That would be nice, also consider disabling

[jira] [Updated] (SPARK-7939) Make URL partition recognition return String by default for all partition column types and values

2015-05-29 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-7939: - Summary: Make URL partition recognition return String by default for all partition column types

[jira] [Created] (SPARK-7937) Cannot compare Hive named_struct. (when using argmax, argmin)

2015-05-29 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-7937: Summary: Cannot compare Hive named_struct. (when using argmax, argmin) Key: SPARK-7937 URL: https://issues.apache.org/jira/browse/SPARK-7937 Project: Spark

[jira] [Updated] (SPARK-7937) Cannot compare Hive named_struct. (when using argmax, argmin)

2015-05-29 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-7937: - Description: Imagine the following SQL: Intention: get last used bank account country.

[jira] [Updated] (SPARK-7937) Cannot compare Hive named_struct. (when using argmax, argmin)

2015-05-29 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-7937: - Description: Imagine the following SQL: Intention: get last used bank account country. ``` sql

[jira] [Created] (SPARK-7939) Make URL partition recognition return String by default for all partition column values

2015-05-29 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-7939: Summary: Make URL partition recognition return String by default for all partition column values Key: SPARK-7939 URL: https://issues.apache.org/jira/browse/SPARK-7939

[jira] [Commented] (SPARK-7937) Cannot compare Hive named_struct. (when using argmax, argmin)

2015-05-29 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564267#comment-14564267 ] Jianshi Huang commented on SPARK-7937: -- Blog for describing Hive's argmax, argmin

[jira] [Commented] (SPARK-6533) Allow using wildcard and other file pattern in Parquet DataSource

2015-05-19 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550028#comment-14550028 ] Jianshi Huang commented on SPARK-6533: -- Ah, right. Tested in 1.4.0, sqlc.load works!

[jira] [Resolved] (SPARK-6533) Allow using wildcard and other file pattern in Parquet DataSource

2015-05-19 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang resolved SPARK-6533. -- Resolution: Won't Fix Don't use sqlc.parquetFile(...), use sqlc.load(..., parquet) instead, or

[jira] [Commented] (SPARK-7614) CLONE - Master fails on 2.11 with compilation error

2015-05-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542596#comment-14542596 ] Jianshi Huang commented on SPARK-7614: -- Yeah, it seems I cannot reopen 7399. That's

[jira] [Commented] (SPARK-4356) Test Scala 2.11 on Jenkins

2015-05-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542603#comment-14542603 ] Jianshi Huang commented on SPARK-4356: -- When can we have 2.11 build tests in Jenkins?

[jira] [Commented] (SPARK-3056) Sort-based Aggregation

2015-05-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542148#comment-14542148 ] Jianshi Huang commented on SPARK-3056: -- Will [SPARK-2926] alone enough for this

[jira] [Created] (SPARK-7614) CLONE - Master fails on 2.11 with compilation error

2015-05-13 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-7614: Summary: CLONE - Master fails on 2.11 with compilation error Key: SPARK-7614 URL: https://issues.apache.org/jira/browse/SPARK-7614 Project: Spark Issue

[jira] [Commented] (SPARK-7399) Master fails on 2.11 with compilation error

2015-05-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542589#comment-14542589 ] Jianshi Huang commented on SPARK-7399: -- Looks like another change makes it broken

[jira] [Commented] (SPARK-6154) Support Kafka, JDBC in Scala 2.11

2015-05-12 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539617#comment-14539617 ] Jianshi Huang commented on SPARK-6154: -- Thanks Aniket, If you can submit the pull

[jira] [Commented] (SPARK-6154) Support Kafka, JDBC in Scala 2.11

2015-05-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534344#comment-14534344 ] Jianshi Huang commented on SPARK-6154: -- Do you mean we need to upgrade the jline

[jira] [Created] (SPARK-6561) Add partition support in saveAsParquet

2015-03-27 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6561: Summary: Add partition support in saveAsParquet Key: SPARK-6561 URL: https://issues.apache.org/jira/browse/SPARK-6561 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-6561) Add partition support in saveAsParquet

2015-03-27 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6561: - Description: Now ParquetRelation2 supports automatic partition discovery which is very nice.

[jira] [Created] (SPARK-6533) Cannot use wildcard and other file pattern in sqlContext.parquetFile if spark.sql.parquet.useDataSourceApi is not set to false

2015-03-25 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6533: Summary: Cannot use wildcard and other file pattern in sqlContext.parquetFile if spark.sql.parquet.useDataSourceApi is not set to false Key: SPARK-6533 URL:

[jira] [Updated] (SPARK-6533) Allow using wildcard and other file pattern in Parquet DataSource

2015-03-25 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6533: - Description: By default, spark.sql.parquet.useDataSourceApi is set to true. And loading parquet

[jira] [Updated] (SPARK-6533) Allow using wildcard and other file pattern in Parquet DataSource

2015-03-25 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6533: - Description: If spark.sql.parquet.useDataSourceApi is not set to false, which is the default.

[jira] [Created] (SPARK-6432) Cannot load parquet data with partitions if not all partition columns match data columns

2015-03-20 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6432: Summary: Cannot load parquet data with partitions if not all partition columns match data columns Key: SPARK-6432 URL: https://issues.apache.org/jira/browse/SPARK-6432

[jira] [Commented] (SPARK-6432) Cannot load parquet data with partitions if not all partition columns match data columns

2015-03-20 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371061#comment-14371061 ] Jianshi Huang commented on SPARK-6432: -- If no partition column appear in the data

[jira] [Updated] (SPARK-6432) Cannot load parquet data with partitions if not all partition columns match data columns

2015-03-20 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6432: - Description: Suppose we have a dataset in the following folder structure: {noformat}

[jira] [Updated] (SPARK-6382) withUDF(...) {...} for supporting temporary UDF definitions in the scope

2015-03-17 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6382: - Description: Currently the scope of UDF registration is global. It's unsuitable for libraries

[jira] [Updated] (SPARK-6382) withUDF(...) {...} for supporting temporary UDF definitions in the scope

2015-03-17 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6382: - Description: Currently the scope of UDF registration is global. It's unsuitable for libraries

[jira] [Created] (SPARK-6382) withUDF(...) {...} for supporting temporary UDF definitions in the scope

2015-03-17 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6382: Summary: withUDF(...) {...} for supporting temporary UDF definitions in the scope Key: SPARK-6382 URL: https://issues.apache.org/jira/browse/SPARK-6382 Project:

[jira] [Commented] (SPARK-6363) make scala 2.11 default language

2015-03-17 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365726#comment-14365726 ] Jianshi Huang commented on SPARK-6363: -- My two cents. The only module that's not

[jira] [Commented] (SPARK-6195) Specialized in-memory column type for decimal

2015-03-17 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365549#comment-14365549 ] Jianshi Huang commented on SPARK-6195: -- Like this optimization! :) Jianshi

[jira] [Commented] (SPARK-6277) Allow Hadoop configurations and env variables to be referenced in spark-defaults.conf

2015-03-11 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356478#comment-14356478 ] Jianshi Huang commented on SPARK-6277: -- I see. Not relating to hadoop config is fine,

[jira] [Created] (SPARK-6277) Allow Hadoop configurations and env variables to be referenced in spark-defaults.conf

2015-03-11 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6277: Summary: Allow Hadoop configurations and env variables to be referenced in spark-defaults.conf Key: SPARK-6277 URL: https://issues.apache.org/jira/browse/SPARK-6277

[jira] [Comment Edited] (SPARK-6201) INSET should coerce types

2015-03-09 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353248#comment-14353248 ] Jianshi Huang edited comment on SPARK-6201 at 3/9/15 5:40 PM: --

[jira] [Commented] (SPARK-6201) INSET should coerce types

2015-03-09 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353248#comment-14353248 ] Jianshi Huang commented on SPARK-6201: -- Implicit coercion outside the Numeric domain

[jira] [Comment Edited] (SPARK-6201) INSET should coerce types

2015-03-09 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353248#comment-14353248 ] Jianshi Huang edited comment on SPARK-6201 at 3/9/15 5:39 PM: --

[jira] [Commented] (SPARK-6154) Build error with Scala 2.11 for v1.3.0-rc2

2015-03-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352417#comment-14352417 ] Jianshi Huang commented on SPARK-6154: -- I see. Here's my build flag: -Pyarn -Phive

[jira] [Updated] (SPARK-6155) Support latest Scala (2.11.6+)

2015-03-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6155: - Summary: Support latest Scala (2.11.6+) (was: Support Scala 2.11.6+) Support latest Scala

[jira] [Updated] (SPARK-6155) Support Scala 2.11.5+

2015-03-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6155: - Issue Type: New Feature (was: Improvement) Support Scala 2.11.5+ -

[jira] [Updated] (SPARK-6155) Support Scala 2.11.5+

2015-03-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6155: - Priority: Major (was: Minor) Support Scala 2.11.5+ -

[jira] [Updated] (SPARK-6155) Support Scala 2.11.6+

2015-03-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6155: - Summary: Support Scala 2.11.6+ (was: Support Scala 2.11.5+) Support Scala 2.11.6+

[jira] [Created] (SPARK-6201) INSET should coerce types

2015-03-06 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6201: Summary: INSET should coerce types Key: SPARK-6201 URL: https://issues.apache.org/jira/browse/SPARK-6201 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-6201) INSET should coerce types

2015-03-06 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6201: - Description: Suppose we have the following table: {code} sqlc.jsonRDD(sc.parallelize(Seq({\a\:

[jira] [Updated] (SPARK-6201) INSET should coerce types

2015-03-06 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6201: - Description: Suppose we have the following table: {code} sqlc.jsonRDD(sc.parallelize(Seq({\a\:

[jira] [Updated] (SPARK-6201) INSET should coerce types

2015-03-06 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6201: - Description: Suppose we have the following table: {code} sqlc.jsonRDD(sc.parallelize(Seq({\a\:

[jira] [Comment Edited] (SPARK-5763) Sort-based Groupby and Join to resolve skewed data

2015-03-05 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348361#comment-14348361 ] Jianshi Huang edited comment on SPARK-5763 at 3/5/15 8:21 AM: --

[jira] [Commented] (SPARK-5763) Sort-based Groupby and Join to resolve skewed data

2015-03-05 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348361#comment-14348361 ] Jianshi Huang commented on SPARK-5763: -- Upvote for this improvement. Jianshi

[jira] [Commented] (SPARK-6155) Build with Scala 2.11.5 failed for Spark v1.3.0-rc2

2015-03-04 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347969#comment-14347969 ] Jianshi Huang commented on SPARK-6155: -- Yeah, please add the feature request. Just a

[jira] [Updated] (SPARK-6154) Build error with Scala 2.11 for v1.3.0-rc2

2015-03-04 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-6154: - Component/s: SQL Build error with Scala 2.11 for v1.3.0-rc2

[jira] [Created] (SPARK-6154) Build error with Scala 2.11 for v1.3.0-rc2

2015-03-03 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-6154: Summary: Build error with Scala 2.11 for v1.3.0-rc2 Key: SPARK-6154 URL: https://issues.apache.org/jira/browse/SPARK-6154 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-5828) Dynamic partition pattern support

2015-02-15 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-5828: Summary: Dynamic partition pattern support Key: SPARK-5828 URL: https://issues.apache.org/jira/browse/SPARK-5828 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-4279) Implementing TinkerPop on top of GraphX

2015-02-05 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308595#comment-14308595 ] Jianshi Huang commented on SPARK-4279: -- Anyone is working on this? Implementing

[jira] [Created] (SPARK-5446) Parquet column pruning should work for Map and Struct

2015-01-28 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-5446: Summary: Parquet column pruning should work for Map and Struct Key: SPARK-5446 URL: https://issues.apache.org/jira/browse/SPARK-5446 Project: Spark Issue

[jira] [Closed] (SPARK-4781) Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet external table in HiveContext)

2014-12-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang closed SPARK-4781. Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet

[jira] [Updated] (SPARK-4781) Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet external table in HiveContext)

2014-12-07 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4781: - Issue Type: Bug (was: Improvement) Column values become all NULL after doing ALTER TABLE CHANGE

[jira] [Created] (SPARK-4780) Support executing multiple statements in sql(...)

2014-12-06 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4780: Summary: Support executing multiple statements in sql(...) Key: SPARK-4780 URL: https://issues.apache.org/jira/browse/SPARK-4780 Project: Spark Issue Type:

[jira] [Created] (SPARK-4781) Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet external table in HiveContext)

2014-12-06 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4781: Summary: Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet external table in HiveContext) Key: SPARK-4781 URL:

[jira] [Updated] (SPARK-4781) Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet external table in HiveContext)

2014-12-06 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4781: - Description: I have a table say created like follows: CREATE EXTERNAL TABLE pmt (

[jira] [Updated] (SPARK-4781) Column values become all NULL after doing ALTER TABLE CHANGE for renaming column names (Parquet external table in HiveContext)

2014-12-06 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4781: - Description: I have a table say created like follows: {code} CREATE EXTERNAL TABLE pmt (

[jira] [Created] (SPARK-4782) Add inferSchema support for RDD[Map[String, Any]]

2014-12-06 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4782: Summary: Add inferSchema support for RDD[Map[String, Any]] Key: SPARK-4782 URL: https://issues.apache.org/jira/browse/SPARK-4782 Project: Spark Issue Type:

[jira] [Created] (SPARK-4760) ANALYZE TABLE table COMPUTE STATISTICS noscan failed estimating table size for tables created from Parquet files

2014-12-05 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4760: Summary: ANALYZE TABLE table COMPUTE STATISTICS noscan failed estimating table size for tables created from Parquet files Key: SPARK-4760 URL:

[jira] [Created] (SPARK-4757) Yarn-client failed to start due to Wrong FS error in distCacheMgr.addResource

2014-12-04 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4757: Summary: Yarn-client failed to start due to Wrong FS error in distCacheMgr.addResource Key: SPARK-4757 URL: https://issues.apache.org/jira/browse/SPARK-4757 Project:

[jira] [Created] (SPARK-4758) Make metastore_db in-memory for HiveContext

2014-12-04 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4758: Summary: Make metastore_db in-memory for HiveContext Key: SPARK-4758 URL: https://issues.apache.org/jira/browse/SPARK-4758 Project: Spark Issue Type:

[jira] [Updated] (SPARK-4758) Make metastore_db in-memory for HiveContext

2014-12-04 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4758: - Description: HiveContext by default will create a local folder metastore_db. This is not very

[jira] [Updated] (SPARK-4758) Make metastore_db in-memory for HiveContext

2014-12-04 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4758: - Description: HiveContext by default will create a local folder metastore_db. This is not very

[jira] [Created] (SPARK-4549) Support BigInt - Decimal in convertToCatalyst in SparkSQL

2014-11-21 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4549: Summary: Support BigInt - Decimal in convertToCatalyst in SparkSQL Key: SPARK-4549 URL: https://issues.apache.org/jira/browse/SPARK-4549 Project: Spark

[jira] [Updated] (SPARK-4549) Support BigInt - Decimal in convertToCatalyst in SparkSQL

2014-11-21 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4549: - Issue Type: Improvement (was: Bug) Support BigInt - Decimal in convertToCatalyst in SparkSQL

[jira] [Created] (SPARK-4551) Allow auto-conversion of field names of case class from camelCase to lower_case convention

2014-11-21 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4551: Summary: Allow auto-conversion of field names of case class from camelCase to lower_case convention Key: SPARK-4551 URL: https://issues.apache.org/jira/browse/SPARK-4551

[jira] [Closed] (SPARK-4199) Drop table if exists raises table not found exception in HiveContext

2014-11-03 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang closed SPARK-4199. Resolution: Invalid Drop table if exists raises table not found exception in HiveContext

[jira] [Commented] (SPARK-4199) Drop table if exists raises table not found exception in HiveContext

2014-11-03 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195045#comment-14195045 ] Jianshi Huang commented on SPARK-4199: -- Turned out it was caused by wrong version of

[jira] [Updated] (SPARK-4199) Drop table if exists raises table not found exception in HiveContext

2014-11-02 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-4199: - Summary: Drop table if exists raises table not found exception in HiveContext (was: Drop table

[jira] [Created] (SPARK-4199) Drop table if exists raises table not found exception

2014-11-02 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-4199: Summary: Drop table if exists raises table not found exception Key: SPARK-4199 URL: https://issues.apache.org/jira/browse/SPARK-4199 Project: Spark Issue

[jira] [Commented] (SPARK-3923) All Standalone Mode services time out with each other

2014-10-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169901#comment-14169901 ] Jianshi Huang commented on SPARK-3923: -- I have similar problem in YARN-client mode.

[jira] [Comment Edited] (SPARK-3923) All Standalone Mode services time out with each other

2014-10-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169901#comment-14169901 ] Jianshi Huang edited comment on SPARK-3923 at 10/13/14 8:45 PM:

[jira] [Created] (SPARK-3906) Support joins of multiple tables in SparkSQL (SQLContext, not HiveQL)

2014-10-11 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-3906: Summary: Support joins of multiple tables in SparkSQL (SQLContext, not HiveQL) Key: SPARK-3906 URL: https://issues.apache.org/jira/browse/SPARK-3906 Project: Spark

[jira] [Resolved] (SPARK-3845) SQLContext(...) should inherit configurations from SparkContext

2014-10-10 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang resolved SPARK-3845. -- Resolution: Fixed Fix Version/s: 1.2.0 SQLContext(...) should inherit configurations

[jira] [Commented] (SPARK-3845) SQLContext(...) should inherit configurations from SparkContext

2014-10-09 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14164882#comment-14164882 ] Jianshi Huang commented on SPARK-3845: -- Looks like it's fixed in latest 1.2.0

[jira] [Created] (SPARK-3845) SQLContext(...) should inherit configurations from SparkContext

2014-10-08 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-3845: Summary: SQLContext(...) should inherit configurations from SparkContext Key: SPARK-3845 URL: https://issues.apache.org/jira/browse/SPARK-3845 Project: Spark

[jira] [Created] (SPARK-3846) KryoException when doing joins in SparkSQL

2014-10-08 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-3846: Summary: KryoException when doing joins in SparkSQL Key: SPARK-3846 URL: https://issues.apache.org/jira/browse/SPARK-3846 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-3846) KryoException when doing joins in SparkSQL

2014-10-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-3846: - Description: The error is reproducible when I join two tables manually. The error message

[jira] [Closed] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-10-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang closed SPARK-2890. Spark SQL should allow SELECT with duplicated columns

[jira] [Updated] (SPARK-3846) KryoException when doing joins in SparkSQL

2014-10-08 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang updated SPARK-3846: - Description: The error is reproducible when I join two tables manually. The error message is

[jira] [Commented] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14095224#comment-14095224 ] Jianshi Huang commented on SPARK-2890: -- I think the fault is on my side. I should've

[jira] [Closed] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-13 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianshi Huang closed SPARK-2890. Resolution: Invalid Spark SQL should allow SELECT with duplicated columns

[jira] [Commented] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-11 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093746#comment-14093746 ] Jianshi Huang commented on SPARK-2890: -- My use case: The result will be parsed into

[jira] [Commented] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-07 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088939#comment-14088939 ] Jianshi Huang commented on SPARK-2890: -- In previous versions, there was no warnings

[jira] [Created] (SPARK-2890) Spark SQL should allow SELECT with duplicated columns

2014-08-06 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-2890: Summary: Spark SQL should allow SELECT with duplicated columns Key: SPARK-2890 URL: https://issues.apache.org/jira/browse/SPARK-2890 Project: Spark Issue

[jira] [Commented] (SPARK-2728) Integer overflow in partition index calculation RangePartitioner

2014-07-31 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080729#comment-14080729 ] Jianshi Huang commented on SPARK-2728: -- I see. Thanks for the fix Sean and Larry!

[jira] [Commented] (SPARK-2728) Integer overflow in partition index calculation RangePartitioner

2014-07-31 Thread Jianshi Huang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080741#comment-14080741 ] Jianshi Huang commented on SPARK-2728: -- Anyone can test it? I'll close the issue. My

[jira] [Created] (SPARK-2728) Integer overflow in partition index calculation RangePartitioner

2014-07-29 Thread Jianshi Huang (JIRA)
Jianshi Huang created SPARK-2728: Summary: Integer overflow in partition index calculation RangePartitioner Key: SPARK-2728 URL: https://issues.apache.org/jira/browse/SPARK-2728 Project: Spark