[jira] [Commented] (SPARK-15112) Dataset filter returns garbage

2016-05-06 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273837#comment-15273837 ] Cheng Lian commented on SPARK-15112: Actually there's another issue that contributes to this bug. The

[jira] [Updated] (SPARK-14803) A bug in EliminateSerialization rule in Catalyst Optimizer

2016-05-06 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14803: --- Description: When I rebased my PR https://github.com/apache/spark/pull/12493 to master, I found a

[jira] [Assigned] (SPARK-15112) Dataset filter returns garbage

2016-05-06 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-15112: -- Assignee: Cheng Lian > Dataset filter returns garbage > -- > >

[jira] [Updated] (SPARK-14139) Dataset loses nullability in operations with RowEncoder

2016-05-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14139: --- Assignee: Wenchen Fan > Dataset loses nullability in operations with RowEncoder >

[jira] [Resolved] (SPARK-14139) Dataset loses nullability in operations with RowEncoder

2016-05-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14139. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12364

[jira] [Updated] (SPARK-14933) Failed to create view out of a parquet or orc table

2016-05-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14933: --- Assignee: Xin Wu > Failed to create view out of a parquet or orc table >

[jira] [Commented] (SPARK-14139) Dataset loses nullability in operations with RowEncoder

2016-05-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272074#comment-15272074 ] Cheng Lian commented on SPARK-14139: As discussed in SPARK-15112. I think we should revert to use

[jira] [Created] (SPARK-15147) Catalog should have a property to indicate case-sensitivity

2016-05-05 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-15147: -- Summary: Catalog should have a property to indicate case-sensitivity Key: SPARK-15147 URL: https://issues.apache.org/jira/browse/SPARK-15147 Project: Spark

[jira] [Comment Edited] (SPARK-15112) Dataset filter returns garbage

2016-05-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271025#comment-15271025 ] Cheng Lian edited comment on SPARK-15112 at 5/5/16 12:28 AM: - The following

[jira] [Commented] (SPARK-15112) Dataset filter returns garbage

2016-05-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271025#comment-15271025 ] Cheng Lian commented on SPARK-15112: The following Spark shell session illustrates this issue:

[jira] [Resolved] (SPARK-14127) [Table related commands] Describe table

2016-05-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14127. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12844

[jira] [Resolved] (SPARK-14237) De-duplicate partition value appending logic in various buildReader() implementations

2016-05-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14237. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12866

[jira] [Created] (SPARK-14981) CatalogTable should contain sorting directions of sorting columns

2016-04-28 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14981: -- Summary: CatalogTable should contain sorting directions of sorting columns Key: SPARK-14981 URL: https://issues.apache.org/jira/browse/SPARK-14981 Project: Spark

[jira] [Updated] (SPARK-14954) Add PARTITIONED BY and CLUSTERED BY clause for data source CTAS syntax

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14954: --- Summary: Add PARTITIONED BY and CLUSTERED BY clause for data source CTAS syntax (was: Add PARTITION

[jira] [Updated] (SPARK-14427) Support persisting partitioned data source relations in Hive compatible format

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14427: --- Assignee: Liang-Chi Hsieh > Support persisting partitioned data source relations in Hive compatible

[jira] [Updated] (SPARK-14954) Add PARTITION BY and BUCKET BY clause for data source CTAS syntax

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14954: --- Summary: Add PARTITION BY and BUCKET BY clause for data source CTAS syntax (was: Add PARTITION BY

[jira] [Assigned] (SPARK-14954) Add PARTITION BY and BUCKET BY clause for "CREATE TABLE ... USING ..." syntax

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-14954: -- Assignee: Cheng Lian > Add PARTITION BY and BUCKET BY clause for "CREATE TABLE ... USING ..."

[jira] [Updated] (SPARK-14954) Add PARTITION BY and BUCKET BY clause for "CREATE TABLE ... USING ..." syntax

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14954: --- Affects Version/s: 2.0.0 Target Version/s: 2.0.0 > Add PARTITION BY and BUCKET BY clause for

[jira] [Created] (SPARK-14954) Add PARTITION BY and BUCKET BY clause for "CREATE TABLE ... USING ..." syntax

2016-04-27 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14954: -- Summary: Add PARTITION BY and BUCKET BY clause for "CREATE TABLE ... USING ..." syntax Key: SPARK-14954 URL: https://issues.apache.org/jira/browse/SPARK-14954 Project:

[jira] [Created] (SPARK-14953) LocalBackend should revive offers periodically

2016-04-27 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14953: -- Summary: LocalBackend should revive offers periodically Key: SPARK-14953 URL: https://issues.apache.org/jira/browse/SPARK-14953 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-14918) ExternalCatalog.TablePartitionSpec doesn't preserve partition column order

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14918. Resolution: Not A Problem > ExternalCatalog.TablePartitionSpec doesn't preserve partition column

[jira] [Commented] (SPARK-14918) ExternalCatalog.TablePartitionSpec doesn't preserve partition column order

2016-04-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259906#comment-15259906 ] Cheng Lian commented on SPARK-14918: Decided to leave this one as "not an issue" since the problem I

[jira] [Resolved] (SPARK-14445) Show columns/partitions

2016-04-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14445. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 1

[jira] [Comment Edited] (SPARK-13983) HiveThriftServer2 can not get "--hiveconf" or ''--hivevar" variables since 1.6 version (both multi-session and single session)

2016-04-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258139#comment-15258139 ] Cheng Lian edited comment on SPARK-13983 at 4/26/16 4:44 PM: - Here's my

[jira] [Commented] (SPARK-14918) ExternalCatalog.TablePartitionSpec doesn't preserve partition column order

2016-04-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258191#comment-15258191 ] Cheng Lian commented on SPARK-14918: Hit this issue while reviewing

[jira] [Created] (SPARK-14918) ExternalCatalog.TablePartitionSpec doesn't preserve partition column order

2016-04-26 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14918: -- Summary: ExternalCatalog.TablePartitionSpec doesn't preserve partition column order Key: SPARK-14918 URL: https://issues.apache.org/jira/browse/SPARK-14918 Project:

[jira] [Commented] (SPARK-13983) HiveThriftServer2 can not get "--hiveconf" or ''--hivevar" variables since 1.6 version (both multi-session and single session)

2016-04-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258139#comment-15258139 ] Cheng Lian commented on SPARK-13983: Here's my (incomplete) finding: Configurations set using

[jira] [Resolved] (SPARK-14875) OutputWriterFactory.newInstance shouldn't be private[sql]

2016-04-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14875. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12652

[jira] [Commented] (SPARK-14875) OutputWriterFactory.newInstance shouldn't be private[sql]

2016-04-23 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255249#comment-15255249 ] Cheng Lian commented on SPARK-14875: Checked with [~cloud_fan], it was accidentally made private

[jira] [Commented] (SPARK-14875) OutputWriterFactory.newInstance shouldn't be private[sql]

2016-04-23 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255248#comment-15255248 ] Cheng Lian commented on SPARK-14875: [~marmbrus] Is there any reason why we made it private in Spark

[jira] [Created] (SPARK-14875) OutputWriterFactory.newInstance shouldn't be private[sql]

2016-04-23 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14875: -- Summary: OutputWriterFactory.newInstance shouldn't be private[sql] Key: SPARK-14875 URL: https://issues.apache.org/jira/browse/SPARK-14875 Project: Spark Issue

[jira] [Resolved] (SPARK-14843) Error while encoding: java.lang.ClassCastException with LibSVMRelation

2016-04-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14843. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12611

[jira] [Updated] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-04-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13928: --- Target Version/s: 2.0.0 > Move org.apache.spark.Logging into org.apache.spark.internal.Logging >

[jira] [Updated] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-04-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13928: --- Assignee: Wenchen Fan > Move org.apache.spark.Logging into org.apache.spark.internal.Logging >

[jira] [Updated] (SPARK-13928) Move org.apache.spark.Logging into org.apache.spark.internal.Logging

2016-04-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13928: --- Fix Version/s: 2.0.0 > Move org.apache.spark.Logging into org.apache.spark.internal.Logging >

[jira] [Commented] (SPARK-14463) read.text broken for partitioned tables

2016-04-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240640#comment-15240640 ] Cheng Lian commented on SPARK-14463: Should we simply throw an exception when text data source is

[jira] [Commented] (SPARK-14389) OOM during BroadcastNestedLoopJoin

2016-04-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239687#comment-15239687 ] Cheng Lian commented on SPARK-14389: Exception thrown by UnsafeRow.copy() inside BNL join doesn't

[jira] [Commented] (SPARK-14495) Distinct aggregation cannot be used in the having clause

2016-04-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239674#comment-15239674 ] Cheng Lian commented on SPARK-14495: This ticket is for branch-1.6. > Distinct aggregation cannot be

[jira] [Commented] (SPARK-13753) Column nullable is derived incorrectly

2016-04-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239637#comment-15239637 ] Cheng Lian commented on SPARK-13753: [~jingweilu] Could you please provide the schema of tables

[jira] [Commented] (SPARK-14463) read.text broken for partitioned tables

2016-04-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239617#comment-15239617 ] Cheng Lian commented on SPARK-14463: Seems that this is because {{buildReader()}} doesn't append

[jira] [Comment Edited] (SPARK-14566) When appending to partitioned persisted table, we should apply a projection over input query plan using existing metastore schema

2016-04-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237690#comment-15237690 ] Cheng Lian edited comment on SPARK-14566 at 4/12/16 6:25 PM: - This bug is

[jira] [Commented] (SPARK-14566) When appending to partitioned persisted table, we should apply a projection over input query plan using existing metastore schema

2016-04-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237690#comment-15237690 ] Cheng Lian commented on SPARK-14566: This bug is exposed after fixing SPARK-14458. These two bugs

[jira] [Created] (SPARK-14566) When appending to partitioned persisted table, we should apply a projection over input query plan using existing metastore schema

2016-04-12 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14566: -- Summary: When appending to partitioned persisted table, we should apply a projection over input query plan using existing metastore schema Key: SPARK-14566 URL:

[jira] [Resolved] (SPARK-14493) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." should always be used with a user defined path

2016-04-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14493. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12303

[jira] [Resolved] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14488. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12303

[jira] [Commented] (SPARK-14372) Dataset.randomSplit() needs a Java version

2016-04-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234779#comment-15234779 ] Cheng Lian commented on SPARK-14372: Ah, sorry for the late reply. It's already taken by others. (At

[jira] [Resolved] (SPARK-14372) Dataset.randomSplit() needs a Java version

2016-04-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14372. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 12184

[jira] [Updated] (SPARK-14372) Dataset.randomSplit() needs a Java version

2016-04-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14372: --- Assignee: Rekha Joshi (was: Subhobrata Dey) > Dataset.randomSplit() needs a Java version >

[jira] [Updated] (SPARK-14372) Dataset.randomSplit() needs a Java version

2016-04-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14372: --- Assignee: Subhobrata Dey > Dataset.randomSplit() needs a Java version >

[jira] [Assigned] (SPARK-14476) Show table name or path in string of DataSourceScan

2016-04-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-14476: -- Assignee: Cheng Lian > Show table name or path in string of DataSourceScan >

[jira] [Created] (SPARK-14493) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." should always be used with a user defined path

2016-04-08 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14493: -- Summary: "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." should always be used with a user defined path Key: SPARK-14493 URL: https://issues.apache.org/jira/browse/SPARK-14493

[jira] [Commented] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232422#comment-15232422 ] Cheng Lian commented on SPARK-14488: Yea, that's why I came to this DDL command, because this command

[jira] [Comment Edited] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232414#comment-15232414 ] Cheng Lian edited comment on SPARK-14488 at 4/8/16 4:17 PM: Discussed with

[jira] [Commented] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232414#comment-15232414 ] Cheng Lian commented on SPARK-14488: Discussed with [~yhuai] offline, and here's the summary:

[jira] [Commented] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232401#comment-15232401 ] Cheng Lian commented on SPARK-14488: Ah, sorry, the logical plan class {{CreateTableUsingAsSelect}}

[jira] [Updated] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Description: The following Spark shell snippet reproduces this bug: {code} sqlContext range 10

[jira] [Updated] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Description: The following Spark shell snippet reproduces this bug: {code} sqlContext range 10

[jira] [Updated] (SPARK-14488) "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Summary: "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." creates persisted table (was: Weird

[jira] [Comment Edited] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232092#comment-15232092 ] Cheng Lian edited comment on SPARK-14488 at 4/8/16 12:27 PM: - Tried the same

[jira] [Updated] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Description: Currently, Spark 2.0 master allows DDL statements like {{CREATE TEMPORARY TABLE ...

[jira] [Commented] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232118#comment-15232118 ] Cheng Lian commented on SPARK-14488: Result of {{EXPLAIN EXTENDED CREATE TEMPORARY TABLE y USING

[jira] [Updated] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Description: Currently, Spark 2.0 master allows DDL statements like {{CREATE TEMPORARY TABLE ...

[jira] [Commented] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232111#comment-15232111 ] Cheng Lian commented on SPARK-14488: However, if {{TEMPORARY + USING + AS SELECT}} is an invalid

[jira] [Commented] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232092#comment-15232092 ] Cheng Lian commented on SPARK-14488: Tried the same snippet using Spark 1.6, and got the following

[jira] [Commented] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232078#comment-15232078 ] Cheng Lian commented on SPARK-14488: Updated title and description of this ticket. > Weird behavior

[jira] [Updated] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Description: Currently, Spark 2.0 master allows DDL statements like {{CREATE TEMPORARY TABLE ...

[jira] [Updated] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Description: Currently, Spark 2.0 master allows DDL statements like {{CREATE TEMPORARY TABLE ...

[jira] [Updated] (SPARK-14488) Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..."

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14488: --- Summary: Weird behavior of DDL "CREATE TEMPORARY TABLE ... USING ... AS SELECT ..." (was: Creating

[jira] [Comment Edited] (SPARK-14488) Creating temporary table using SQL DDL shouldn't write files to file system

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232030#comment-15232030 ] Cheng Lian edited comment on SPARK-14488 at 4/8/16 11:10 AM: - Oh, wait...

[jira] [Commented] (SPARK-14488) Creating temporary table using SQL DDL shouldn't write files to file system

2016-04-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15232030#comment-15232030 ] Cheng Lian commented on SPARK-14488: Oh, wait... Since there's a {{USING}} in the DDL statement, are

[jira] [Created] (SPARK-14488) Creating temporary table using SQL DDL shouldn't write files to file system

2016-04-08 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14488: -- Summary: Creating temporary table using SQL DDL shouldn't write files to file system Key: SPARK-14488 URL: https://issues.apache.org/jira/browse/SPARK-14488 Project:

[jira] [Created] (SPARK-14458) Wrong data schema is passed to FileFormat data sources that can't infer schema

2016-04-07 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14458: -- Summary: Wrong data schema is passed to FileFormat data sources that can't infer schema Key: SPARK-14458 URL: https://issues.apache.org/jira/browse/SPARK-14458 Project:

[jira] [Resolved] (SPARK-13589) Flaky test: ParquetHadoopFsRelationSuite.test all data types - ByteType

2016-04-06 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-13589. Resolution: Resolved Fixed by SPARK-13537 > Flaky test: ParquetHadoopFsRelationSuite.test all

[jira] [Created] (SPARK-14407) Hide HadoopFsRelation related data source API to execution package

2016-04-05 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14407: -- Summary: Hide HadoopFsRelation related data source API to execution package Key: SPARK-14407 URL: https://issues.apache.org/jira/browse/SPARK-14407 Project: Spark

[jira] [Resolved] (SPARK-14404) HDFSMetadataLogSuite overrides Hadoop FileSystem implementation but doesn't recover it

2016-04-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14404. Resolution: Not A Bug The scheme of {{FakeFileSystem}} is randomly generated, so it doesn't

[jira] [Created] (SPARK-14404) HDFSMetadataLogSuite overrides Hadoop FileSystem implementation but doesn't recover it

2016-04-05 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14404: -- Summary: HDFSMetadataLogSuite overrides Hadoop FileSystem implementation but doesn't recover it Key: SPARK-14404 URL: https://issues.apache.org/jira/browse/SPARK-14404

[jira] [Updated] (SPARK-14404) HDFSMetadataLogSuite overrides Hadoop FileSystem implementation but doesn't recover it

2016-04-05 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14404: --- Component/s: SQL > HDFSMetadataLogSuite overrides Hadoop FileSystem implementation but doesn't >

[jira] [Created] (SPARK-14372) Dataset.randomSplit() needs a Java version

2016-04-04 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14372: -- Summary: Dataset.randomSplit() needs a Java version Key: SPARK-14372 URL: https://issues.apache.org/jira/browse/SPARK-14372 Project: Spark Issue Type:

[jira] [Created] (SPARK-14369) Implement preferredLocations() for FileScanRDD

2016-04-04 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14369: -- Summary: Implement preferredLocations() for FileScanRDD Key: SPARK-14369 URL: https://issues.apache.org/jira/browse/SPARK-14369 Project: Spark Issue Type:

[jira] [Created] (SPARK-14295) buildReader implementation for LibSVM

2016-03-31 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14295: -- Summary: buildReader implementation for LibSVM Key: SPARK-14295 URL: https://issues.apache.org/jira/browse/SPARK-14295 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-14274) Add FileFormat.prepareRead to collect necessary global information

2016-03-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14274: --- Summary: Add FileFormat.prepareRead to collect necessary global information (was: Replaces

[jira] [Updated] (SPARK-14274) Replaces inferSchema with prepareRead to collect necessary global information

2016-03-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14274: --- Description: One problem of our newly introduced {{FileFormat.buildReader()}} method is that it

[jira] [Created] (SPARK-14274) Replaces inferSchema with prepareRead to collect necessary global information

2016-03-30 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14274: -- Summary: Replaces inferSchema with prepareRead to collect necessary global information Key: SPARK-14274 URL: https://issues.apache.org/jira/browse/SPARK-14274 Project:

[jira] [Created] (SPARK-14273) Add FileFormat.isSplittable to indicate whether a format is splittable

2016-03-30 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14273: -- Summary: Add FileFormat.isSplittable to indicate whether a format is splittable Key: SPARK-14273 URL: https://issues.apache.org/jira/browse/SPARK-14273 Project: Spark

[jira] [Resolved] (SPARK-14114) implement buildReader for text data source

2016-03-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14114. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11934

[jira] [Created] (SPARK-14244) Physical Window operator uses global SizeBasedWindowFunction.n attribute generated on both driver and executor side

2016-03-29 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14244: -- Summary: Physical Window operator uses global SizeBasedWindowFunction.n attribute generated on both driver and executor side Key: SPARK-14244 URL:

[jira] [Resolved] (SPARK-14208) Rename "spark.sql.parquet.fileScan"

2016-03-29 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-14208. Resolution: Fixed Issue resolved by pull request 12003

[jira] [Created] (SPARK-14237) De-duplicate partition value appending logic in various buildReader() implementations

2016-03-29 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14237: -- Summary: De-duplicate partition value appending logic in various buildReader() implementations Key: SPARK-14237 URL: https://issues.apache.org/jira/browse/SPARK-14237

[jira] [Updated] (SPARK-14206) buildReader implementation for CSV

2016-03-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14206: --- Affects Version/s: 2.0.0 Target Version/s: 2.0.0 Fix Version/s: (was: 2.0.0) >

[jira] [Assigned] (SPARK-14206) buildReader implementation for CSV

2016-03-28 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-14206: -- Assignee: Cheng Lian > buildReader implementation for CSV >

[jira] [Created] (SPARK-14208) Rename "spark.sql.parquet.fileScan"

2016-03-28 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14208: -- Summary: Rename "spark.sql.parquet.fileScan" Key: SPARK-14208 URL: https://issues.apache.org/jira/browse/SPARK-14208 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-14206) buildReader implementation for CSV

2016-03-28 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14206: -- Summary: buildReader implementation for CSV Key: SPARK-14206 URL: https://issues.apache.org/jira/browse/SPARK-14206 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-13456) Cannot create encoders for case classes defined in Spark shell after upgrading to Scala 2.11

2016-03-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-13456. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11931

[jira] [Updated] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases

2016-03-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14146: --- Affects Version/s: 2.0.0 Target Version/s: 2.0.0 Component/s: Spark Core

[jira] [Updated] (SPARK-14114) implement buildReader for text data source

2016-03-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-14114: --- Affects Version/s: 2.0.0 Target Version/s: 2.0.0 Fix Version/s: (was: 2.0.0) >

[jira] [Created] (SPARK-14116) buildReader implementation for ORC

2016-03-24 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-14116: -- Summary: buildReader implementation for ORC Key: SPARK-14116 URL: https://issues.apache.org/jira/browse/SPARK-14116 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-13549) Refactor the Optimizer Rule CollapseProject

2016-03-23 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-13549. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 11427

[jira] [Updated] (SPARK-13549) Refactor the Optimizer Rule CollapseProject

2016-03-23 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13549: --- Assignee: Xiao Li > Refactor the Optimizer Rule CollapseProject >

[jira] [Resolved] (SPARK-13772) DataType mismatch about decimal

2016-03-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-13772. Resolution: Fixed Fix Version/s: 1.6.2 Issue resolved by pull request 11605

[jira] [Updated] (SPARK-13772) DataType mismatch about decimal

2016-03-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13772: --- Assignee: cen yuhai > DataType mismatch about decimal > --- > >

<    1   2   3   4   5   6   7   8   9   10   >