[jira] [Created] (SPARK-18451) Always set -XX:+HeapDumpOnOutOfMemoryError for Spark tests

2016-11-15 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18451: -- Summary: Always set -XX:+HeapDumpOnOutOfMemoryError for Spark tests Key: SPARK-18451 URL: https://issues.apache.org/jira/browse/SPARK-18451 Project: Spark Issue

[jira] [Created] (SPARK-18403) ObjectHashAggregateSuite is being flaky (occasional OOM errors)

2016-11-10 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18403: -- Summary: ObjectHashAggregateSuite is being flaky (occasional OOM errors) Key: SPARK-18403 URL: https://issues.apache.org/jira/browse/SPARK-18403 Project: Spark

[jira] [Commented] (SPARK-18390) Optimized plan tried to use Cartesian join when it is not enabled

2016-11-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15652202#comment-15652202 ] Cheng Lian commented on SPARK-18390: I think this issue has already been fixed by SPA

[jira] [Updated] (SPARK-18390) Optimized plan tried to use Cartesian join when it is not enabled

2016-11-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18390: --- Description: {code} val df2 = spark.range(1e9.toInt).withColumn("one", lit(1)) val df3 = spark.range(

[jira] [Updated] (SPARK-18338) ObjectHashAggregateSuite fails under Maven builds

2016-11-07 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18338: --- Description: Test case initialization order under Maven and SBT are different. Maven always creates

[jira] [Created] (SPARK-18338) ObjectHashAggregateSuite fails under Maven builds

2016-11-07 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18338: -- Summary: ObjectHashAggregateSuite fails under Maven builds Key: SPARK-18338 URL: https://issues.apache.org/jira/browse/SPARK-18338 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-17972) Query planning slows down dramatically for large query plans even when sub-trees are cached

2016-11-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17972: --- Description: The following Spark shell snippet creates a series of query plans that grow exponential

[jira] [Resolved] (SPARK-11879) Checkpoint support for DataFrame/Dataset

2016-11-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-11879. Resolution: Duplicate > Checkpoint support for DataFrame/Dataset >

[jira] [Commented] (SPARK-11879) Checkpoint support for DataFrame/Dataset

2016-11-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630537#comment-15630537 ] Cheng Lian commented on SPARK-11879: Sorry that I didn't notice this ticket while wor

[jira] [Commented] (SPARK-18209) More robust view canonicalization without full SQL expansion

2016-11-01 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626823#comment-15626823 ] Cheng Lian commented on SPARK-18209: One problem of the proposed approach is that our

[jira] [Created] (SPARK-18186) Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support

2016-10-31 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18186: -- Summary: Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support Key: SPARK-18186 URL: https://issues.apache.org/jira/browse/SPARK-18186 Proj

[jira] [Commented] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602974#comment-15602974 ] Cheng Lian commented on SPARK-18053: Yea, reproduced using 2.0. > ARRAY equality is

[jira] [Commented] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602969#comment-15602969 ] Cheng Lian commented on SPARK-18053: Hm, the user mailing list thread said that it fa

[jira] [Created] (SPARK-18058) AnalysisException may be thrown when union two DFs whose struct fields have different nullability

2016-10-21 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18058: -- Summary: AnalysisException may be thrown when union two DFs whose struct fields have different nullability Key: SPARK-18058 URL: https://issues.apache.org/jira/browse/SPARK-18058

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Description: The new Tungsten execution engine has very robust memory management and speed for simpl

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Description: The new Tungsten execution engine has very robust memory management and speed for simpl

[jira] [Updated] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18053: --- Labels: correctness (was: ) > ARRAY equality is broken in Spark 2.0 > --

[jira] [Updated] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-18053: --- Description: The following Spark shell reproduces this issue: {code} case class Test(a: Seq[Int]) Seq

[jira] [Created] (SPARK-18053) ARRAY equality is broken in Spark 2.0

2016-10-21 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-18053: -- Summary: ARRAY equality is broken in Spark 2.0 Key: SPARK-18053 URL: https://issues.apache.org/jira/browse/SPARK-18053 Project: Spark Issue Type: Bug C

[jira] [Resolved] (SPARK-18012) Simplify WriterContainer code

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-18012. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15551 [https://github.

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Attachment: [Design Doc] Support for Arbitrary Aggregation States.pdf > Introduce a JVM object based

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Attachment: (was: [Design Doc] Support for Arbitrary Aggregation States.pdf) > Introduce a JVM o

[jira] [Updated] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17949: --- Attachment: [Design Doc] Support for Arbitrary Aggregation States.pdf > Introduce a JVM object based

[jira] [Created] (SPARK-17972) Query planning slows down dramatically for large query plans even when sub-trees are cached

2016-10-17 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-17972: -- Summary: Query planning slows down dramatically for large query plans even when sub-trees are cached Key: SPARK-17972 URL: https://issues.apache.org/jira/browse/SPARK-17972

[jira] [Assigned] (SPARK-17949) Introduce a JVM object based aggregate operator

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-17949: -- Assignee: Cheng Lian > Introduce a JVM object based aggregate operator > -

[jira] [Commented] (SPARK-10954) Parquet version in the "created_by" metadata field of Parquet files written by Spark 1.5 and 1.6 is wrong

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576623#comment-15576623 ] Cheng Lian commented on SPARK-10954: [~hyukjin.kwon], yes, confirmed. Thanks! > Parq

[jira] [Closed] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian closed SPARK-9783. - Resolution: Not A Problem This issue is no longer a problem since we re-implemented the JSON data source

[jira] [Commented] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576523#comment-15576523 ] Cheng Lian commented on SPARK-9783: --- Yes, I'm closing this. Thanks! > Use SqlNewHadoopR

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576513#comment-15576513 ] Cheng Lian commented on SPARK-17636: [~MasterDDT], yes, just as what [~hyukjin.kwon]

[jira] [Updated] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2016-10-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17636: --- Description: There's a *PushedFilters* for a simple numeric field, but not for a numeric field insid

[jira] [Comment Edited] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15561376#comment-15561376 ] Cheng Lian edited comment on SPARK-17845 at 10/10/16 6:43 AM: -

[jira] [Comment Edited] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15561376#comment-15561376 ] Cheng Lian edited comment on SPARK-17845 at 10/10/16 6:00 AM: -

[jira] [Commented] (SPARK-17845) Improve window function frame boundary API in DataFrame

2016-10-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15561376#comment-15561376 ] Cheng Lian commented on SPARK-17845: One thing is that ANSI SQL also allows using arb

[jira] [Commented] (SPARK-17725) Spark should not write out parquet files with schema containing non-nullable fields

2016-09-29 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533109#comment-15533109 ] Cheng Lian commented on SPARK-17725: Reproducing this issue by writing a Parquet file

[jira] [Resolved] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16516. Resolution: Fixed Issue resolved by pull request 14172 [https://github.com/apache/spark/pull/14172]

[jira] [Updated] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16777: --- Fix Version/s: (was: 2.2.0) 2.1.0 > Parquet schema converter depends on deprec

[jira] [Updated] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16516: --- Fix Version/s: 2.1.0 > Support for pushing down filters for decimal and timestamp types in ORC >

[jira] [Updated] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16516: --- Assignee: Hyukjin Kwon > Support for pushing down filters for decimal and timestamp types in ORC > --

[jira] [Updated] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16777: --- Fix Version/s: 2.1.0 2.0.2 > Parquet schema converter depends on deprecated APIs >

[jira] [Resolved] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16777. Resolution: Fixed Fix Version/s: (was: 2.0.2) (was: 2.1.0)

[jira] [Updated] (SPARK-16777) Parquet schema converter depends on deprecated APIs

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16777: --- Assignee: Hyukjin Kwon > Parquet schema converter depends on deprecated APIs > --

[jira] [Comment Edited] (SPARK-8824) Support Parquet time related logical types

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525313#comment-15525313 ] Cheng Lian edited comment on SPARK-8824 at 9/27/16 7:09 AM: Si

[jira] [Commented] (SPARK-8824) Support Parquet time related logical types

2016-09-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15525313#comment-15525313 ] Cheng Lian commented on SPARK-8824: --- Since we've already upgraded parquet-mr in Spark ma

[jira] [Commented] (SPARK-17572) Write.df is failing on spark cluster

2016-09-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15506051#comment-15506051 ] Cheng Lian commented on SPARK-17572: Yea, I know you are not using HDFS. But Spark al

[jira] [Commented] (SPARK-17572) Write.df is failing on spark cluster

2016-09-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505921#comment-15505921 ] Cheng Lian commented on SPARK-17572: Which version of Hadoop are you using? Does it w

[jira] [Updated] (SPARK-17572) Write.df is failing on spark cluster

2016-09-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17572: --- Description: Hi, We have spark cluster with four nodes, all four nodes have NFS partition shared(t

[jira] [Resolved] (SPARK-17289) Sort based partial aggregation breaks due to SPARK-12978

2016-08-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-17289. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14865 [https://github.

[jira] [Updated] (SPARK-17289) Sort based partial aggregation breaks due to SPARK-12978

2016-08-30 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-17289: --- Assignee: Takeshi Yamamuro > Sort based partial aggregation breaks due to SPARK-12978 > -

[jira] [Updated] (SPARK-16283) Implement percentile_approx SQL function

2016-08-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16283: --- Assignee: (was: Sean Zhong) > Implement percentile_approx SQL function >

[jira] [Updated] (SPARK-16283) Implement percentile_approx SQL function

2016-08-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16283: --- Assignee: Sean Zhong > Implement percentile_approx SQL function > ---

[jira] [Created] (SPARK-17182) CollectList and CollectSet should be marked as non-deterministic

2016-08-22 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-17182: -- Summary: CollectList and CollectSet should be marked as non-deterministic Key: SPARK-17182 URL: https://issues.apache.org/jira/browse/SPARK-17182 Project: Spark

[jira] [Resolved] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16975. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull reque

[jira] [Updated] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-12 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16975: --- Assignee: Dongjoon Hyun > Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

[jira] [Resolved] (SPARK-16867) createTable and alterTable in ExternalCatalog should not take db

2016-08-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16867. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14476 [https://github.

[jira] [Commented] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

2016-08-02 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403567#comment-15403567 ] Cheng Lian commented on SPARK-16842: First of all, the cost of schema discovery can b

[jira] [Updated] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16621: --- Assignee: Dongjoon Hyun > Generate stable SQLs in SQLBuilder > -- > >

[jira] [Resolved] (SPARK-16621) Generate stable SQLs in SQLBuilder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16621. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14257 [https://github.

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-1: --- Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only h

[jira] [Updated] (SPARK-16666) Kryo encoder for custom complex classes

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-1: --- Description: I'm trying to create a dataset with some geo data using spark and esri. If `Foo` only h

[jira] [Updated] (SPARK-16734) Make sure examples in all language bindings are consistent

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16734: --- Priority: Minor (was: Major) > Make sure examples in all language bindings are consistent >

[jira] [Resolved] (SPARK-16663) desc table should be consistent between data source and hive serde tables

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16663. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14302 [https://github.

[jira] [Created] (SPARK-16734) Make sure examples in all language bindings are consistent

2016-07-26 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-16734: -- Summary: Make sure examples in all language bindings are consistent Key: SPARK-16734 URL: https://issues.apache.org/jira/browse/SPARK-16734 Project: Spark Issue

[jira] [Resolved] (SPARK-16706) support java map in encoder

2016-07-26 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16706. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14344 [https://github.

[jira] [Updated] (SPARK-16698) json parsing regression - "." in keys

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16698: --- Assignee: Hyukjin Kwon > json parsing regression - "." in keys >

[jira] [Resolved] (SPARK-16698) json parsing regression - "." in keys

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16698. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Issue resolved by pull reque

[jira] [Updated] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16668: --- Assignee: Sameer Agarwal > Test parquet reader for row groups containing both dictionary and plain >

[jira] [Resolved] (SPARK-16668) Test parquet reader for row groups containing both dictionary and plain encoded pages

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16668. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14304 [https://github.

[jira] [Resolved] (SPARK-16691) move BucketSpec to catalyst module and use it in CatalogTable

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16691. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14331 [https://github.

[jira] [Resolved] (SPARK-16660) CreateViewCommand should not take CatalogTable

2016-07-25 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16660. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14297 [https://github.

[jira] [Updated] (SPARK-16703) Extra space in WindowSpecDefinition SQL representation

2016-07-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16703: --- Description: For a {{WindowSpecDefinition}} whose {{partitionSpec}} is empty, there's an extra space

[jira] [Updated] (SPARK-16703) Extra space in WindowSpecDefinition SQL representation

2016-07-24 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16703: --- Description: For a {{WindowSpecDefinition}} whose {{partitionSpec}} is empty, there's an extra space

[jira] [Created] (SPARK-16703) Extra space in WindowSpecDefinition SQL representation

2016-07-24 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-16703: -- Summary: Extra space in WindowSpecDefinition SQL representation Key: SPARK-16703 URL: https://issues.apache.org/jira/browse/SPARK-16703 Project: Spark Issue Type

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-22 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389141#comment-15389141 ] Cheng Lian commented on SPARK-16646: Could you please help check Hive's behavior here

[jira] [Commented] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-07-21 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387791#comment-15387791 ] Cheng Lian commented on SPARK-16632: Oh, I see, thanks for the explanation. > Vector

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387172#comment-15387172 ] Cheng Lian commented on SPARK-16646: Thanks for the help! I'm not working on this. >

[jira] [Updated] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16646: --- Reporter: Cheng Lian (was: liancheng) > LEAST doesn't accept numeric arguments with different data t

[jira] [Updated] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16646: --- Assignee: Hyukjin Kwon > LEAST doesn't accept numeric arguments with different data types > -

[jira] [Updated] (SPARK-16648) LAST_VALUE(FALSE) OVER () throws IndexOutOfBoundsException

2016-07-20 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16648: --- Reporter: Cheng Lian (was: liancheng) > LAST_VALUE(FALSE) OVER () throws IndexOutOfBoundsException >

[jira] [Commented] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-07-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385451#comment-15385451 ] Cheng Lian commented on SPARK-16632: Discussed with [~yhuai] after merging [PR #1427

[jira] [Updated] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-07-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16632: --- Assignee: Marcelo Vanzin > Vectorized parquet reader fails to read certain fields from Hive tables >

[jira] [Commented] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-07-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385399#comment-15385399 ] Cheng Lian commented on SPARK-16632: [~vanzin] Did you post the wrong stack trace? Th

[jira] [Resolved] (SPARK-16632) Vectorized parquet reader fails to read certain fields from Hive tables

2016-07-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16632. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14272 [https://github.

[jira] [Updated] (SPARK-16633) lag/lead does not return the default value when the offset row does not exist

2016-07-19 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16633: --- Attachment: window_function_bug.html JIRA went down right before [~yhuai] tried to upload the noteboo

[jira] [Commented] (SPARK-16576) Move plan SQL generation code from SQLBuilder into logical operators

2016-07-18 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381969#comment-15381969 ] Cheng Lian commented on SPARK-16576: [~rxin] I wrote the first version of the {{SQLBu

[jira] [Resolved] (SPARK-16529) SQLTestUtils.withTempDatabase should set `default` database before dropping

2016-07-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16529. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 14184 [https://github.

[jira] [Updated] (SPARK-16529) SQLTestUtils.withTempDatabase should set `default` database before dropping

2016-07-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16529: --- Assignee: Dongjoon Hyun > SQLTestUtils.withTempDatabase should set `default` database before dropping

[jira] [Resolved] (SPARK-16448) RemoveAliasOnlyProject should not remove alias with metadata

2016-07-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16448. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14106 [https://github.

[jira] [Updated] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

2016-07-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16343: --- Assignee: Jiang Xingbo > Improve the PushDownPredicate rule to pushdown predicates currectly in > no

[jira] [Updated] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

2016-07-14 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-16343: --- Affects Version/s: 2.0.0 > Improve the PushDownPredicate rule to pushdown predicates currectly in >

[jira] [Resolved] (SPARK-16343) Improve the PushDownPredicate rule to pushdown predicates currectly in non-deterministic condition

2016-07-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16343. Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 14012 [https://github.

[jira] [Resolved] (SPARK-16303) Update SQL examples and programming guide for Scala and Java language bindings

2016-07-13 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16303. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 14119 [https://github.

[jira] [Resolved] (SPARK-16381) Update SQL examples and programming guide for R language binding

2016-07-11 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-16381. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 14082 [https://github.

[jira] [Comment Edited] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+

2016-07-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369481#comment-15369481 ] Cheng Lian edited comment on SPARK-16344 at 7/10/16 8:07 AM: -

[jira] [Commented] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+

2016-07-10 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369481#comment-15369481 ] Cheng Lian commented on SPARK-16344: Thanks to [~rdblue]'s comment about why there're

[jira] [Commented] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+

2016-07-09 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369032#comment-15369032 ] Cheng Lian commented on SPARK-16344: I was re-thinking about [~rdblue]'s comment abov

[jira] [Comment Edited] (SPARK-16303) Update SQL examples and programming guide for Scala and Java language bindings

2016-07-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367345#comment-15367345 ] Cheng Lian edited comment on SPARK-16303 at 7/8/16 7:28 AM: T

[jira] [Comment Edited] (SPARK-16303) Update SQL examples and programming guide for Scala and Java language bindings

2016-07-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367345#comment-15367345 ] Cheng Lian edited comment on SPARK-16303 at 7/8/16 7:27 AM: T

[jira] [Commented] (SPARK-16303) Update SQL examples and programming guide for Scala and Java language bindings

2016-07-08 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367345#comment-15367345 ] Cheng Lian commented on SPARK-16303: Thanks for working on this! I'd suggest to send

[jira] [Comment Edited] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+

2016-07-07 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366069#comment-15366069 ] Cheng Lian edited comment on SPARK-16344 at 7/8/16 12:12 AM: -

[jira] [Commented] (SPARK-16344) Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+

2016-07-07 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366069#comment-15366069 ] Cheng Lian commented on SPARK-16344: Thanks for the detailed response! Spark SQL als

<    1   2   3   4   5   6   7   8   9   10   >