[jira] [Created] (SPARK-9935) EqualNotNull not processed in ORC

2015-08-13 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-9935: --- Summary: EqualNotNull not processed in ORC Key: SPARK-9935 URL: https://issues.apache.org/jira/browse/SPARK-9935 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-9935) EqualNotNull not processed in ORC

2015-08-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9935: Component/s: SQL EqualNotNull not processed in ORC -

[jira] [Updated] (SPARK-9935) EqualNotNull not processed in OrcRelation

2015-08-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9935: Summary: EqualNotNull not processed in OrcRelation (was: EqualNotNull not processed in ORC)

[jira] [Updated] (SPARK-9935) EqualNotNull not processed in OrcRelation

2015-08-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9935: Description: it is an issue followed by SPARK-9814. Now datasources (after {{selectFilters()}} in

[jira] [Created] (SPARK-10035) Parquet filters does not process EqualNullSafe filter.

2015-08-16 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-10035: Summary: Parquet filters does not process EqualNullSafe filter. Key: SPARK-10035 URL: https://issues.apache.org/jira/browse/SPARK-10035 Project: Spark Issue

[jira] [Commented] (SPARK-10030) Managed memory leak detected when cache table

2015-08-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699000#comment-14699000 ] Hyukjin Kwon commented on SPARK-10030: -- Such use case issue (maybe rather question)

[jira] [Comment Edited] (SPARK-10030) Managed memory leak detected when cache table

2015-08-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699000#comment-14699000 ] Hyukjin Kwon edited comment on SPARK-10030 at 8/17/15 4:26 AM:

[jira] [Commented] (SPARK-10035) Parquet filters does not process EqualNullSafe filter.

2015-08-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699110#comment-14699110 ] Hyukjin Kwon commented on SPARK-10035: -- Assign this to me please :) [~lian cheng]

[jira] [Commented] (SPARK-10030) Managed memory leak detected when cache table

2015-08-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699180#comment-14699180 ] Hyukjin Kwon commented on SPARK-10030: -- My apologies for misunderstading the issue.

[jira] [Created] (SPARK-10180) JDBCRDD does not process EqualNullSafe filter.

2015-08-24 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-10180: Summary: JDBCRDD does not process EqualNullSafe filter. Key: SPARK-10180 URL: https://issues.apache.org/jira/browse/SPARK-10180 Project: Spark Issue Type:

[jira] [Updated] (SPARK-10180) JDBCRDD does not process EqualNullSafe filter.

2015-08-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-10180: - Description: Simply {{JDBCRelation}} passes EqualNullSafe (source.filter) but

[jira] [Updated] (SPARK-10180) JDBCRDD does not process EqualNullSafe filter.

2015-08-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-10180: - Description: Simply {{JDBCRelation}} passes EqualNullSafe source.filter but {{compileFilter()}}

[jira] [Issue Comment Deleted] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Comment: was deleted (was: I just made it. https://github.com/apache/spark/pull/8096)

[jira] [Commented] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681294#comment-14681294 ] Hyukjin Kwon commented on SPARK-9814: - I just made it :) EqualNotNull not passing to

[jira] [Commented] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681293#comment-14681293 ] Hyukjin Kwon commented on SPARK-9814: - I just made it.

[jira] [Created] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-9814: --- Summary: EqualNotNull not passing to data sources Key: SPARK-9814 URL: https://issues.apache.org/jira/browse/SPARK-9814 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Description: When data sources (such as Parquet) tries to filter data when reading from HDFS (not

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Description: When data sources (such as Parquet) tries to filter data when reading from HDFS (not

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Environment: (was: Centos 6.6) EqualNotNull not passing to data sources

[jira] [Updated] (SPARK-9814) EqualNotNull not passing to data sources

2015-08-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Component/s: (was: Input/Output) SQL EqualNotNull not passing to data sources

[jira] [Updated] (SPARK-9814) EqualNullSafe not passing to data sources

2015-08-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9814: Summary: EqualNullSafe not passing to data sources (was: EqualNotNull not passing to data sources)

[jira] [Updated] (SPARK-9935) EqualNullSafe not processed in OrcRelation

2015-08-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-9935: Summary: EqualNullSafe not processed in OrcRelation (was: EqualNotNull not processed in

[jira] [Commented] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000112#comment-15000112 ] Hyukjin Kwon commented on SPARK-10978: -- I am sorry to add some comments more here but.. are you sure

[jira] [Comment Edited] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000112#comment-15000112 ] Hyukjin Kwon edited comment on SPARK-10978 at 11/11/15 8:29 AM: I am

[jira] [Commented] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998014#comment-14998014 ] Hyukjin Kwon commented on SPARK-11621: -- I would like to work this. > ORC filter pushdown not

[jira] [Created] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-09 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11621: Summary: ORC filter pushdown not working properly after new unhandled filter interface. Key: SPARK-11621 URL: https://issues.apache.org/jira/browse/SPARK-11621

[jira] [Comment Edited] (SPARK-9557) Refactor ParquetFilterSuite and remove old ParquetFilters code

2015-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995922#comment-14995922 ] Hyukjin Kwon edited comment on SPARK-9557 at 11/9/15 2:24 AM: -- But I would

[jira] [Commented] (SPARK-9557) Refactor ParquetFilterSuite and remove old ParquetFilters code

2015-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995922#comment-14995922 ] Hyukjin Kwon commented on SPARK-9557: - But I would like to just add a just little corrections. I

[jira] [Commented] (SPARK-9557) Refactor ParquetFilterSuite and remove old ParquetFilters code

2015-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995912#comment-14995912 ] Hyukjin Kwon commented on SPARK-9557: - This is done by https://github.com/apache/spark/pull/8275. >

[jira] [Comment Edited] (SPARK-10113) Support for unsigned Parquet logical types

2015-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995992#comment-14995992 ] Hyukjin Kwon edited comment on SPARK-10113 at 11/9/15 3:50 AM: --- I think I

[jira] [Commented] (SPARK-10113) Support for unsigned Parquet logical types

2015-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995992#comment-14995992 ] Hyukjin Kwon commented on SPARK-10113: -- I think I can work on this but we need to make several

[jira] [Commented] (SPARK-10113) Support for unsigned Parquet logical types

2015-11-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996026#comment-14996026 ] Hyukjin Kwon commented on SPARK-10113: -- [~lian cheng] Just to double check.., Could I ask if this is

[jira] [Comment Edited] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998101#comment-14998101 ] Hyukjin Kwon edited comment on SPARK-10978 at 11/10/15 7:34 AM: I think

[jira] [Updated] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11621: - Description: After the new interface to get rid of filters predicate-push-downed which are

[jira] [Commented] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998089#comment-14998089 ] Hyukjin Kwon commented on SPARK-11621: -- Ops, I guess [~lian cheng] he must be working on this since

[jira] [Commented] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998101#comment-14998101 ] Hyukjin Kwon commented on SPARK-10978: -- I think we should add another interface such as {{def

[jira] [Comment Edited] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14998101#comment-14998101 ] Hyukjin Kwon edited comment on SPARK-10978 at 11/10/15 7:32 AM: I think

[jira] [Commented] (SPARK-11620) parquet.hadoop.ParquetOutputCommitter.commitJob() throws parquet.io.ParquetEncodingException

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001746#comment-15001746 ] Hyukjin Kwon commented on SPARK-11620: -- Can you tell me your Spark version? Spark 1.5.1 uses

[jira] [Updated] (SPARK-11687) Mixed usage of fold and foldLeft, reduce and reduceLeft and reduceOption and reduceLeftOption

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11687: - Description: As can be seen here

[jira] [Updated] (SPARK-11687) Mixed usage of fold and foldLeft, reduce and reduceLeft and reduceOption and reduceLeftOption

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11687: - Description: As can be seen here

[jira] [Created] (SPARK-11694) Parquet logical types are not being tested properly

2015-11-12 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11694: Summary: Parquet logical types are not being tested properly Key: SPARK-11694 URL: https://issues.apache.org/jira/browse/SPARK-11694 Project: Spark Issue

[jira] [Commented] (SPARK-11687) Mixed usage of fold and foldLeft, reduce and reduceLeft and reduceOption and reduceLeftOption

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001767#comment-15001767 ] Hyukjin Kwon commented on SPARK-11687: -- I think some might disagree with this. I would like to work

[jira] [Commented] (SPARK-11692) Support for Parquet logical types, JSON and BISON (embedded types)

2015-11-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001855#comment-15001855 ] Hyukjin Kwon commented on SPARK-11692: -- I will work on this. > Support for Parquet logical types,

[jira] [Updated] (SPARK-11692) Support for Parquet logical types, JSON and BSON (embedded types)

2015-11-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11692: - Summary: Support for Parquet logical types, JSON and BSON (embedded types) (was: Support for

[jira] [Created] (SPARK-11687) Mixed usage of fold and foldLeft, reduce and reduceLeft and reduceOption and reduceLeftOption

2015-11-11 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11687: Summary: Mixed usage of fold and foldLeft, reduce and reduceLeft and reduceOption and reduceLeftOption Key: SPARK-11687 URL: https://issues.apache.org/jira/browse/SPARK-11687

[jira] [Updated] (SPARK-11687) Mixed usage of fold and foldLeft, reduce and reduceLeft and reduceOption and reduceLeftOption

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11687: - Description: As can be seen here

[jira] [Created] (SPARK-11692) Support for Parquet logical types, JSON and BISON (embedded types)

2015-11-12 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11692: Summary: Support for Parquet logical types, JSON and BISON (embedded types) Key: SPARK-11692 URL: https://issues.apache.org/jira/browse/SPARK-11692 Project: Spark

[jira] [Commented] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001327#comment-15001327 ] Hyukjin Kwon commented on SPARK-11621: -- This is a duplicate (rather a sebset of)

[jira] [Comment Edited] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001327#comment-15001327 ] Hyukjin Kwon edited comment on SPARK-11621 at 11/11/15 11:30 PM: - This is

[jira] [Created] (SPARK-11676) Parquet filter tests all pass if filters are not really pushed down

2015-11-11 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11676: Summary: Parquet filter tests all pass if filters are not really pushed down Key: SPARK-11676 URL: https://issues.apache.org/jira/browse/SPARK-11676 Project: Spark

[jira] [Created] (SPARK-11677) ORC filter tests all pass if filters are actually not pushed down.

2015-11-11 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11677: Summary: ORC filter tests all pass if filters are actually not pushed down. Key: SPARK-11677 URL: https://issues.apache.org/jira/browse/SPARK-11677 Project: Spark

[jira] [Commented] (SPARK-11677) ORC filter tests all pass if filters are actually not pushed down.

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001391#comment-15001391 ] Hyukjin Kwon commented on SPARK-11677: -- I will work on this. > ORC filter tests all pass if

[jira] [Commented] (SPARK-11676) Parquet filter tests all pass if filters are not really pushed down

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001393#comment-15001393 ] Hyukjin Kwon commented on SPARK-11676: -- I will work on this. > Parquet filter tests all pass if

[jira] [Updated] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-11-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11621: - Description: After the new interface to get rid of filters predicate-push-downed which are

[jira] [Updated] (SPARK-11677) ORC filter tests all pass if filters are actually not pushed down.

2015-11-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11677: - Component/s: SQL > ORC filter tests all pass if filters are actually not pushed down. >

[jira] [Commented] (SPARK-11692) Support for Parquet logical types, JSON and BSON (embedded types)

2015-11-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008497#comment-15008497 ] Hyukjin Kwon commented on SPARK-11692: -- I mistakenly added this for backporting to branch 1.6. So, I

[jira] [Issue Comment Deleted] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-10978: - Comment: was deleted (was: I see. so, as far as I understand, for now and for the internal

[jira] [Commented] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999718#comment-14999718 ] Hyukjin Kwon commented on SPARK-10978: -- I see. so, as far as I understand, for now and for the

[jira] [Commented] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14999719#comment-14999719 ] Hyukjin Kwon commented on SPARK-10978: -- I see. so, as far as I understand, for now and for the

[jira] [Commented] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-11-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1434#comment-1434 ] Hyukjin Kwon commented on SPARK-10978: -- Thank you for the detailed response! > Allow

[jira] [Updated] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11500: - Description: When executing {{sqlContext.read.option("mergeSchema", "true").parquet(pathOne,

[jira] [Updated] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11500: - Description: When executing {{sqlContext.read.option("mergeSchema", "true").parquet(pathOne,

[jira] [Created] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-03 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11500: Summary: Not deterministic order of columns when using merging schemas. Key: SPARK-11500 URL: https://issues.apache.org/jira/browse/SPARK-11500 Project: Spark

[jira] [Commented] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14992841#comment-14992841 ] Hyukjin Kwon commented on SPARK-11500: -- Yes please :) > Not deterministic order of columns when

[jira] [Comment Edited] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989325#comment-14989325 ] Hyukjin Kwon edited comment on SPARK-11500 at 11/4/15 10:54 AM: It is

[jira] [Commented] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989325#comment-14989325 ] Hyukjin Kwon commented on SPARK-11500: -- It is non-deterministic and also incorrect in the output,

[jira] [Comment Edited] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989325#comment-14989325 ] Hyukjin Kwon edited comment on SPARK-11500 at 11/4/15 10:59 AM: It is

[jira] [Comment Edited] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989325#comment-14989325 ] Hyukjin Kwon edited comment on SPARK-11500 at 11/4/15 11:03 AM: It is

[jira] [Commented] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958879#comment-14958879 ] Hyukjin Kwon commented on SPARK-11103: -- For me, I think Spark should appropriately set filters for

[jira] [Commented] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957965#comment-14957965 ] Hyukjin Kwon commented on SPARK-11103: -- In this case, this should be fine because filter is not

[jira] [Commented] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964337#comment-14964337 ] Hyukjin Kwon commented on SPARK-11103: -- [~lian cheng] This looks clearly an issue and I made three

[jira] [Comment Edited] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964337#comment-14964337 ] Hyukjin Kwon edited comment on SPARK-11103 at 10/20/15 1:06 AM: [~lian

[jira] [Comment Edited] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956988#comment-14956988 ] Hyukjin Kwon edited comment on SPARK-11103 at 10/15/15 12:13 AM: - I

[jira] [Commented] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956988#comment-14956988 ] Hyukjin Kwon commented on SPARK-11103: -- I tested this case. The problem was, Parquet filters are

[jira] [Created] (SPARK-11044) Parquet writer version fixed as version1

2015-10-09 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-11044: Summary: Parquet writer version fixed as version1 Key: SPARK-11044 URL: https://issues.apache.org/jira/browse/SPARK-11044 Project: Spark Issue Type:

[jira] [Commented] (SPARK-11500) Not deterministic order of columns when using merging schemas.

2015-11-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008383#comment-15008383 ] Hyukjin Kwon commented on SPARK-11500: -- At the end, you were right. Although I didn't use

[jira] [Issue Comment Deleted] (SPARK-10030) Managed memory leak detected when cache table

2015-09-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-10030: - Comment: was deleted (was: Such use case issue (maybe rather question) should go to user mailing

[jira] [Commented] (SPARK-6982) Data Frame and Spark SQL should allow filtering on key portion of incremental parquet files

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046556#comment-15046556 ] Hyukjin Kwon commented on SPARK-6982: - Would this be done now by partition key [partition

[jira] [Comment Edited] (SPARK-6982) Data Frame and Spark SQL should allow filtering on key portion of incremental parquet files

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046556#comment-15046556 ] Hyukjin Kwon edited comment on SPARK-6982 at 12/8/15 8:15 AM: -- Would this be

[jira] [Comment Edited] (SPARK-6982) Data Frame and Spark SQL should allow filtering on key portion of incremental parquet files

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046556#comment-15046556 ] Hyukjin Kwon edited comment on SPARK-6982 at 12/8/15 8:15 AM: -- Would this be

[jira] [Commented] (SPARK-12124) Spark Sql MongoDB Cross Join Not Working

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037684#comment-15037684 ] Hyukjin Kwon commented on SPARK-12124: -- I think this might be asked to

[jira] [Created] (SPARK-12126) JDBC datasource processes filters only commonly pushed down.

2015-12-03 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-12126: Summary: JDBC datasource processes filters only commonly pushed down. Key: SPARK-12126 URL: https://issues.apache.org/jira/browse/SPARK-12126 Project: Spark

[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames are not passed through to jdbc source

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037670#comment-15037670 ] Hyukjin Kwon commented on SPARK-9182: - Filed here https://issues.apache.org/jira/browse/SPARK-12126.

[jira] [Updated] (SPARK-12126) JDBC datasource processes filters only commonly pushed down.

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12126: - Description: As suggested

[jira] [Updated] (SPARK-12236) JDBC filter tests all pass if filters are not really pushed down

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12236: - Description: It is similar with https://issues.apache.org/jira/browse/SPARK-11676 and

[jira] [Updated] (SPARK-12236) JDBC filter tests all pass if filters are not really pushed down

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12236: - Affects Version/s: 1.6.0 > JDBC filter tests all pass if filters are not really pushed down >

[jira] [Closed] (SPARK-11621) ORC filter pushdown not working properly after new unhandled filter interface.

2015-12-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-11621. Resolution: Fixed Fix Version/s: 1.6.0 > ORC filter pushdown not working properly after new

[jira] [Commented] (SPARK-9278) DataFrameWriter.insertInto inserts incorrect data

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047830#comment-15047830 ] Hyukjin Kwon commented on SPARK-9278: - The result might be definitely different as I ran the codes

[jira] [Comment Edited] (SPARK-9278) DataFrameWriter.insertInto inserts incorrect data

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047830#comment-15047830 ] Hyukjin Kwon edited comment on SPARK-9278 at 12/9/15 1:21 AM: -- The result

[jira] [Created] (SPARK-12236) JDBC filter tests all pass if filters are not really pushed down

2015-12-08 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-12236: Summary: JDBC filter tests all pass if filters are not really pushed down Key: SPARK-12236 URL: https://issues.apache.org/jira/browse/SPARK-12236 Project: Spark

[jira] [Commented] (SPARK-12236) JDBC filter tests all pass if filters are not really pushed down

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048199#comment-15048199 ] Hyukjin Kwon commented on SPARK-12236: -- I would like to work on this. > JDBC filter tests all pass

[jira] [Commented] (SPARK-12227) Support drop multiple columns specified by Column class in DataFrame API

2015-12-08 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048116#comment-15048116 ] Hyukjin Kwon commented on SPARK-12227: -- Could I quickly do this if it is not a big deal? I was

[jira] [Updated] (SPARK-12127) Spark sql Support Cross join with Mongo DB

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12127: - Environment: Linux Ubuntu 14.4 (was: Linex Ubuntu 14.4) > Spark sql Support Cross join with

[jira] [Commented] (SPARK-12127) Spark sql Support Cross join with Mongo DB

2015-12-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15039627#comment-15039627 ] Hyukjin Kwon commented on SPARK-12127: -- Is this a duplicate of

[jira] [Updated] (SPARK-12249) JDBC non-equality comparison operator not pushed down.

2015-12-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12249: - Summary: JDBC non-equality comparison operator not pushed down. (was: JDBC not-equality

[jira] [Created] (SPARK-12249) JDBC not-equality comparison operator not pushed down.

2015-12-09 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-12249: Summary: JDBC not-equality comparison operator not pushed down. Key: SPARK-12249 URL: https://issues.apache.org/jira/browse/SPARK-12249 Project: Spark Issue

[jira] [Comment Edited] (SPARK-12307) ParquetFormat options should be exposed through the DataFrameReader/Writer options API

2015-12-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057067#comment-15057067 ] Hyukjin Kwon edited comment on SPARK-12307 at 12/15/15 12:40 AM: - Just my

[jira] [Commented] (SPARK-12307) ParquetFormat options should be exposed through the DataFrameReader/Writer options API

2015-12-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057067#comment-15057067 ] Hyukjin Kwon commented on SPARK-12307: -- Just my personal though. I felt in the same way with this

[jira] [Created] (SPARK-12315) isnotnull operator not pushed down for JDBC datasource.

2015-12-13 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-12315: Summary: isnotnull operator not pushed down for JDBC datasource. Key: SPARK-12315 URL: https://issues.apache.org/jira/browse/SPARK-12315 Project: Spark

[jira] [Commented] (SPARK-12314) isnull operator not pushed down for JDBC datasource.

2015-12-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055344#comment-15055344 ] Hyukjin Kwon commented on SPARK-12314: -- I will work on this. > isnull operator not pushed down for

  1   2   3   4   5   6   7   8   9   10   >