[jira] [Updated] (SPARK-15444) Default value mismatch of param linkPredictionCol for GeneralizedLinearRegression

2016-05-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-15444: Description: There is a default value mismatch of param linkPredictionCol for

[jira] [Created] (SPARK-15444) Default value mismatch of param linkPredictionCol for GeneralizedLinearRegression

2016-05-20 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15444: --- Summary: Default value mismatch of param linkPredictionCol for GeneralizedLinearRegression Key: SPARK-15444 URL: https://issues.apache.org/jira/browse/SPARK-15444

[jira] [Updated] (SPARK-15444) Default value mismatch of param linkPredictionCol for GeneralizedLinearRegression

2016-05-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-15444: Priority: Blocker (was: Major) > Default value mismatch of param linkPredictionCol for

[jira] [Created] (SPARK-15342) PySpark test for non ascii column name does not actually test with unicode column name

2016-05-16 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15342: --- Summary: PySpark test for non ascii column name does not actually test with unicode column name Key: SPARK-15342 URL: https://issues.apache.org/jira/browse/SPARK-15342

[jira] [Created] (SPARK-15268) Make JavaTypeInference work with UDTRegistration

2016-05-11 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15268: --- Summary: Make JavaTypeInference work with UDTRegistration Key: SPARK-15268 URL: https://issues.apache.org/jira/browse/SPARK-15268 Project: Spark Issue

[jira] [Created] (SPARK-15180) Support subexpression elimination in Fliter

2016-05-06 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15180: --- Summary: Support subexpression elimination in Fliter Key: SPARK-15180 URL: https://issues.apache.org/jira/browse/SPARK-15180 Project: Spark Issue

[jira] [Commented] (SPARK-14906) Move VectorUDT and MatrixUDT in PySpark to new ML package

2016-05-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15267704#comment-15267704 ] Liang-Chi Hsieh commented on SPARK-14906: - ok. I will do it soon. > Move VectorUDT and MatrixUDT

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-04-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245427#comment-15245427 ] Liang-Chi Hsieh commented on SPARK-14083: - Based on [~joshrosen]'s code, I added some comments

[jira] [Closed] (SPARK-14432) Add API to calculate the approximate quantiles for multiple columns

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-14432. --- Resolution: Duplicate > Add API to calculate the approximate quantiles for multiple columns

[jira] [Closed] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-14627. --- Resolution: Won't Fix > Avoid shilfting encoder when delta is zero >

[jira] [Updated] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14627: Summary: Avoid shilfting encoder when delta is zero (was: In TypedAggregateExpression

[jira] [Updated] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-14627: Description: We can also improve encoder's shift method to return itself when shift delta

[jira] [Reopened] (SPARK-14627) Avoid shilfting encoder when delta is zero

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reopened SPARK-14627: - > Avoid shilfting encoder when delta is zero > -- >

[jira] [Closed] (SPARK-14627) In TypedAggregateExpression update method we call encoder.shift many times

2016-04-14 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-14627. --- Resolution: Won't Fix > In TypedAggregateExpression update method we call encoder.shift many

[jira] [Created] (SPARK-15211) Select features column from LibSVMRelation causes failure

2016-05-08 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15211: --- Summary: Select features column from LibSVMRelation causes failure Key: SPARK-15211 URL: https://issues.apache.org/jira/browse/SPARK-15211 Project: Spark

[jira] [Created] (SPARK-15225) Replace SQLContext with SparkSession in Encoder documentation

2016-05-09 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15225: --- Summary: Replace SQLContext with SparkSession in Encoder documentation Key: SPARK-15225 URL: https://issues.apache.org/jira/browse/SPARK-15225 Project: Spark

[jira] [Created] (SPARK-15240) Use buffer variables for update/merge expressions instead duplicate serialization/deserialization

2016-05-09 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-15240: --- Summary: Use buffer variables for update/merge expressions instead duplicate serialization/deserialization Key: SPARK-15240 URL:

[jira] [Updated] (SPARK-15240) Use buffer variables to improve buffer serialization/deserialization in TungstenAggregate

2016-05-09 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-15240: Summary: Use buffer variables to improve buffer serialization/deserialization in

[jira] [Commented] (SPARK-16628) OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

2016-07-26 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393436#comment-15393436 ] Liang-Chi Hsieh commented on SPARK-16628: - I submitted another PR to implement the option 2

[jira] [Created] (SPARK-16767) existsRecursively method in UserDefinedType is not correct

2016-07-28 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-16767: --- Summary: existsRecursively method in UserDefinedType is not correct Key: SPARK-16767 URL: https://issues.apache.org/jira/browse/SPARK-16767 Project: Spark

[jira] [Created] (SPARK-17056) Fix a wrong assert in MemoryStore

2016-08-14 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17056: --- Summary: Fix a wrong assert in MemoryStore Key: SPARK-17056 URL: https://issues.apache.org/jira/browse/SPARK-17056 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-16622) Fix NullPointerException when the returned value of the called method in Invoke is null

2016-07-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-16622: Summary: Fix NullPointerException when the returned value of the called method in Invoke

[jira] [Created] (SPARK-16622) Throws NullPointerException when the returned value of the called method in Invoke is null

2016-07-19 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-16622: --- Summary: Throws NullPointerException when the returned value of the called method in Invoke is null Key: SPARK-16622 URL: https://issues.apache.org/jira/browse/SPARK-16622

[jira] [Created] (SPARK-16640) Add codegen for Elt function

2016-07-19 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-16640: --- Summary: Add codegen for Elt function Key: SPARK-16640 URL: https://issues.apache.org/jira/browse/SPARK-16640 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-16628) OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

2016-07-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385586#comment-15385586 ] Liang-Chi Hsieh commented on SPARK-16628: - I've tried to address this issue by the PR with the

[jira] [Created] (SPARK-16849) Improve subquery execution by deduplicating the subqueries with the same results

2016-08-01 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-16849: --- Summary: Improve subquery execution by deduplicating the subqueries with the same results Key: SPARK-16849 URL: https://issues.apache.org/jira/browse/SPARK-16849

[jira] [Commented] (SPARK-16628) OrcConversions should not convert an ORC table represented by MetastoreRelation to HadoopFsRelation if metastore schema does not match schema stored in ORC files

2016-07-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389173#comment-15389173 ] Liang-Chi Hsieh commented on SPARK-16628: - I think it depends whether Hive also writes wrong

[jira] [Created] (SPARK-16362) Suport ArrayType and StructType in vectorization Parquet reader

2016-07-04 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-16362: --- Summary: Suport ArrayType and StructType in vectorization Parquet reader Key: SPARK-16362 URL: https://issues.apache.org/jira/browse/SPARK-16362 Project: Spark

[jira] [Updated] (SPARK-17107) Remove redundant pushdown rule for Union

2016-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17107: Summary: Remove redundant pushdown rule for Union (was: Remove redundant pushdown rule

[jira] [Created] (SPARK-17107) Remove redundant pushdown rule for set

2016-08-17 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17107: --- Summary: Remove redundant pushdown rule for set Key: SPARK-17107 URL: https://issues.apache.org/jira/browse/SPARK-17107 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17104) LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation

2016-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17104: Description: Currently LogicalRelation.newInstance() simply creates another

[jira] [Created] (SPARK-19433) ML Pipeline with long stages takes long time to finish

2017-02-01 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19433: --- Summary: ML Pipeline with long stages takes long time to finish Key: SPARK-19433 URL: https://issues.apache.org/jira/browse/SPARK-19433 Project: Spark

[jira] [Created] (SPARK-19443) The function to generate constraints takes too long when the query plan grows continuously

2017-02-02 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19443: --- Summary: The function to generate constraints takes too long when the query plan grows continuously Key: SPARK-19443 URL: https://issues.apache.org/jira/browse/SPARK-19443

[jira] [Updated] (SPARK-19443) The function to generate constraints takes too long when the query plan grows continuously

2017-02-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-19443: Description: This issue is originally reported and discussed at

[jira] [Updated] (SPARK-19433) ML Pipeline with long stages takes long time to finish

2017-02-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-19433: Description: This issue is originally reported and discussed at

[jira] [Updated] (SPARK-19425) Make ExtractEquiJoinKeys support UDT columns

2017-02-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-19425: Description: DataFrame.except doesn't work for UDT columns. It is because

[jira] [Updated] (SPARK-19425) Make ExtractEquiJoinKeys support UDT columns

2017-02-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-19425: Description: DataFrame.except doesn't work for UDT columns. It is because

[jira] (SPARK-19411) Remove the metadata used to mark optional columns in merged Parquet schema for filter predicate pushdown

2017-01-31 Thread Liang-Chi Hsieh (JIRA)
Title: Message Title Liang-Chi Hsieh created an issue

[jira] [Created] (SPARK-19425) Make df.except work for UDT

2017-02-01 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19425: --- Summary: Make df.except work for UDT Key: SPARK-19425 URL: https://issues.apache.org/jira/browse/SPARK-19425 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-19425) Make df.except work for UDT

2017-02-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848483#comment-15848483 ] Liang-Chi Hsieh commented on SPARK-19425: - I remember affects version can be None before. But

[jira] [Closed] (SPARK-18824) Add optimizer rule to reorder expensive Filter predicates like ScalaUDF

2017-02-07 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-18824. --- Resolution: Won't Fix > Add optimizer rule to reorder expensive Filter predicates like

[jira] [Created] (SPARK-19508) Improve error message when binding service fails

2017-02-07 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19508: --- Summary: Improve error message when binding service fails Key: SPARK-19508 URL: https://issues.apache.org/jira/browse/SPARK-19508 Project: Spark Issue

[jira] [Updated] (SPARK-19508) Improve error message when binding service fails

2017-02-07 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-19508: Description: Utils provides a helper function to bind service on port. This function can

[jira] [Updated] (SPARK-19425) Make ExtractEquiJoinKeys support UDT columns

2017-02-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-19425: Summary: Make ExtractEquiJoinKeys support UDT columns (was: Make df.except work for UDT)

[jira] [Closed] (SPARK-15180) Support subexpression elimination in Fliter

2017-02-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-15180. --- Resolution: Won't Fix > Support subexpression elimination in Fliter >

[jira] [Commented] (SPARK-15180) Support subexpression elimination in Fliter

2017-02-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851491#comment-15851491 ] Liang-Chi Hsieh commented on SPARK-15180: - [~hyukjin.kwon] Yes. I resolved this. Thanks! >

[jira] [Commented] (SPARK-15911) Remove additional Project to be consistent with SQL when insert into table

2017-02-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851488#comment-15851488 ] Liang-Chi Hsieh commented on SPARK-15911: - [~hyukjin.kwon] Thanks! > Remove additional Project

[jira] [Commented] (SPARK-18539) Cannot filter by nonexisting column in parquet file

2017-01-26 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840857#comment-15840857 ] Liang-Chi Hsieh commented on SPARK-18539: - [~lian cheng] Yea, I see. The term {{optional}} is

[jira] (SPARK-6307) Executers fetches the same rdd-block 100's or 1000's of times

2017-01-30 Thread Liang-Chi Hsieh (JIRA)
Title: Message Title Liang-Chi Hsieh commented on SPARK-6307

[jira] [Created] (SPARK-19355) Use map output statistices to improve global limit's parallelism

2017-01-24 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19355: --- Summary: Use map output statistices to improve global limit's parallelism Key: SPARK-19355 URL: https://issues.apache.org/jira/browse/SPARK-19355 Project:

[jira] [Commented] (SPARK-19653) `Vector` Type Should Be A First-Class Citizen In Spark SQL

2017-02-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872938#comment-15872938 ] Liang-Chi Hsieh commented on SPARK-19653: - Actually some Spark SQL functions like the mentioned

[jira] [Comment Edited] (SPARK-19653) `Vector` Type Should Be A First-Class Citizen In Spark SQL

2017-02-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872938#comment-15872938 ] Liang-Chi Hsieh edited comment on SPARK-19653 at 2/18/17 3:12 AM: --

[jira] [Commented] (SPARK-19217) Offer easy cast from vector to array

2017-02-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873159#comment-15873159 ] Liang-Chi Hsieh commented on SPARK-19217: - The native casting of UserDefinedType from/to other

[jira] [Commented] (SPARK-15678) Not use cache on appends and overwrites

2017-02-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884061#comment-15884061 ] Liang-Chi Hsieh commented on SPARK-15678: - [~kiszk][~gen] I created SPARK-19736 for the reported

[jira] [Commented] (SPARK-19352) Sorting issues on relatively big datasets

2017-02-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884016#comment-15884016 ] Liang-Chi Hsieh commented on SPARK-19352: - I think this is in fact solved by SPARK-19563.

[jira] [Created] (SPARK-19736) refreshByPath should clear all cached plans with the specified path

2017-02-24 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19736: --- Summary: refreshByPath should clear all cached plans with the specified path Key: SPARK-19736 URL: https://issues.apache.org/jira/browse/SPARK-19736 Project:

[jira] [Closed] (SPARK-19530) Use guava weigher for code cache eviction

2017-02-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-19530. --- Resolution: Won't Fix > Use guava weigher for code cache eviction >

[jira] [Created] (SPARK-19665) Improve constraint propagation

2017-02-20 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19665: --- Summary: Improve constraint propagation Key: SPARK-19665 URL: https://issues.apache.org/jira/browse/SPARK-19665 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-19493) Remove Java 7 support

2017-02-08 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859166#comment-15859166 ] Liang-Chi Hsieh commented on SPARK-19493: - +1 > Remove Java 7 support > - >

[jira] [Created] (SPARK-19530) Use guava weigher for code cache eviction

2017-02-08 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19530: --- Summary: Use guava weigher for code cache eviction Key: SPARK-19530 URL: https://issues.apache.org/jira/browse/SPARK-19530 Project: Spark Issue Type:

[jira] [Created] (SPARK-19244) Sort MemoryConsumers according to their memory usage when spilling

2017-01-16 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19244: --- Summary: Sort MemoryConsumers according to their memory usage when spilling Key: SPARK-19244 URL: https://issues.apache.org/jira/browse/SPARK-19244 Project:

[jira] [Created] (SPARK-19274) Make GlobalLimit without shuffling data to single partition

2017-01-18 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-19274: --- Summary: Make GlobalLimit without shuffling data to single partition Key: SPARK-19274 URL: https://issues.apache.org/jira/browse/SPARK-19274 Project: Spark

[jira] [Comment Edited] (SPARK-19311) UDFs disregard UDT type hierarchy

2017-01-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831888#comment-15831888 ] Liang-Chi Hsieh edited comment on SPARK-19311 at 1/20/17 3:06 PM: --

[jira] [Commented] (SPARK-19311) UDFs disregard UDT type hierarchy

2017-01-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831888#comment-15831888 ] Liang-Chi Hsieh commented on SPARK-19311: - [~Gregor Moehler] I think you already have the fixing.

[jira] [Closed] (SPARK-19274) Make GlobalLimit without shuffling data to single partition

2017-01-20 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-19274. --- Resolution: Won't Fix > Make GlobalLimit without shuffling data to single partition >

[jira] [Commented] (SPARK-19752) OrcGetSplits fails with 0 size files

2017-03-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890151#comment-15890151 ] Liang-Chi Hsieh commented on SPARK-19752: - Do you have a short example code that can reproduce

[jira] [Commented] (SPARK-19752) OrcGetSplits fails with 0 size files

2017-02-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889616#comment-15889616 ] Liang-Chi Hsieh commented on SPARK-19752: - >From the log, looks like it is a problem in Hive? >

[jira] [Updated] (SPARK-17357) Simplified predicates should be able to pushdown through operators because of the rule order in Optimizer

2016-09-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17357: Summary: Simplified predicates should be able to pushdown through operators because of the

[jira] [Created] (SPARK-17357) Simplified predicates should be able to pushdown through operators

2016-09-01 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17357: --- Summary: Simplified predicates should be able to pushdown through operators Key: SPARK-17357 URL: https://issues.apache.org/jira/browse/SPARK-17357 Project:

[jira] [Updated] (SPARK-17357) Simplified predicates can't be pushed down through operators because of the rule order in Optimizer

2016-09-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17357: Summary: Simplified predicates can't be pushed down through operators because of the rule

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-24 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518790#comment-15518790 ] Liang-Chi Hsieh commented on SPARK-17556: - For 1). It is true only if your driver is outside of

[jira] [Comment Edited] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-25 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522051#comment-15522051 ] Liang-Chi Hsieh edited comment on SPARK-17527 at 9/26/16 4:58 AM: -- Do

[jira] [Commented] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-25 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522051#comment-15522051 ] Liang-Chi Hsieh commented on SPARK-17527: - Do you have a small spark code snippet which can

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-30 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-30 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: (was: executor-side-broadcast.pdf) > Executor side broadcast for broadcast

[jira] [Updated] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-29 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17556: Attachment: executor-side-broadcast.pdf > Executor side broadcast for broadcast joins >

[jira] [Created] (SPARK-17821) Expression Canonicalization should support Add and Or

2016-10-06 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17821: --- Summary: Expression Canonicalization should support Add and Or Key: SPARK-17821 URL: https://issues.apache.org/jira/browse/SPARK-17821 Project: Spark

[jira] [Updated] (SPARK-17590) Analyze CTE definitions at once and allow CTE subquery to define CTE

2016-09-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17590: Summary: Analyze CTE definitions at once and allow CTE subquery to define CTE (was:

[jira] [Updated] (SPARK-17590) Analyze CTE definitions at once and allow CTE subquery to define CTE

2016-09-18 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17590: Description: We substitute logical plan with CTE definitions in the analyzer rule

[jira] [Created] (SPARK-17590) Analyze CTE definitions at once

2016-09-18 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17590: --- Summary: Analyze CTE definitions at once Key: SPARK-17590 URL: https://issues.apache.org/jira/browse/SPARK-17590 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-19 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503595#comment-15503595 ] Liang-Chi Hsieh commented on SPARK-17527: - Of course. > mergeSchema with `_OPTIONAL_` metadata

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh commented on SPARK-17556: - [~scwf]I already submitted a PR for this. Can you also

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh commented on SPARK-17556: - [~Fei Wang] I quickly go through your design doc.

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:49 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:48 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516490#comment-15516490 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:45 PM: --

[jira] [Comment Edited] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516478#comment-15516478 ] Liang-Chi Hsieh edited comment on SPARK-17556 at 9/23/16 1:45 PM: --

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516577#comment-15516577 ] Liang-Chi Hsieh commented on SPARK-17556: - In other words, from the jira description we say "the

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516568#comment-15516568 ] Liang-Chi Hsieh commented on SPARK-17556: - OK. You create the broadcast object on one executor.

[jira] [Commented] (SPARK-17556) Executor side broadcast for broadcast joins

2016-09-23 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516668#comment-15516668 ] Liang-Chi Hsieh commented on SPARK-17556: - No. It doesn't. I think the point is not only the

[jira] [Created] (SPARK-17574) Cache ShuffleExchange RDD when the exchange is reused

2016-09-17 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-17574: --- Summary: Cache ShuffleExchange RDD when the exchange is reused Key: SPARK-17574 URL: https://issues.apache.org/jira/browse/SPARK-17574 Project: Spark

[jira] [Commented] (SPARK-17527) mergeSchema with `_OPTIONAL_` metadata fails

2016-09-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498980#comment-15498980 ] Liang-Chi Hsieh commented on SPARK-17527: - Can you provide more hints about how to reproduce

[jira] [Updated] (SPARK-17574) Cache ShuffleExchange RDD when the exchange is reused

2016-09-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17574: Component/s: (was: SQL) Spark Core > Cache ShuffleExchange RDD when

[jira] [Closed] (SPARK-17574) Cache ShuffleExchange RDD when the exchange is reused

2016-09-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh closed SPARK-17574. --- Resolution: Won't Fix > Cache ShuffleExchange RDD when the exchange is reused >

[jira] [Updated] (SPARK-17104) LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation

2016-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17104: Summary: LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation

[jira] [Updated] (SPARK-17104) LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation

2016-08-17 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-17104: Description: Currently LogicalRelation.newInstance() simply creates another

<    1   2   3   4   5   6   7   8   9   10   >