[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2017-08-23 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138856#comment-16138856 ] Evan Chan commented on SPARK-12449: --- Andrew and others: Is there a plan to make this CatalystSource

[jira] [Updated] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2017-02-17 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evan Chan updated SPARK-13219: -- Hi Gagan, That is an interesting optimization but not the same one that Venu speaks of (I worked on

[jira] [Reopened] (SPARK-8133) sticky partitions

2016-11-14 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evan Chan reopened SPARK-8133: -- I think this is worth looking into again - for streaming. My team is creating spark streaming pipelines

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-10-20 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592429#comment-15592429 ] Evan Chan commented on SPARK-15687: --- [~kiszk] thanks for the PR... would you mind pointing me to the

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

2016-07-14 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376418#comment-15376418 ] Evan Chan edited comment on SPARK-15687 at 7/14/16 6:41 AM: [~rxin] I like

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-07-14 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376418#comment-15376418 ] Evan Chan commented on SPARK-15687: --- [~rxin] I like the idea of an Iterator[ColumnBatch]. I would

[jira] [Commented] (SPARK-13662) [SQL][Hive] Have SHOW TABLES return additional fields from Hive MetaStore

2016-04-17 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245129#comment-15245129 ] Evan Chan commented on SPARK-13662: --- Vijay, That would be awesome! Please go ahead. >

[jira] [Created] (SPARK-13662) [SQL][Hive] Have SHOW TABLES return additional fields from Hive MetaStore

2016-03-03 Thread Evan Chan (JIRA)
Evan Chan created SPARK-13662: - Summary: [SQL][Hive] Have SHOW TABLES return additional fields from Hive MetaStore Key: SPARK-13662 URL: https://issues.apache.org/jira/browse/SPARK-13662 Project: Spark

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-23 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160340#comment-15160340 ] Evan Chan commented on SPARK-13219: --- [~smilegator] [~doodlegum] Guys, let me explain the strategy

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-23 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160333#comment-15160333 ] Evan Chan commented on SPARK-13219: --- Sorry, could you explain how SPARK-12957 affects this one? >

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-22 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157292#comment-15157292 ] Evan Chan commented on SPARK-13219: --- [~smilegator] [~doodlegum] what is the URL to the latest patch?

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2016-02-17 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151823#comment-15151823 ] Evan Chan commented on SPARK-12449: --- I think in the case of sources.Expressions, by the time they are

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2016-02-17 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151345#comment-15151345 ] Evan Chan commented on SPARK-12449: --- [~stephank85] would you have any code to share? :D > Pushing

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2016-02-17 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151268#comment-15151268 ] Evan Chan commented on SPARK-12449: --- I agree with [~maxseiden] on a gradual approach to push more down

[jira] [Commented] (SPARK-12449) Pushing down arbitrary logical plans to data sources

2016-02-10 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141243#comment-15141243 ] Evan Chan commented on SPARK-12449: --- [~rxin] I agree with [~stephank85] and others that this would be a

[jira] [Commented] (SPARK-13219) Pushdown predicate propagation in SparkSQL with join

2016-02-08 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138024#comment-15138024 ] Evan Chan commented on SPARK-13219: --- [~smilegator] does your PR take care of the case where no JOIN

[jira] [Commented] (SPARK-12639) Improve Explain for DataSources with Handled Predicate Pushdowns

2016-01-07 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088017#comment-15088017 ] Evan Chan commented on SPARK-12639: --- +1 > Improve Explain for DataSources with Handled Predicate

[jira] [Commented] (SPARK-11838) Spark SQL query fragment RDD reuse

2016-01-06 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15086649#comment-15086649 ] Evan Chan commented on SPARK-11838: --- Based on everything that is said, it seems instead of having

[jira] [Commented] (SPARK-6703) Provide a way to discover existing SparkContext's

2015-04-15 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495777#comment-14495777 ] Evan Chan commented on SPARK-6703: -- I should note: Having the jobserver support generic

[jira] [Commented] (SPARK-6703) Provide a way to discover existing SparkContext's

2015-04-13 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493303#comment-14493303 ] Evan Chan commented on SPARK-6703: -- Hey folks, Thought I would just put in my 2 cents as

[jira] [Resolved] (SPARK-1283) Create spark-contrib repo for 1.0

2015-01-27 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evan Chan resolved SPARK-1283. -- Resolution: Won't Fix Yeah this is not needed anymore, with spark-packages.org Create spark-contrib

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

2014-10-13 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169545#comment-14169545 ] Evan Chan commented on SPARK-2593: -- Hmmm :( I believe Spark already uses a

[jira] [Commented] (SPARK-3298) [SQL] registerAsTable / registerTempTable overwrites old tables

2014-09-22 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143716#comment-14143716 ] Evan Chan commented on SPARK-3298: -- Sounds good, thanks! -Evan Never doubt that a small

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

2014-09-17 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136920#comment-14136920 ] Evan Chan commented on SPARK-2593: -- [~pwendell] I'd have to agree with Helena and

[jira] [Commented] (SPARK-3298) [SQL] registerAsTable / registerTempTable overwrites old tables

2014-09-02 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119373#comment-14119373 ] Evan Chan commented on SPARK-3298: -- I can't really think of a good way to prevent people

[jira] [Created] (SPARK-3295) [Spark SQL] schemaRdd1 ++ schemaRdd2 does not return another SchemaRdd

2014-08-29 Thread Evan Chan (JIRA)
Evan Chan created SPARK-3295: Summary: [Spark SQL] schemaRdd1 ++ schemaRdd2 does not return another SchemaRdd Key: SPARK-3295 URL: https://issues.apache.org/jira/browse/SPARK-3295 Project: Spark

[jira] [Created] (SPARK-3297) [Spark SQL][UI] SchemaRDD toString with many columns messes up Storage tab display

2014-08-29 Thread Evan Chan (JIRA)
Evan Chan created SPARK-3297: Summary: [Spark SQL][UI] SchemaRDD toString with many columns messes up Storage tab display Key: SPARK-3297 URL: https://issues.apache.org/jira/browse/SPARK-3297 Project:

[jira] [Created] (SPARK-3299) [SQL] Public API in SQLContext to list tables

2014-08-29 Thread Evan Chan (JIRA)
Evan Chan created SPARK-3299: Summary: [SQL] Public API in SQLContext to list tables Key: SPARK-3299 URL: https://issues.apache.org/jira/browse/SPARK-3299 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112931#comment-14112931 ] Evan Chan commented on SPARK-3215: -- [~vanzin] we should chat. I'm planning to move our

[jira] [Commented] (SPARK-2360) CSV import to SchemaRDDs

2014-08-21 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106200#comment-14106200 ] Evan Chan commented on SPARK-2360: -- +1 for this feature. I just had to write something

[jira] [Commented] (SPARK-2593) Add ability to pass an existing Akka ActorSystem into Spark

2014-07-22 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070476#comment-14070476 ] Evan Chan commented on SPARK-2593: -- I would say that the base SparkContext should have

[jira] [Commented] (SPARK-1283) Create spark-contrib repo for 1.0

2014-05-13 Thread Evan Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992974#comment-13992974 ] Evan Chan commented on SPARK-1283: -- ping Any more comments? Objections to creating a