[GitHub] spark pull request: [SPARK-3609][SQL] Adds sizeInBytes statistics ...

2014-09-19 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/2468#discussion_r17817992 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala --- @@ -122,6 +122,16 @@ object NativeType

[GitHub] spark pull request: Fix flakey HiveQuerySuite test

2014-07-21 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1514#issuecomment-49667765 Thanks for the fix. Looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2561][SQL] Fix apply schema

2014-07-21 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1470#issuecomment-49686947 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-2393][SQL] Cost estimation optimization...

2014-07-22 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r15243150 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala --- @@ -66,8 +66,8 @@ abstract class SparkPlan extends QueryPlan

[GitHub] spark pull request: [SPARK-2393][SQL] Cost estimation optimization...

2014-07-22 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r15243796 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -26,6 +26,26 @@ import

[GitHub] spark pull request: [SPARK-2393][SQL] Cost estimation optimization...

2014-07-22 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1238#issuecomment-49785200 I have addressed the latest round of review comments rebased onto latest master. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-2531] [SQL] Make BroadcastNestedLoopJoi...

2014-07-23 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1448#issuecomment-49911126 Thanks for the comments @chenghao-intel and @marmbrus. As Michael said I'll revisit this after the codegen PR. --- If your project is set up for it, you can

[GitHub] spark pull request: [WIP][SPARK-2054][SQL] Code Generation for Exp...

2014-07-23 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/993#discussion_r15311508 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateProjection.scala --- @@ -0,0 +1,218

[GitHub] spark pull request: [SPARK-2675] Increase EVENT_QUEUE_CAPACITY by ...

2014-07-24 Thread concretevitamin
GitHub user concretevitamin opened a pull request: https://github.com/apache/spark/pull/1579 [SPARK-2675] Increase EVENT_QUEUE_CAPACITY by 20x. JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2675 @pwendell @andrewor You can merge this pull request into a Git

[GitHub] spark pull request: [SPARK-2675] Increase EVENT_QUEUE_CAPACITY by ...

2014-07-24 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1579#issuecomment-50065677 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2675] Increase EVENT_QUEUE_CAPACITY by ...

2014-07-25 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1579#issuecomment-50183155 Unfortunately I may not be able to find time to run an experiment soon. If anyone is interested, I think `org.apache.spark.util.SizeEstimator` is reasonable

[GitHub] spark pull request: [SPARK-2410][SQL] Merging Hive Thrift/JDBC ser...

2014-07-26 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1600#issuecomment-50224519 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2674] [SQL] [PySpark] support datetime ...

2014-07-26 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1601#discussion_r15434945 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SchemaRDD.scala --- @@ -395,6 +395,11 @@ class SchemaRDD

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-28 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1346#issuecomment-50423690 @yhuai @marmbrus I am not sure if this has been discussed before, but what do you guys think about adding a version of `applySchema(RDD[Array[String

[GitHub] spark pull request: [SPARK-2179][SQL] Public API for DataTypes and...

2014-07-28 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1346#issuecomment-50423851 To add to this: for my own purpose, I can certainly hack something together based off this branch in a custom Spark build, but just want to throw this thought

[GitHub] spark pull request: [SPARK-2393][SQL] Cost estimation optimization...

2014-07-29 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1238#issuecomment-50522265 Rebased addressed review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2393][SQL] Cost estimation optimization...

2014-07-29 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1238#issuecomment-50522286 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-2531] [SQL] Make BroadcastNestedLoopJoi...

2014-07-30 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1448#discussion_r15599752 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -332,71 +342,88 @@ case class BroadcastNestedLoopJoin

[GitHub] spark pull request: [SPARK-2531 SPARK-2436] [SQL] Optimize the B...

2014-07-31 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1448#issuecomment-50807987 I have rebased made changes according to the previous review comments. Also updated the title and description of the PR, combining two JIRA tickets

[GitHub] spark pull request: [SPARK-2316] Avoid O(blocks) operations in lis...

2014-07-31 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1679#issuecomment-50827351 This patch is much appreciated -- thanks for working on this! On Thu, Jul 31, 2014 at 3:02 PM, Apache Spark QA notificati...@github.com wrote

[GitHub] spark pull request: [SPARK-2316] Avoid O(blocks) operations in lis...

2014-08-01 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1679#issuecomment-50946639 How many listeners are used in these benchmarks? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-2675] Increase EVENT_QUEUE_CAPACITY by ...

2014-08-02 Thread concretevitamin
Github user concretevitamin closed the pull request at: https://github.com/apache/spark/pull/1579 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request: [SPARK-2783][SQL] Basic support for analyze in...

2014-08-03 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1741#discussion_r15733255 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala --- @@ -21,12 +21,15 @@ import java.io.{BufferedReader, File

[GitHub] spark pull request: [SPARK-2783][SQL] Basic support for analyze in...

2014-08-03 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1741#discussion_r15733260 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -280,7 +281,7 @@ private[hive] case class

[GitHub] spark pull request: [SPARK-2783][SQL] Basic support for analyze in...

2014-08-03 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1741#discussion_r15733265 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -280,7 +281,7 @@ private[hive] case class

[GitHub] spark pull request: [SPARK-2783][SQL] Basic support for analyze in...

2014-08-03 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1741#discussion_r15733274 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala --- @@ -92,6 +95,64 @@ class HiveContext(sc: SparkContext) extends

[GitHub] spark pull request: [SPARK-2783][SQL] Basic support for analyze in...

2014-08-03 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1741#discussion_r15733277 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala --- @@ -92,6 +95,64 @@ class HiveContext(sc: SparkContext) extends

[GitHub] spark pull request: [SPARK-2179] [SQL] Public API for DataTypes an...

2014-08-04 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1774#discussion_r15790429 --- Diff: docs/sql-programming-guide.md --- @@ -152,6 +152,41 @@ val teenagers = sqlContext.sql(SELECT name FROM people WHERE age = 13 AND age

[GitHub] spark pull request: [SPARK-2179] [SQL] Public API for DataTypes an...

2014-08-04 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1774#discussion_r15790441 --- Diff: docs/sql-programming-guide.md --- @@ -152,6 +152,41 @@ val teenagers = sqlContext.sql(SELECT name FROM people WHERE age = 13 AND age

[GitHub] spark pull request: [SPARK-2179] [SQL] Public API for DataTypes an...

2014-08-04 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1774#discussion_r15790528 --- Diff: docs/sql-programming-guide.md --- @@ -225,6 +260,54 @@ ListString teenagerNames = teenagers.map(new FunctionRow, String

[GitHub] spark pull request: [SPARK-2179] [SQL] Public API for DataTypes an...

2014-08-04 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1774#discussion_r15790538 --- Diff: docs/sql-programming-guide.md --- @@ -259,6 +342,40 @@ for teenName in teenNames.collect(): print teenName {% endhighlight

[GitHub] spark pull request: [SPARK-2860][SQL] Fix coercion of CASE WHEN.

2014-08-05 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1785#discussion_r15828435 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala --- @@ -336,28 +338,33 @@ trait HiveTypeCoercion

[GitHub] spark pull request: [SPARK-2860][SQL] Fix coercion of CASE WHEN.

2014-08-05 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1785#discussion_r15828596 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala --- @@ -336,28 +338,33 @@ trait HiveTypeCoercion

[GitHub] spark pull request: [SPARK-2860][SQL] Fix coercion of CASE WHEN.

2014-08-05 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1785#issuecomment-51232808 A few minor comments otherwise LGTM! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Set Spark SQL Hive compatibility test shuffle ...

2014-08-05 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1784#issuecomment-51234268 Can we reset the original value in afterAll()? There's a test in `SQLConfSuite` that depends on that option, and in the future people might easily add tests

[GitHub] spark pull request: [SQL] Tighten the visibility of various SQLCon...

2014-08-05 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1794#discussion_r15856595 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala --- @@ -17,17 +17,17 @@ package org.apache.spark.sql +import

[GitHub] spark pull request: [SQL] Tighten the visibility of various SQLCon...

2014-08-05 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1794#issuecomment-51295283 Hey @rxin -- I think this is good to go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2406][SQL] Initial support for using Pa...

2014-08-06 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1819#discussion_r15913270 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala --- @@ -78,6 +78,14 @@ class HiveContext(sc: SparkContext) extends

[GitHub] spark pull request: [SPARK-2406][SQL] Initial support for using Pa...

2014-08-06 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1819#discussion_r15914067 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala --- @@ -78,6 +78,14 @@ class HiveContext(sc: SparkContext) extends

[GitHub] spark pull request: [SPARK-2315] Implement drop, dropRight and dro...

2014-08-07 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1839#issuecomment-51520540 Jenkins, this is okay to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2678][Core][SQL] A workaround for SPARK...

2014-08-07 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1801#issuecomment-51533012 @liancheng @andrewor14 @pwendell With this patch things like `./bin/spark-shell --master local[2]` errors out (bad options: --master). I had to workaround

[GitHub] spark pull request: [SPARK-2678][Core][SQL] A workaround for SPARK...

2014-08-07 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1801#issuecomment-51533306 Oh, it's been reported by #1825. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-09 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/956#discussion_r13577888 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -269,6 +271,22 @@ class SQLContext(@transient val sparkContext

[GitHub] spark pull request: [SPARK-1852] prevents queries with sorts submi...

2014-06-09 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/948#issuecomment-45574240 Hey @liancheng - I think this refactoring will solve this particular ticket (i.e. queries w/ sorts will not be eagerly executed anymore). However, I don't see why

[GitHub] spark pull request: [SPARK-1508][SQL] Add SQLConf to SQLContext.

2014-06-10 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/956#discussion_r13608831 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -269,6 +271,22 @@ class SQLContext(@transient val sparkContext

[GitHub] spark pull request: SPARK-2053: add Catalyst expressions for CASE ...

2014-06-11 Thread concretevitamin
GitHub user concretevitamin opened a pull request: https://github.com/apache/spark/pull/1055 SPARK-2053: add Catalyst expressions for CASE WHEN. JIRA ticket: https://issues.apache.org/jira/browse/SPARK-2053 This PR adds support for two types of CASE statements present

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-11 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13683459 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-11 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13683618 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-11 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13683741 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-11 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13683777 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-11 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13683959 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-12 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13723141 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-12 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13723727 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-12 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13724773 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once command

2014-06-12 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1071#discussion_r13728707 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala --- @@ -22,45 +22,69 @@ import org.apache.spark.rdd.RDD import

[GitHub] spark pull request: [SPARK-2094][SQL] Exactly once command

2014-06-12 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1071#discussion_r13728877 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala --- @@ -22,45 +22,69 @@ import org.apache.spark.rdd.RDD import

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-12 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13728987 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +203,139 @@ case class If(predicate

[GitHub] spark pull request: [SQL] Support transforming TreeNodes with Opti...

2014-06-12 Thread concretevitamin
GitHub user concretevitamin opened a pull request: https://github.com/apache/spark/pull/1074 [SQL] Support transforming TreeNodes with Option children. Thanks goes to @marmbrus for his implementation. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-13 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13772326 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala --- @@ -202,3 +201,78 @@ case class If(predicate

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-13 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1055#issuecomment-46068679 The latest build only contains some PySpark failures I think. On Fri, Jun 13, 2014 at 3:43 PM, UCB AMPLab notificati...@github.com wrote

[GitHub] spark pull request: [SQL] Support transforming TreeNodes with Opti...

2014-06-13 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1074#issuecomment-46076441 Ah, this is tricky. Good to know and thanks for the fix. On Friday, June 13, 2014, Michael Armbrust notificati...@github.com wrote: This test

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-16 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13821017 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala --- @@ -917,6 +919,19 @@ private[hive] object HiveQl { case Token

[GitHub] spark pull request: [SPARK-2053][SQL] Add Catalyst expressions for...

2014-06-17 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1055#discussion_r13874731 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala --- @@ -282,4 +290,33 @@ trait HiveTypeCoercion

[GitHub] spark pull request: [SPARK-2176][SQL] Extra unnecessary exchange o...

2014-06-18 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1116#issuecomment-46469383 Sorry for introducing the bug in the first place. Just throwing a thought out there: I have found that there are a lot of arguably hidden assumptions

[GitHub] spark pull request: [SPARK-2177][SQL] describe table result contai...

2014-06-18 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1118#discussion_r13937717 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/hiveOperators.scala --- @@ -445,7 +445,19 @@ case class NativeCommand

[GitHub] spark pull request: [SQL] SPARK-1800 Add broadcast hash join opera...

2014-06-18 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/734#discussion_r13941073 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -142,6 +136,68 @@ case class HashJoin

[GitHub] spark pull request: [SQL] SPARK-1800 Add broadcast hash join opera...

2014-06-18 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/734#discussion_r13941218 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -142,6 +136,68 @@ case class HashJoin

[GitHub] spark pull request: [SPARK-2196] [SQL] Fix nullability of CaseWhen...

2014-06-19 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1133#issuecomment-46604867 Thanks, looks good to me. By the way, is there a simple way to add a test for this? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin

2014-06-19 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1127#discussion_r13989743 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -245,6 +245,74 @@ case class LeftSemiJoinBNL

[GitHub] spark pull request: Spark SQL add LeftSemiBloomFilterBroadcastJoin

2014-06-19 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1127#discussion_r13989753 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -245,6 +245,74 @@ case class LeftSemiJoinBNL

[GitHub] spark pull request: [SPARK-1412][SQL] Disable partial aggregation ...

2014-06-20 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1152#issuecomment-46709082 @rxin If we are simply trying to read the default values for the params, but not user-set ones (i.e. in the absence of a `SQLContext` in `execute()`, I think we

[GitHub] spark pull request: [SQL][SPARK-2212]HashJoin(Shuffled)

2014-06-20 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1147#discussion_r14043578 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -36,158 +37,211 @@ case object BuildLeft extends BuildSide

[GitHub] spark pull request: Spark SQL basicOperators add Except operator

2014-06-20 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-46732664 For this to take effects, the parser (`SQLParser`) needs to be changed to be able to parse EXCEPT return the correct operators. --- If your project is set up

[GitHub] spark pull request: [WIP] [SQL] SPARK-1800 Add broadcast hash join...

2014-06-20 Thread concretevitamin
GitHub user concretevitamin opened a pull request: https://github.com/apache/spark/pull/1163 [WIP] [SQL] SPARK-1800 Add broadcast hash join operator, and simple size-based auto conversion optimization. This PR is based off Michael's [PR 734](https://github.com/apache/spark/pull

[GitHub] spark pull request: [SQL]Extract the joinkeys from join condition

2014-06-23 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1190#discussion_r14114049 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -160,6 +160,43 @@ object HashFilteredJoin extends

[GitHub] spark pull request: [SQL]Extract the joinkeys from join condition

2014-06-23 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1190#discussion_r14114058 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -160,6 +160,43 @@ object HashFilteredJoin extends

[GitHub] spark pull request: [SQL] SPARK-1800 Add broadcast hash join opera...

2014-06-24 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1163#issuecomment-47033189 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SQL] SPARK-1800 Add broadcast hash join opera...

2014-06-24 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1163#issuecomment-47045612 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SARPK-2263][SQL] Support inserting MAPK, V ...

2014-06-24 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1205#discussion_r14167337 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala --- @@ -370,6 +370,16 @@ class HiveQuerySuite extends

[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...

2014-06-25 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47134060 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...

2014-06-25 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47146951 Hi @edrevo -- thanks, the implementation looks really concise! I haven't thought about this, but what do people think about instead of having syntax like `AVG

[GitHub] spark pull request: [SQL] SPARK-1800 Add broadcast hash join opera...

2014-06-25 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1163#discussion_r14215683 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins.scala --- @@ -67,73 +62,72 @@ case class HashJoin( @transient lazy val

[GitHub] spark pull request: [SQL] SPARK-1800 Add broadcast hash join opera...

2014-06-25 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1163#issuecomment-47181980 Sounds good. On Wednesday, June 25, 2014, Michael Armbrust notificati...@github.com wrote: Thanks, I've merged this into master. I did

[GitHub] spark pull request: [SQL] [WIP] Prototype implementation of size e...

2014-06-26 Thread concretevitamin
GitHub user concretevitamin opened a pull request: https://github.com/apache/spark/pull/1238 [SQL] [WIP] Prototype implementation of size estimations for Catalyst logical plans. The idea is that every Catalyst logical plan gets hold of an Estimates class, the usage of which

[GitHub] spark pull request: [HOTFIX] Synchronize on SQLContext.settings in...

2014-07-01 Thread concretevitamin
GitHub user concretevitamin opened a pull request: https://github.com/apache/spark/pull/1277 [HOTFIX] Synchronize on SQLContext.settings in tests. Let's see if this fixes the ongoing series of test failures in a master build machine (https://amplab.cs.berkeley.edu/jenkins/job

[GitHub] spark pull request: Update SQLConf.scala

2014-07-01 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47717333 Yeah, what is motivating this change? When this class got introduced, @rxin commented that java.util.ConcurrentHashMap had bad memory footprint and suggested

[GitHub] spark pull request: Update SQLConf.scala

2014-07-01 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47717946 Sorry, I didn't realize Reynold had already commented on this thread. The current changes with Option look good. --- If your project is set up for it, you can

[GitHub] spark pull request: [HOTFIX] Synchronize on SQLContext.settings in...

2014-07-01 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1277#issuecomment-47733815 Probably not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2186: Spark SQL DSL support for simple a...

2014-07-02 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1211#issuecomment-47805034 Hi Ximo, Sorry for the delay. Many of the committers are busy running the Spark Summit these few days so a lot of PRs are backlog'd

[GitHub] spark pull request: Update SQLConf.scala

2014-07-02 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1272#issuecomment-47866266 Jenkins, ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SQL] [WIP] Prototype implementation of size e...

2014-07-03 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r14538726 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -26,6 +26,18 @@ import

[GitHub] spark pull request: [SQL] [WIP] Prototype implementation of size e...

2014-07-03 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1238#issuecomment-47989913 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SQL] Prototype implementation of size estimat...

2014-07-03 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1238#issuecomment-47996391 This is no longer WIP -- tests added previous review comments addressed. +@marmbrus let me know what you think is missing that should also go

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-07-06 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1297#discussion_r14578164 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ImmutableVector.scala --- @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-2365] Add IndexedRDD, an efficient upda...

2014-07-06 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1297#discussion_r14578170 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ImmutableVector.scala --- @@ -0,0 +1,219 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-2409] Make SQLConf thread safe.

2014-07-08 Thread concretevitamin
Github user concretevitamin commented on the pull request: https://github.com/apache/spark/pull/1334#issuecomment-48387302 LGTM On Tuesday, July 8, 2014, UCB AMPLab notificati...@github.com wrote: Merged build started. — Reply to this email directly

[GitHub] spark pull request: [SPARK-2409] Make SQLConf thread safe.

2014-07-08 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1334#discussion_r14674761 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala --- @@ -26,6 +26,8 @@ import scala.collection.JavaConverters._ * queried

[GitHub] spark pull request: [WIP][SPARK-2179][SQL] Public API for DataType...

2014-07-09 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1346#discussion_r14735318 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -89,6 +89,16 @@ class SQLContext(@transient val sparkContext

[GitHub] spark pull request: [SPARK-2393][SQL] Prototype implementation of ...

2014-07-09 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r14737891 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -61,8 +61,6 @@ private[sql] abstract class

[GitHub] spark pull request: [SPARK-2393][SQL] Prototype implementation of ...

2014-07-09 Thread concretevitamin
Github user concretevitamin commented on a diff in the pull request: https://github.com/apache/spark/pull/1238#discussion_r14738387 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala --- @@ -26,6 +26,17 @@ import

  1   2   3   >