[GitHub] spark pull request #17699: [SPARK-20405][SQL] Dataset.withNewExecutionId sho...

2017-04-20 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17699 [SPARK-20405][SQL] Dataset.withNewExecutionId should be private ## What changes were proposed in this pull request? Dataset.withNewExecutionId is only used in Dataset itself and should be private

[GitHub] spark pull request #17698: [SPARK-20403][SQL][Documentation]Modify the instr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17698#discussion_r112383091 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -1036,3 +1036,8 @@ case class UpCast(child: Expression

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112382152 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ArrowConvertersSuite.scala --- @@ -0,0 +1,568 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112381608 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ArrowConvertersSuite.scala --- @@ -0,0 +1,568 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112376143 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112376037 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112375921 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112375496 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112373858 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-20 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112373805 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112370906 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112370321 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112368956 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112368367 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/ArrowConverters.scala --- @@ -0,0 +1,432 @@ +/* +* Licensed to the Apache Software Foundation

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112365872 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1635,21 +1636,49 @@ def toDF(self, *cols): return DataFrame(jdf, self.sql_ctx

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112365773 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1635,21 +1636,49 @@ def toDF(self, *cols): return DataFrame(jdf, self.sql_ctx

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-04-19 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r112365501 --- Diff: python/pyspark/serializers.py --- @@ -182,6 +182,23 @@ def loads(self, obj): raise NotImplementedError +class

[GitHub] spark issue #15821: [SPARK-13534][PySpark] Using Apache Arrow to increase pe...

2017-04-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15821 @BryanCutler Are you going to update this for arrow 0.3? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15821: [SPARK-13534][PySpark] Using Apache Arrow to increase pe...

2017-04-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15821 Please move ArrowConverters.scala somewhere else that's not top level, e.g. execution.arrow --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #17678: [SPARK-20381][SQL] Add SQL metrics of numOutputRows for ...

2017-04-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17678 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17678: [SPARK-20381][SQL] Add SQL metrics of numOutputRows for ...

2017-04-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17678 Is there a codegen version we need to worry about? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17690: Fixed typos in docs

2017-04-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17690 Thanks - merging in master/branch-2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17692: [SPARK-20398] [SQL] range() operator should include canc...

2017-04-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17692 Merging in master/branch-2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17648: [SPARK-19851] Add support for EVERY and ANY (SOME) aggre...

2017-04-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17648 Can we just do a logical rewrite to turn them into "condA + condB + condC > 0" (for Some/Any) ? --- If your project is set up for it, you can reply to this email and have your reply app

[GitHub] spark issue #17657: [TEST][MINOR] Replace repartitionBy with distribute in C...

2017-04-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17657 Merging in master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17661: [SPARK-20349] [SQL] [REVERT-Branch2.1] ListFunctions ret...

2017-04-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17661 Merging in branch-2.1. Can you close your PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17664: Typo fix: distitrbuted -> distributed

2017-04-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17664 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15398: [SPARK-17647][SQL] Fix backslash escaping in 'LIKE' patt...

2017-04-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15398 I pushed a commit. Hopefully that fixes it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15398: [SPARK-17647][SQL] Fix backslash escaping in 'LIKE' patt...

2017-04-17 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15398 I've resolved the conflict and merged this in master/branch-2.1. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17630: [SPARK-20318][SQL] Use Catalyst type for min/max in Colu...

2017-04-15 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17630 Thanks for the explanation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17630: [SPARK-20318][SQL] Use Catalyst type for min/max in Colu...

2017-04-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17630 Wait - are we storing UTF8Strings directly in the catalog for statistics? That doesn't make sense ... if we are not, then we are not using internal types. In that case we should document clearly

[GitHub] spark issue #17633: [SPARK-20331][SQL] Enhanced Hive partition pruning predi...

2017-04-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17633 Then it should work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17633: [SPARK-20331][SQL] Enhanced Hive partition pruning predi...

2017-04-14 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17633 Does this work for non-Hive tables? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17623: [SPARK-20292][SQL][WIP] Clean up string represent...

2017-04-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17623#discussion_r111505420 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala --- @@ -149,7 +149,7 @@ case class Cast(child: Expression

[GitHub] spark pull request #17196: [SPARK-19855][SQL] Create an internal FilePartiti...

2017-04-13 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/17196 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17630: [SPARK-20318][SQL] Use Catalyst type for min/max in Colu...

2017-04-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17630 When we update Spark and change the internal format, we'd still need to keep the current implementation. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #17630: [SPARK-20318][SQL] Use Catalyst type for min/max in Colu...

2017-04-13 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17630 hm this means we will forever need to be able to read the internal format, doesn't it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #17614: [SPARK-20302][SQL] Short circuit cast when from and to t...

2017-04-12 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17614 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17616: [SPARK-20304][SQL] AssertNotNull should not inclu...

2017-04-12 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17616 [SPARK-20304][SQL] AssertNotNull should not include path in string representation ## What changes were proposed in this pull request? AssertNotNull's toString/simpleString dumps the entire

[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...

2017-04-12 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17616 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17614: [SPARK-20302][SQL] Short circuit cast when from a...

2017-04-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17614#discussion_r111064001 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala --- @@ -288,4 +288,30 @@ object DataType { case (fromDataType

[GitHub] spark issue #17604: [SPARK-20289][SQL] Use StaticInvoke to box primitive typ...

2017-04-11 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17604 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17604: [SPARK-20289][SQL] Use StaticInvoke to box primit...

2017-04-11 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17604 [SPARK-20289][SQL] Use StaticInvoke to box primitive types ## What changes were proposed in this pull request? Dataset typed API currently uses NewInstance to box primitive types (i.e. calling

[GitHub] spark issue #17599: [SPARK-17564][Tests]Fix flaky RequestTimeoutIntegrationS...

2017-04-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17599 Merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17599: [SPARK-17564][Tests]Fix flaky RequestTimeoutIntegrationS...

2017-04-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17599 LGTM pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17596: [SPARK-12837][SQL] reduce the serialized size of accumul...

2017-04-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17596 BTW a potential, better way to solve this is to combine all the metrics into a single accumulator. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #17596: [SPARK-12837][SQL] reduce the serialized size of ...

2017-04-10 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17596#discussion_r110765367 --- Diff: core/src/main/scala/org/apache/spark/util/InternalLongAccumulator.scala --- @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #17595: [SPARK-20283][SQL] Add preOptimizationBatches

2017-04-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17595 Merging this since as long as it compiles the change should be fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request #17595: [SPARK-20283][SQL] Add preOptimizationBatches

2017-04-10 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17595 [SPARK-20283][SQL] Add preOptimizationBatches ## What changes were proposed in this pull request? We currently have postHocOptimizationBatches, but not preOptimizationBatches. This patch adds

[GitHub] spark issue #17592: [SPARK-20243][TESTS] DebugFilesystem.assertNoOpenStreams...

2017-04-10 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17592 Should this go into branch-2.1 as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17574: [SPARK-20264][SQL] asm should be non-test dependency in ...

2017-04-09 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17574 Meh let's not bother. There isn't any harm in the current setup since it's already a transitive dependency. Why waste time on those? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request #17574: [SPARK-20264][SQL] asm should be non-test depende...

2017-04-07 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17574 [SPARK-20264][SQL] asm should be non-test dependency in sql/core ## What changes were proposed in this pull request? sq/core module currently declares asm as a test scope dependency. Transitively

[GitHub] spark pull request #17573: [SPARK-20262][SQL] AssertNotNull should throw Nul...

2017-04-07 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17573 [SPARK-20262][SQL] AssertNotNull should throw NullPointerException ## What changes were proposed in this pull request? AssertNotNull currently throws RuntimeException. It should throw

[GitHub] spark issue #17570: [SPARK-20255] Move listLeafFiles() to InMemoryFileIndex

2017-04-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17570 Merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17570: [SPARK-20255] Move listLeafFiles() to InMemoryFileIndex

2017-04-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17570 LGTM pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17570: [SPARK-20255] Move listLeafFiles() to InMemoryFileIndex

2017-04-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17570 Jenkins, add to whitelist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17555: [SPARK-19495][SQL] Make SQLConf slightly more ext...

2017-04-06 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17555 [SPARK-19495][SQL] Make SQLConf slightly more extensible - addendum ## What changes were proposed in this pull request? This is a tiny addendum to SPARK-19495 to remove the private visibility

[GitHub] spark issue #17554: [MINOR][DOCS] Fix typo in Hive Examples

2017-04-06 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17554 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17541: [SPARK-20229][SQL] add semanticHash to QueryPlan

2017-04-05 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17541#discussion_r110013198 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/broadcastMode.scala --- @@ -26,10 +26,7 @@ import

[GitHub] spark issue #17471: [SPARK-3577] Report Spill size on disk for UnsafeExterna...

2017-04-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17471 cc @cloud-fan / @ueshin / @sameeragarwal can you review this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17521: [SPARK-20204][SQL] remove SimpleCatalystConf and Catalys...

2017-04-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17521 @nsyca can you look into it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17521: [SPARK-20204][SQL] remove SimpleCatalystConf and Catalys...

2017-04-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17521 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17522: [SPARK-18278] [Scheduler] Documentation to point to Kube...

2017-04-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17522 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17521: [SPARK-20204][SQL] remove SimpleCatalystConf and Catalys...

2017-04-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17521 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17499: [SPARK-20161][CORE] Default log4j properties file should...

2017-04-04 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17499 Great - please close this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17487: [Spark-20145] Fix range case insensitive bug in SQL

2017-04-03 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17487 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #17505: [SPARK-20187][SQL] Replace loadTable with moveFil...

2017-04-03 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17505#discussion_r109553390 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala --- @@ -242,6 +251,16 @@ private[client] class Shim_v0_12 extends Shim

[GitHub] spark issue #17499: [SPARK-20161][CORE] Default log4j properties file should...

2017-04-03 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17499 Maybe Hive can do it in Hive. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17521: [SPARK-20204][SQL] separate SQLConf into catalyst confs ...

2017-04-03 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17521 To be clear, I don't think we should have two separate places to define config entries. If this is what the pr is doing, I strongly veto. --- If your project is set up for it, you can reply

[GitHub] spark issue #17522: [SPARK-18278] [Scheduler] Documentation to point to Kube...

2017-04-03 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17522 Seems fine to me, since the number of external resource managers are small. We should definitely make it clear there is no firm commitment currently to merge this into Spark though. --- If your

[GitHub] spark issue #17518: [SPARK-20198] [SQL] Remove the inconsistency in table/fu...

2017-04-03 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17518 Is this an API change or just a documentation change? The title suggests you are changing public facing APIs? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #17476: [SPARK-20151][SQL] Account for partition pruning in scan...

2017-03-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17476 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17490: [SPARK-20167]In SqlBase.g4,some of the comments is not c...

2017-03-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17490 I don't think the change makes sense ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17476: [SPARK-20151][SQL] Account for partition pruning ...

2017-03-30 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17476#discussion_r109092194 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileIndex.scala --- @@ -72,4 +72,14 @@ trait FileIndex { /** Schema

[GitHub] spark pull request #17476: [SPARK-20151][SQL] Account for partition pruning ...

2017-03-30 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17476#discussion_r109092246 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/CatalogFileIndex.scala --- @@ -111,7 +113,8 @@ private class

[GitHub] spark issue #17476: [SPARK-20151][SQL] Account for partition pruning in scan...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17476 cc @ericl, @bogdanrdc, @adrian-ionescu, @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #17476: [SPARK-20151][SQL] Account for partition pruning ...

2017-03-29 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17476 [SPARK-20151][SQL] Account for partition pruning in scan metadataTime metrics ## What changes were proposed in this pull request? After SPARK-20136, we report metadata timing metrics in scan

[GitHub] spark issue #17475: [SPARK-20148] [SQL] Extend the file commit API to allow ...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17475 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17465: [SPARK-20136][SQL] Add num files and metadata operation ...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17465 Let me merge this now. I will send a follow-up PR to take the logical planning time into account (otherwise in the vast majority of cases, i.e. pruned partitions, the metadata operation time

[GitHub] spark issue #17465: [SPARK-20136][SQL] Add num files and metadata operation ...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17465 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17470: [SPARK-20146][SQL] fix comment missing issue for thrift ...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17470 Merging in master. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17475: [SPARK-20148] [SQL] Extend the file commit API to allow ...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17475 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17465: [SPARK-20136][SQL] Add num files and metadata operation ...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17465 cc @ericl, @bogdanrdc, @adrian-ionescu, @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #17465: [SPARK-20136][SQL] Add num files and metadata ope...

2017-03-29 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17465 [SPARK-20136][SQL] Add num files and metadata operation timing to scan operator metrics ## What changes were proposed in this pull request? This patch adds explicit metadata operation timing

[GitHub] spark issue #17464: [SPARK-20134][SQL] SQLMetrics.postDriverMetricUpdates to...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17464 Merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17464: [SPARK-20134][SQL] SQLMetrics.postDriverMetricUpd...

2017-03-29 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/17464#discussion_r108600240 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala --- @@ -477,9 +477,11 @@ private case class MyPlan(sc

[GitHub] spark pull request #17464: [SPARK-20134][SQL] SQLMetrics.postDriverMetricUpd...

2017-03-28 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17464 [SPARK-20134][SQL] SQLMetrics.postDriverMetricUpdates to simplify driver side metric updates ## What changes were proposed in this pull request? It is not super intuitive how to update SQLMetric

[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...

2017-03-25 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17424 Hm - so this would require us to update the test suite every time there is an update to the docs? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #17420: [SPARK-20070][SQL] Fix 2.10 build

2017-03-24 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17420 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17399: [DOCS] Clarify round mode for format_number & round func...

2017-03-24 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17399 Thanks - merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17186: [SPARK-19846][SQL] Add a flag to disable constraint prop...

2017-03-24 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17186 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17399: Update functions.scala

2017-03-24 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17399 @roxannemoslehi can you fix the title? We can then merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17399: Update functions.scala

2017-03-23 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17399 Yea we definitely need a better title. Thanks for contributing though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17397: [SPARK-20070][SQL] Redact DataSourceScanExec treeString

2017-03-23 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17397 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #17396: Typo fixup in comment

2017-03-23 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17396 Merging in master. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #17312: [SPARK-19973] Display num of executors for the stage.

2017-03-23 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17312 That would be pretty confusing wouldn't it? The table has 3 entries and the title says only 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #17312: [SPARK-19973] Display num of executors for the stage.

2017-03-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17312 Your screenshot had 3 executors. Why does it say 2? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17359: [SPARK-20028][SQL] Add aggreagate expression nGrams

2017-03-22 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17359 Why do we want this? Seems extremely low usage on this function in the wild. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17380: clarify array_contains function description

2017-03-21 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17380 Thanks - merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method

2017-03-19 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17343 Can you add some documentation inline so in the future we'd know why specific implementations were chosen? --- If your project is set up for it, you can reply to this email and have your reply

<    3   4   5   6   7   8   9   10   11   12   >