[GitHub] spark issue #14289: [SPARK-16656] [SQL] Try to make CreateTableAsSelectSuite...

2016-07-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14289 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #14289: [SPARK-16656] [SQL] Try to make CreateTableAsSele...

2016-07-20 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14289 [SPARK-16656] [SQL] Try to make CreateTableAsSelectSuite more stable ## What changes were proposed in this pull request? https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62593

[GitHub] spark pull request #14281: [SPARK-16644][SQL] Aggregate should not propagate...

2016-07-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14281#discussion_r71614119 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/ConstraintPropagationSuite.scala --- @@ -79,13 +79,15 @@ class

[GitHub] spark pull request #14284: [SPARK-16633] [SPARK-16642] Fixes three issues re...

2016-07-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14284#discussion_r71598723 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLWindowFunctionSuite.scala --- @@ -367,4 +367,50 @@ class SQLWindowFunctionSuite

[GitHub] spark pull request #14284: [SPARK-16633] [SPARK-16642] Fixes three issues re...

2016-07-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14284#discussion_r71598678 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLWindowFunctionSuite.scala --- @@ -357,14 +356,59 @@ class SQLWindowFunctionSuite extends

[GitHub] spark pull request #14284: [SPARK-16633] [SPARK-16642] Fixes three issues re...

2016-07-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14284#discussion_r71588935 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLWindowFunctionSuite.scala --- @@ -367,4 +367,50 @@ class SQLWindowFunctionSuite

[GitHub] spark issue #14284: [SPARK-16633] [SPARK-16642] Fixes three issues related t...

2016-07-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14284 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #14284: [SPARK-16633] [SPARK-16642] Fixes three issues related t...

2016-07-20 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14284 Without a good reason and providing a way  to make lead and lag respect Bulls, we should not change the behavior. On Wed, Jul 20, 2016 at 2:04 AM -0700, "A

[GitHub] spark pull request #14284: [SPARK-16633] [SPARK-16642] Fixes three issues re...

2016-07-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14284#discussion_r71489063 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/WindowExec.scala --- @@ -582,25 +582,43 @@ private[execution] final class

[GitHub] spark pull request #14284: [SPARK-16633] [SPARK-16642] Fixes three issues re...

2016-07-20 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14284#discussion_r71488537 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala --- @@ -382,7 +382,7 @@ abstract class

[GitHub] spark pull request #14284: [SPARK-16633] [SPARK-16642] Fixes three issues re...

2016-07-20 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14284 [SPARK-16633] [SPARK-16642] Fixes three issues related to window functions ## What changes were proposed in this pull request? This PR contains three changes. First, this PR changes the

[GitHub] spark issue #14272: [SPARK-16632][sql] Respect Hive schema when merging parq...

2016-07-19 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14272 yea. I think the fix is pretty safe. After discussion with @liancheng, seems the more general fix is to just to use the requested catalyst schema to initialize the vectorized reader. --- If your

[GitHub] spark pull request #14267: [SPARK-15705] [SQL] Change the default value of s...

2016-07-19 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14267 [SPARK-15705] [SQL] Change the default value of spark.sql.hive.convertMetastoreOrc to false. ## What changes were proposed in this pull request? In 2.0, we add a new logic to convert

[GitHub] spark issue #14014: [SPARK-16344][SQL] Decoding Parquet array of struct with...

2016-07-18 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14014 Let's also update the description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r71277147 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -442,13 +445,23 @@ private[parquet

spark git commit: [SPARK-16303][DOCS][EXAMPLES] Minor Scala/Java example update

2016-07-18 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 24ea87519 -> ef2a6f131 [SPARK-16303][DOCS][EXAMPLES] Minor Scala/Java example update ## What changes were proposed in this pull request? This PR moves one and the last hard-coded Scala example snippet from the SQL programming guide in

spark git commit: [SPARK-16303][DOCS][EXAMPLES] Minor Scala/Java example update

2016-07-18 Thread yhuai
Repository: spark Updated Branches: refs/heads/master e5fbb182c -> 1426a0805 [SPARK-16303][DOCS][EXAMPLES] Minor Scala/Java example update ## What changes were proposed in this pull request? This PR moves one and the last hard-coded Scala example snippet from the SQL programming guide into `

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r71276489 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRecordMaterializer.scala --- @@ -30,10 +30,11 @@ import

[GitHub] spark issue #14245: [SPARK-16303][DOCS][EXAMPLES] Minor Scala/Java example u...

2016-07-18 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14245 Thanks. Merging to master and branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #14155: [SPARK-16498][SQL][WIP] move hive hack for data s...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r71273081 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -146,6 +151,15 @@ case class CatalogTable

[GitHub] spark pull request #14155: [SPARK-16498][SQL][WIP] move hive hack for data s...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r71272934 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala --- @@ -303,6 +303,7 @@ object

[GitHub] spark pull request #14155: [SPARK-16498][SQL][WIP] move hive hack for data s...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r71272434 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -313,18 +313,48 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] spark pull request #14155: [SPARK-16498][SQL][WIP] move hive hack for data s...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14155#discussion_r71272290 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -146,6 +151,15 @@ case class CatalogTable

[GitHub] spark issue #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid unnecessa...

2016-07-18 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14036 @techaddict Can you test the performance with and without your change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

spark git commit: [SPARK-16515][SQL] set default record reader and writer for script transformation

2016-07-18 Thread yhuai
ark, but would fail now. ## How was this patch tested? added a test case in SQLQuerySuite. Closes #14169 Author: Daoyuan Wang Author: Yin Huai Closes #14249 from yhuai/scriptTransformation. (cherry picked from commit 96e9afaae93318250334211cc80ed0fee3d055b9) Signed-off-by: Yin Huai Proj

spark git commit: [SPARK-16515][SQL] set default record reader and writer for script transformation

2016-07-18 Thread yhuai
ark, but would fail now. ## How was this patch tested? added a test case in SQLQuerySuite. Closes #14169 Author: Daoyuan Wang Author: Yin Huai Closes #14249 from yhuai/scriptTransformation. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/

[GitHub] spark issue #14249: [SPARK-16515][SQL]set default record reader and writer f...

2016-07-18 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14249 I am merging this PR to master and branch 2.0. Thanks @adrian-wang --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #14249: [SPARK-16515][SQL]set default record reader and w...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14249#discussion_r71227856 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -1329,7 +1332,7 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

2016-07-18 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14028 Merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #14249: [SPARK-16515][SQL]set default record reader and w...

2016-07-18 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14249 [SPARK-16515][SQL]set default record reader and writer for script transformation ## What changes were proposed in this pull request? In ScriptInputOutputSchema, we read default RecordReader and

[GitHub] spark pull request #14169: [SPARK-16515][SQL]set default record reader and w...

2016-07-18 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14169#discussion_r71192358 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -1306,7 +1306,7 @@ class SparkSqlAstBuilder(conf: SQLConf

spark git commit: [SPARK-16351][SQL] Avoid per-record type dispatch in JSON when writing

2016-07-18 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 8ea3f4eae -> 2877f1a52 [SPARK-16351][SQL] Avoid per-record type dispatch in JSON when writing ## What changes were proposed in this pull request? Currently, `JacksonGenerator.apply` is doing type-based dispatch for each row to write appro

[GitHub] spark issue #14245: [MINOR][DOCS][EXAMPLES] Minor Scala example update

2016-07-17 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14245 LGTM. Can we reuse a existing jira number? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #14169: [SPARK-16515][SQL]set default record reader and w...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14169#discussion_r71102534 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -1340,10 +1340,17 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71097210 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096802 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096761 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096584 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096571 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096401 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096388 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71096347 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonParser.scala --- @@ -35,184 +34,306 @@ import

[GitHub] spark pull request #14102: [SPARK-16434][SQL] Avoid per-record type dispatch...

2016-07-17 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14102#discussion_r71095725 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONOptions.scala --- @@ -51,7 +53,8 @@ private[sql] class JSONOptions

[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

2016-07-17 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14028 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #14028: [SPARK-16351][SQL] Avoid per-record type dispatch in JSO...

2016-07-17 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14028 LGTM pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #14169: [SPARK-16515][SQL]set default record reader and w...

2016-07-15 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14169#discussion_r71058385 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala --- @@ -1329,7 +1329,7 @@ class SparkSqlAstBuilder(conf: SQLConf

[GitHub] spark pull request #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not s...

2016-07-14 Thread yhuai
Github user yhuai closed the pull request at: https://github.com/apache/spark/pull/14139 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

spark git commit: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently drop exceptions in file listing

2016-07-14 Thread yhuai
his patch tested? Manually tested. **Note: This is a backport of https://github.com/apache/spark/pull/13987** Author: Yin Huai Closes #14139 from yhuai/SPARK-16313-branch-1.6. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/com

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-14 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 Thank you! I am merging this PR to branch 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not s...

2016-07-14 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14139#discussion_r70843685 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -273,6 +273,20 @@ private[hive] class HiveMetastoreCatalog(val

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 @rxin I think this version is the minimal change. Since the partition discovery logic in inside HadoopFsRelation in 1.6 and the refresh is triggered by using lazy val, passing a flag down will

[GitHub] spark pull request #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not s...

2016-07-13 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14139#discussion_r70727924 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -273,6 +273,22 @@ private[hive] class HiveMetastoreCatalog(val

spark git commit: [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema

2016-07-13 Thread yhuai
Repository: spark Updated Branches: refs/heads/branch-2.0 9e3a59858 -> 550d0e7dc [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema What changes were proposed in this pull request? If we create a table pointing to a parquet/json datasets without specif

spark git commit: [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema

2016-07-13 Thread yhuai
Repository: spark Updated Branches: refs/heads/master fb2e8eeb0 -> c5ec87982 [SPARK-16482][SQL] Describe Table Command for Tables Requiring Runtime Inferred Schema What changes were proposed in this pull request? If we create a table pointing to a parquet/json datasets without specifying

[GitHub] spark issue #14148: [SPARK-16482] [SQL] Describe Table Command for Tables Re...

2016-07-13 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14148 LGTM. Merging to master and branch 2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #14148: [SPARK-16482] [SQL] Describe Table Command for Ta...

2016-07-12 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14148#discussion_r70571914 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -413,38 +413,36 @@ case class DescribeTableCommand(table

[GitHub] spark pull request #14148: [SPARK-16482] [SQL] Describe Table Command for Ta...

2016-07-12 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14148#discussion_r70570551 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala --- @@ -105,7 +105,7 @@ case class

[GitHub] spark pull request #14148: [SPARK-16482] [SQL] Describe Table Command for Ta...

2016-07-12 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14148#discussion_r70570489 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -431,7 +431,7 @@ case class DescribeTableCommand(table

[GitHub] spark issue #13701: [SPARK-15639][SQL] Try to push down filter at RowGroups ...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/13701 @viirya Thank you for updating this. Our schedules are pretty packed for the release. We can take a look at it once 2.0 is released. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 let me take another look to see if there is a better change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 tes this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-12 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

spark git commit: [SPARK-12639][SQL] Mark Filters Fully Handled By Sources with *

2016-07-11 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 9cc74f95e -> b1e5281c5 [SPARK-12639][SQL] Mark Filters Fully Handled By Sources with * ## What changes were proposed in this pull request? In order to make it clear which filters are fully handled by the underlying datasource we will mark

[GitHub] spark issue #11317: [SPARK-12639] [SQL] Mark Filters Fully Handled By Source...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/11317 lgtm. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #11317: [SPARK-12639] [SQL] Mark Filters Fully Handled By Source...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/11317 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #11317: [SPARK-12639] [SQL] Mark Filters Fully Handled By Source...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/11317 tes thsi please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

spark git commit: [SPARK-16349][SQL] Fall back to isolated class loader when classes not found.

2016-07-11 Thread yhuai
Repository: spark Updated Branches: refs/heads/master 7f38b9d5f -> b4fbe140b [SPARK-16349][SQL] Fall back to isolated class loader when classes not found. Some Hadoop classes needed by the Hive metastore client jars are not present in Spark's packaging (for example, "org/apache/hadoop/mapred/M

[GitHub] spark issue #14020: [SPARK-16349][sql] Fall back to isolated class loader wh...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14020 lgtm. Merging to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #14020: [SPARK-16349][sql] Fall back to isolated class lo...

2016-07-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14020#discussion_r70337582 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala --- @@ -220,9 +220,15 @@ private[hive] class

[GitHub] spark issue #14020: [SPARK-16349][sql] Fall back to isolated class loader wh...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14020 also cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #14020: [SPARK-16349][sql] Fall back to isolated class lo...

2016-07-11 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14020#discussion_r70335850 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala --- @@ -220,9 +220,15 @@ private[hive] class

[GitHub] spark issue #14020: [SPARK-16349][sql] Fall back to isolated class loader wh...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14020 Will putting that jar in Spark's classpath work? Seems so? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark issue #13973: [SPARK-16182] [CORE] Utils.scala -- terminateProcess() s...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/13973 @srowen Seems this commit breaks 1.6 builds (https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-1.6-test-sbt-hadoop-1.0/248/)? --- If your project is set up

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 Let me see if we can have a flag to determine if we want to swallow the FNF (like what https://github.com/apache/spark/pull/13987/files does). --- If your project is set up for it, you can reply to

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently...

2016-07-11 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14139 I think there will be one warning when we create a table. Or maybe there is no warning during table creation because the refresh is called lazily. --- If your project is set up for it, you can

[GitHub] spark pull request #14139: [SPARK-16313][SQL][BRANCH-1.6] Spark should not s...

2016-07-11 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14139 [SPARK-16313][SQL][BRANCH-1.6] Spark should not silently drop exceptions in file listing ## What changes were proposed in this pull request? Spark silently drops exceptions during file listing

[GitHub] spark issue #13991: [SPARK-16318][SQL] Implement all remaining xpath functio...

2016-07-10 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/13991 OK. Thanks. Then, it will be good to add more tests for cases that are not covered by those hive tests. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #11317: [SPARK-12639] [SQL] Mark Filters Fully Handled By Source...

2016-07-10 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/11317 tes this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #13991: [SPARK-16318][SQL] Implement all remaining xpath functio...

2016-07-10 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/13991 As a follow-up task. Can you take a look at the following query files and add useful tests in your test? Thanks. ``` .//sql/hive/src/test/resources/ql/src/test/queries/clientpositive

spark git commit: [SPARK-16453][BUILD] release-build.sh is missing hive-thriftserver for scala 2.10

2016-07-08 Thread yhuai
by release-build.sh. Author: Yin Huai Closes #14108 from yhuai/SPARK-16453. (cherry picked from commit 60ba436b7010436c77dfe5219a9662accc25bffa) Signed-off-by: Yin Huai Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/07f562f5 T

spark git commit: [SPARK-16453][BUILD] release-build.sh is missing hive-thriftserver for scala 2.10

2016-07-08 Thread yhuai
ase-build.sh. Author: Yin Huai Closes #14108 from yhuai/SPARK-16453. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/60ba436b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/60ba436b Diff: http://git-wip-us.apache.

[GitHub] spark issue #14108: [SPARK-16453] [BUILD] release-build.sh is missing hive-t...

2016-07-08 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14108 Thanks. Merging to master and branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #14108: [SPARK-16453] [BUILD] release-build.sh is missing hive-t...

2016-07-08 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14108 @srowen Does it look good? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #14108: [SPARK-16453] [BUILD] release-build.sh is missing...

2016-07-08 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14108#discussion_r70144675 --- Diff: dev/create-release/release-build.sh --- @@ -258,7 +258,7 @@ if [[ "$1" == "publish-snapshot" ]]; then -Phive

[GitHub] spark pull request #14108: [SPARK-16453] [BUILD] release-build.sh is missing...

2016-07-08 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14108#discussion_r70144693 --- Diff: dev/create-release/release-build.sh --- @@ -258,7 +258,7 @@ if [[ "$1" == "publish-snapshot" ]]; then -Phive

[GitHub] spark pull request #14108: [SPARK-16453] [BUILD] release-build.sh is missing...

2016-07-08 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14108#discussion_r70144390 --- Diff: dev/create-release/release-build.sh --- @@ -258,7 +258,7 @@ if [[ "$1" == "publish-snapshot" ]]; then -Phive

[GitHub] spark issue #14108: [SPARK-16453] [BUILD] release-build.sh is missing hive-t...

2016-07-08 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/14108 cc @JoshRosen @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #14108: [SPARK-16453] [BUILD] release-build.sh is missing...

2016-07-08 Thread yhuai
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/14108 [SPARK-16453] [BUILD] release-build.sh is missing hive-thriftserver for scala 2.10 ## What changes were proposed in this pull request? This PR adds hive-thriftserver profile to scala 2.10 build

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70030627 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70030569 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70030381 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70030343 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70029947 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70029907 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14014#discussion_r70029843 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -482,13 +482,105 @@ private[parquet

[GitHub] spark pull request #14028: [SPARK-16351][SQL] Avoid per-record type dispatch...

2016-07-07 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/14028#discussion_r69936170 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JacksonGenerator.scala --- @@ -17,74 +17,180 @@ package

<    5   6   7   8   9   10   11   12   13   14   >