[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170323025 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170323026 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49054/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170323016 **[Test build #49054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49054/consoleFull)** for PR 10678 at commit [`da6baf2`](https://github.com/apache/spark/commit/da6baf25488767ce6e73538b03f9195bba92b84e). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10680#issuecomment-170321675 **[Test build #49055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49055/consoleFull)** for PR 10680 at commit [`f8b4fdb`](https://github.com/apache/spark/commit/f8b4fdb91870e0702bd2735f9a5a58dea4a63fff). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10679#issuecomment-170321648 [Test build #49053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49053/console) for PR 10679 at commit [`1d60cb6`](https://github.com/apache/spark/commit/1d60cb619bf76b61ae13294db0dc4ad07bccbb96). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10679#issuecomment-170321658 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49053/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10679#issuecomment-170321656 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10679#issuecomment-170321624 [Test build #49053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49053/consoleFull) for PR 10679 at commit [`1d60cb6`](https://github.com/apache/spark/commit/1d60cb619bf76b61ae13294db0dc4ad07bccbb96). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170321338 **[Test build #49054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49054/consoleFull)** for PR 10678 at commit [`da6baf2`](https://github.com/apache/spark/commit/da6baf25488767ce6e73538b03f9195bba92b84e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170320786 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170320650 Actually, I might choose to handle those separately in a followup next week in order to have more time to think through a few additional cases, including the exclusion of certain Jetty versions. I want to get the enforcer infra and the Netty fix in first to unblock a 1.5 doc building problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170320386 I've opened two pull requests to backport the `dev/test-dependencies` infra to branch-1.5 and 1.6, which should allow me to backport this change as well. I agree about the servlet API JARs, that's a good case. We should also exclude the ASM classes, since pulling in old versions of ASM is a leading cause of Java 8 compatibility issues. I'll update this in a little bit in order to cover those two obvious cases. There might be a couple of other candidates that are worth banning, but I'd like to cover them in followups. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10359][PROJECT-INFRA] Backport dev/test...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/10680 [SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.5 This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.6. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark test-deps-16-backport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10680.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10680 commit f97d4d56fd7c8c0d0dd417921ed47b7dd1814531 Author: Josh Rosen Date: 2015-12-30T20:47:42Z [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / #5093, which are currently disabled due to bugs. This patch is based on pwendell's work in #8531. Closes #8531. Author: Josh Rosen Author: Patrick Wendell Closes #10461 from JoshRosen/SPARK-10359. commit e1ab0440e66431ea4046b408b3b0a3087e0bd78c Author: Josh Rosen Date: 2016-01-01T04:23:19Z [SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in #10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen Closes #10543 from JoshRosen/dep-script-fixes. commit f8b4fdb91870e0702bd2735f9a5a58dea4a63fff Author: Josh Rosen Date: 2016-01-04T09:04:29Z [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching This patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script. pwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead. I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail. Author: Josh Rosen Closes #10558 from JoshRosen/build-dep-tests-round-3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA] Backport dev/test...
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/10679 [SPARK-12612][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.5 This patch backports the `dev/test-dependencies` script (from #10461) to branch-1.5. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JoshRosen/spark test-deps-15-backport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10679.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10679 commit 34e0b8aeafbcee1adcdf868b02a01d1df7d627f8 Author: Josh Rosen Date: 2015-12-25T23:07:05Z [SPARK-12477][HOTIFX] Fix test compilation in branch-1.5 commit f06c6c1f47849852e317e463de79cc82f561877e Author: Josh Rosen Date: 2015-12-30T20:47:42Z [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath. This supplants the checks added in SPARK-4123 / #5093, which are currently disabled due to bugs. This patch is based on pwendell's work in #8531. Closes #8531. Author: Josh Rosen Author: Patrick Wendell Closes #10461 from JoshRosen/SPARK-10359. commit e74675e859deb9fa839362620931b37ba4840b36 Author: Josh Rosen Date: 2016-01-01T04:23:19Z [SPARK-10359][PROJECT-INFRA] Multiple fixes to dev/test-dependencies.sh script This patch includes multiple fixes for the `dev/test-dependencies.sh` script (which was introduced in #10461): - Use `build/mvn --force` instead of `mvn` in one additional place. - Explicitly set a zero exit code on success. - Set `LC_ALL=C` to make `sort` results agree across machines (see https://stackoverflow.com/questions/28881/). - Set `should_run_build_tests=True` for `build` module (this somehow got lost). Author: Josh Rosen Closes #10543 from JoshRosen/dep-script-fixes. commit f43d1bdaabd101580e1ca7da59ce2e3d46392360 Author: Josh Rosen Date: 2016-01-04T09:04:29Z [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching This patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script. pwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead. I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail. Author: Josh Rosen Closes #10558 from JoshRosen/build-dep-tests-round-3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12504][SQL] [Backport-1.6] Masking cred...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10669#issuecomment-170316548 **[Test build #2357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2357/consoleFull)** for PR 10669 at commit [`212b4db`](https://github.com/apache/spark/commit/212b4dbf3c3a33c884d019068bdc6eb7fd25190c). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170315844 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170315845 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49052/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170315806 **[Test build #49052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49052/consoleFull)** for PR 10678 at commit [`5ca4630`](https://github.com/apache/spark/commit/5ca463035bc6eaebd15e7cf332faeea157e5593e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10678#discussion_r49271668
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -523,14 +523,37 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case s @ Sort(ordering, global, p @ Project(projectList, child))
if !s.resolved && p.resolved =>
-val (newOrdering, missing) = resolveAndFindMissing(ordering, p,
child)
+val (newOrdering, missing, newChild): (Seq[SortOrder],
Seq[Attribute], LogicalPlan) =
+ child match {
+case Project(
--- End diff --
yeah, thank you! Will change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10678#discussion_r49271654
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -523,14 +523,37 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case s @ Sort(ordering, global, p @ Project(projectList, child))
if !s.resolved && p.resolved =>
-val (newOrdering, missing) = resolveAndFindMissing(ordering, p,
child)
+val (newOrdering, missing, newChild): (Seq[SortOrder],
Seq[Attribute], LogicalPlan) =
+ child match {
+case Project(
--- End diff --
and it also becomes more clear later which operator is each
variable/expression for because you will need to specify "p1.projectList",
"w.projectList", etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10678#discussion_r49271651
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -523,14 +523,37 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case s @ Sort(ordering, global, p @ Project(projectList, child))
if !s.resolved && p.resolved =>
-val (newOrdering, missing) = resolveAndFindMissing(ordering, p,
child)
+val (newOrdering, missing, newChild): (Seq[SortOrder],
Seq[Attribute], LogicalPlan) =
+ child match {
+case Project(
--- End diff --
btw i think this is way too complicated. you can simplify it if you don't
specify all the fields, e.g.
```scala
case p1 @ Project(_, w @ Window(_, _, _, _, p2: Project)) =>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12735] Consolidate & move spark-ec2 to ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10673 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12735] Consolidate & move spark-ec2 to ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10673#issuecomment-170313629 @shivaram I'm going to merge this one first. Let me know if there is a problem with the move. I can easily revert this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10665 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10678#discussion_r49271406
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -523,14 +523,37 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case s @ Sort(ordering, global, p @ Project(projectList, child))
if !s.resolved && p.resolved =>
-val (newOrdering, missing) = resolveAndFindMissing(ordering, p,
child)
+val (newOrdering, missing, newChild): (Seq[SortOrder],
Seq[Attribute], LogicalPlan) =
--- End diff --
Sure, will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12504][SQL] [Backport-1.6] Masking cred...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10669#issuecomment-170312652 **[Test build #2357 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2357/consoleFull)** for PR 10669 at commit [`212b4db`](https://github.com/apache/spark/commit/212b4dbf3c3a33c884d019068bdc6eb7fd25190c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/10678#discussion_r49271363
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -523,14 +523,37 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case s @ Sort(ordering, global, p @ Project(projectList, child))
if !s.resolved && p.resolved =>
-val (newOrdering, missing) = resolveAndFindMissing(ordering, p,
child)
+val (newOrdering, missing, newChild): (Seq[SortOrder],
Seq[Attribute], LogicalPlan) =
--- End diff --
can you add some comment here about why we need two separate cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12504][SQL] [Backport-1.6] Masking cred...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10669#issuecomment-170308345 **[Test build #2356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2356/consoleFull)** for PR 10669 at commit [`212b4db`](https://github.com/apache/spark/commit/212b4dbf3c3a33c884d019068bdc6eb7fd25190c). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12504][SQL] [Backport-1.6] Masking cred...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10669#issuecomment-170308143 **[Test build #2356 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2356/consoleFull)** for PR 10669 at commit [`212b4db`](https://github.com/apache/spark/commit/212b4dbf3c3a33c884d019068bdc6eb7fd25190c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10678#issuecomment-170308011 **[Test build #49052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49052/consoleFull)** for PR 10678 at commit [`5ca4630`](https://github.com/apache/spark/commit/5ca463035bc6eaebd15e7cf332faeea157e5593e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12705] [SQL] AnalysisException: Sorting...
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/10678 [SPARK-12705] [SQL] AnalysisException: Sorting columns are not in Project of Window Function JIRA: https://issues.apache.org/jira/browse/SPARK-12705 Unable to resolve sorting columns if they are not in `outputSet` of `Window` Function. To fix this analysis failure, extra logics are added as what we are doing for a similar case: `Attributes` in `Sort` are not in the `outputSet of its Child node. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark sortWindows Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10678.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10678 commit c2fcaa8e488d12419c7b7c5032ccadab38f20b68 Author: gatorsmile Date: 2016-01-10T03:21:14Z window function: Sorting columns are not in Project --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12539][follow-up] always sort in partit...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10638#issuecomment-170305993 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12539][follow-up] always sort in partit...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10638#issuecomment-170305994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49051/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12539][follow-up] always sort in partit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10638#issuecomment-170305946 **[Test build #49051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49051/consoleFull)** for PR 10638 at commit [`247ddad`](https://github.com/apache/spark/commit/247ddadacb717c9f475f7a036626853ebee5ae4c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/10674#discussion_r49270748 --- Diff: network/common/pom.xml --- @@ -55,6 +55,7 @@ com.google.guava guava + compile --- End diff -- Ah, gotcha; (this is why I shouldn't review before morning coffee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10674#discussion_r49270677 --- Diff: network/common/pom.xml --- @@ -55,6 +55,7 @@ com.google.guava guava + compile --- End diff -- > Isn't compile the default scope? i.e. couldn't you leave out the explicit compile scope here and just not specify a scope? > Isn't compile the default scope? i.e. couldn't you leave out the explicit compile scope here and just not specify a scope? It's set to `provided` in the parent pom.xml: https://github.com/apache/spark/blob/b23c4521f5df905e4fe4d79dd5b670286e2697f7/pom.xml#L365 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12539][follow-up] always sort in partit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10638#issuecomment-170299362 **[Test build #49051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49051/consoleFull)** for PR 10638 at commit [`247ddad`](https://github.com/apache/spark/commit/247ddadacb717c9f475f7a036626853ebee5ae4c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170298109 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170298110 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49050/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170298048 **[Test build #49050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49050/consoleFull)** for PR 10677 at commit [`736e8d2`](https://github.com/apache/spark/commit/736e8d2d0c20c2e5eb089dfc011c40f20f7c90e6). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class BitwiseReverse(child: Expression, width: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170291988 **[Test build #49050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49050/consoleFull)** for PR 10677 at commit [`736e8d2`](https://github.com/apache/spark/commit/736e8d2d0c20c2e5eb089dfc011c40f20f7c90e6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10577#issuecomment-170291891 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49048/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10577#issuecomment-170291889 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10577#issuecomment-170291824 **[Test build #49048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49048/consoleFull)** for PR 10577 at commit [`f3d23dc`](https://github.com/apache/spark/commit/f3d23dc452afa5d6848cf4469751f33cf3036f57). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170287977 **[Test build #49049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49049/consoleFull)** for PR 10677 at commit [`0e8317d`](https://github.com/apache/spark/commit/0e8317db91e081a30beb22adfb8c3463adcc51ec). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class BitwiseReverse(child: Expression, width: Int)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170287981 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49049/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170287980 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10677#issuecomment-170287921 **[Test build #49049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49049/consoleFull)** for PR 10677 at commit [`0e8317d`](https://github.com/apache/spark/commit/0e8317db91e081a30beb22adfb8c3463adcc51ec). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12706] [SQL] grouping() and grouping_id...
GitHub user davies opened a pull request: https://github.com/apache/spark/pull/10677 [SPARK-12706] [SQL] grouping() and grouping_id() Grouping() returns a column is aggregated or not, grouping_id() returns the aggregation levels. grouping()/grouping_id() could be used with window function, but does not work in having/sort clause, will be fixed by another PR. You can merge this pull request into a Git repository by running: $ git pull https://github.com/davies/spark grouping Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10677.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10677 commit bcb8d9eecb1b83ae0d62fb88fc5a74c0e3fa9a88 Author: Davies Liu Date: 2016-01-09T07:55:30Z add grouping() and grouping_id() commit 0e8317db91e081a30beb22adfb8c3463adcc51ec Author: Davies Liu Date: 2016-01-09T21:47:31Z make GROUPING__ID compatible with Hive --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10577#issuecomment-170283685 **[Test build #49048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49048/consoleFull)** for PR 10577 at commit [`f3d23dc`](https://github.com/apache/spark/commit/f3d23dc452afa5d6848cf4469751f33cf3036f57). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/10577#issuecomment-170282770 The latest change contains the fix for multi-child Union ALL in conversion from logical plan to SQL after introducing SQL Generation: https://github.com/apache/spark/pull/10541 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12340] Fix overflow in various take fun...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10670 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12269][Streaming][Kinesis] Update aws-j...
Github user BrianLondon commented on the pull request: https://github.com/apache/spark/pull/10256#issuecomment-170270526 Manifest updated. Is there documented somewhere what all the tests Travis runs are? It seems to differ from what's described at http://spark.apache.org/docs/latest/building-spark.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170269614 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170269576 **[Test build #49047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49047/consoleFull)** for PR 10674 at commit [`16c228d`](https://github.com/apache/spark/commit/16c228d9227731e1228e75b419b06bcf29e8765c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170269616 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49047/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170268730 **[Test build #49047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49047/consoleFull)** for PR 10674 at commit [`16c228d`](https://github.com/apache/spark/commit/16c228d9227731e1228e75b419b06bcf29e8765c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user jaceklaskowski commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170268746 I'm using the following commands to do the build: ``` $ ./dev/change-scala-version.sh 2.11 $ ./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.7.1 -Dscala-2.11 -Phive -Phive-thriftserver -DskipTests clean install ``` (I've been using sbt but somehow it got "broken" lately, i.e. it doesn't build all modules - I'm going to report it later). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12436][SQL] Now InferSchema should retu...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10474#issuecomment-170268671 Can you add tests and also make changes to the write path (parquet/orc)? Otherwise, when you write a json dataset with null type to parquet, it will fail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user jaceklaskowski commented on a diff in the pull request: https://github.com/apache/spark/pull/10674#discussion_r49266731 --- Diff: network/common/pom.xml --- @@ -55,6 +55,7 @@ com.google.guava guava + compile --- End diff -- Thought so, but the line has fixed standalone Master to start (after the line got removed in https://github.com/apache/spark/commit/659fd9d04b988d48960eac4f352ca37066f43f5c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170268121 @jaceklaskowski, to help me understand the scope of this bug could you tell us how you built Spark in your failing example? Did you use the Maven or SBT builds? Which build profiles? I'm wondering because there it looks like there are some differences between how the two build tools handle the Guava dependency. Specifically, it looks like SBT always puts Guava in compile scope: https://github.com/apache/spark/blob/3d77cffec093bed4d330969f1a996f3358b9a772/pom.xml#L2302 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12735] Consolidate & move spark-ec2 to ...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/10673#issuecomment-170268088 Sounds fine with me and I agree with your position. :+1: I hope this makes it easier for people to contribute to spark-ec2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170267761 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/10674#discussion_r49266568 --- Diff: network/common/pom.xml --- @@ -55,6 +55,7 @@ com.google.guava guava + compile --- End diff -- Isn't compile the default scope? i.e. couldn't you leave out the explicit `compile` scope here and just not specify a scope? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12735] Consolidate & move spark-ec2 to ...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/10673#issuecomment-170267628 Given the scope of other deprecations / changes in Spark 2.0, I personally think that it's okay to leave out the wrapper script which forwards to the new repo. Note that I would support the wrapper / forwarder script if this change was targeted at a 1.x release, but for 2.x I think it's okay to just make this change and add migration instructions in the release notes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266453
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -219,13 +219,21 @@ object JdbcUtils extends Logging {
val sb = new StringBuilder()
val dialect = JdbcDialects.get(url)
df.schema.fields foreach { field => {
- val name = field.name
+ val name = dialect.quoteIdentifier(field.name)
val typ: String = getJdbcType(field.dataType,
dialect).databaseTypeDefinition
val nullable = if (field.nullable) "" else "NOT NULL"
sb.append(s", $name $typ $nullable")
}}
if (sb.length < 2) "" else sb.substring(2)
}
+
+ /**
+ * Parse the table name string for this RDD.
--- End diff --
This is confusing, since the variable `table` already sounds like it would
be the table name.
Can you call it `schemaQualifiedTableName` or something more explicit?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266438
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -90,6 +90,15 @@ abstract class JdbcDialect extends Serializable {
}
/**
+ * Parses a table name in case of containing special characters like .
seperating database name
+ * followed by table name (eg "some database"."some-table-name") or
+ * in case it contains characters that require quotes (e.g. space).
+ */
+ def parseTableName(tableName: String): String = {
--- End diff --
The method contract here is confusing to me: is this supposed to introduce
an additional layer of quotes? I find it confusing to couple both the
responsibilities of quoting identifiers and parsing table names from a
schema-qualified name.
I'd prefer to have this return just the table name, unquoted, and leave it
up to later components to escape it. If you _do_ think we should do quoting
here, please explain that part of the API contract in the Scaladoc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266410
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala ---
@@ -41,6 +41,26 @@ private case object MySQLDialect extends JdbcDialect {
override def quoteIdentifier(colName: String): String = {
s"`$colName`"
}
+
+ override def parseTableName(tableName: String): String = {
+val tableName1 = tableName.replace("\"", "").replace("\'", "")
--- End diff --
Also, `tableName1` is a bad name; if you were going to go with this
approach, I'd call it something like `tableNameWithoutQuotes` (but I'm not sure
that this unconditional quote-stripping is correct).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266387
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala ---
@@ -41,6 +41,26 @@ private case object MySQLDialect extends JdbcDialect {
override def quoteIdentifier(colName: String): String = {
s"`$colName`"
}
+
+ override def parseTableName(tableName: String): String = {
+val tableName1 = tableName.replace("\"", "").replace("\'", "")
--- End diff --
Why is it correct to just drop quotes here? What if the schema name also
contains a dot? Don't you need to keep track of quotes during the string
processing in order to figure out where the schema name ends and the table name
begins?
Note that both the schema and table names could contain dots (AFAIK, if not
please comment with a citation explaining why not), so simply grabbing the last
component after a dot might not be correct.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266392
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala ---
@@ -41,6 +41,26 @@ private case object MySQLDialect extends JdbcDialect {
override def quoteIdentifier(colName: String): String = {
s"`$colName`"
}
+
+ override def parseTableName(tableName: String): String = {
+val tableName1 = tableName.replace("\"", "").replace("\'", "")
+if (tableName1.contains(".")) {
+ val tableNameList = tableName1.split('.')
+ tableNameList.foldLeft("") { (leftStr, rightStr) =>
+if (!"".equals(rightStr.trim())) {
+ if ("".equals(leftStr.trim())) {
+leftStr + s"`$rightStr`"
+ } else {
+leftStr + "." + s"`$rightStr`"
+ }
+} else {
+ leftStr
+}
+ }
+} else {
+ s"`$tableName1`"
--- End diff --
Is this supposed to return quoted identifiers? Or should that be the
responsibility of another function? Just want to make sure that we're not
double-quoting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266372
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala ---
@@ -41,6 +41,26 @@ private case object MySQLDialect extends JdbcDialect {
override def quoteIdentifier(colName: String): String = {
s"`$colName`"
}
+
+ override def parseTableName(tableName: String): String = {
+val tableName1 = tableName.replace("\"", "").replace("\'", "")
+if (tableName1.contains(".")) {
+ val tableNameList = tableName1.split('.')
+ tableNameList.foldLeft("") { (leftStr, rightStr) =>
--- End diff --
Also, can you add unit tests for this function? _Not_ end-to-end tests, but
a test which exercises this method in isolation, similar to the ones that I
have in `spark-redshift`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49266355
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/MySQLDialect.scala ---
@@ -41,6 +41,26 @@ private case object MySQLDialect extends JdbcDialect {
override def quoteIdentifier(colName: String): String = {
s"`$colName`"
}
+
+ override def parseTableName(tableName: String): String = {
+val tableName1 = tableName.replace("\"", "").replace("\'", "")
+if (tableName1.contains(".")) {
+ val tableNameList = tableName1.split('.')
+ tableNameList.foldLeft("") { (leftStr, rightStr) =>
--- End diff --
Woah, this is kinda dense and confusing. Can you please rewrite this in a
simpler fashion and add comments? This is just my drive-by first impression.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12735] Consolidate & move spark-ec2 to ...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/10673#issuecomment-170266531 There was once talk of leaving the `spark-ec2` executable in here and just downloading the script from the AMPLab repo on execution. That way users can continue to use the script as though it lived in the Spark repo. Are we not doing that anymore? It's fine with me either way; I'm just checking. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user pjfanning commented on the pull request: https://github.com/apache/spark/pull/10403#issuecomment-170256145 Hi Naveen - your new tests look good to me. I do notice though that you probably need to resync with master due to conflicts (there is a 'This branch has conflicts that must be resolved' warning). https://help.github.com/articles/syncing-a-fork/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170247768 Yeah I think this is correct. The comment that was here previously suggested this was related to use of `Optional` in the public API, but I think it needs to stay in compile scope for the shading to work as desired. If there are no other thoughts today I'll merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12437][SQL] Encapsulate the table and c...
Github user naveenminchu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10403#discussion_r49264644
--- Diff:
docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala
---
@@ -150,4 +150,91 @@ class MySQLIntegrationSuite extends
DockerJDBCIntegrationSuite {
df2.write.jdbc(jdbcUrl, "datescopy", new Properties)
df3.write.jdbc(jdbcUrl, "stringscopy", new Properties)
}
+
+ test("Write test with SaveMode set to overwrite") {
--- End diff --
@pjfanning Added test as suggested
For example:
create table `escaped names` (`key` BIGINT, `long description` TEXT);
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12737]Decrease the redundant activeIds ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10676#issuecomment-170241170 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12737]Decrease the redundant activeIds ...
GitHub user qbwu opened a pull request: https://github.com/apache/spark/pull/10676 [SPARK-12737]Decrease the redundant activeIds sent to remote mirrors in "aggregateMessagesWithActiveSet" JIRA Issue: https://issues.apache.org/jira/browse/SPARK-12737 An optimization to the message exchanging in the computation model of GraphX, to reduce shuffle size in each graph computing turn, then improve the performance. You can merge this pull request into a Git repository by running: $ git pull https://github.com/qbwu/spark decrease_active_msgs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10676.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10676 commit 834e06bcb6ebdd90f5d5ab5363f85f9fb54a36eb Author: Qiubin Wu Date: 2016-01-09T13:05:11Z In "GraphIml.scala::aggregateMessagesWithActiveSet", decrease the redundant activeIds sent to remote mirrors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5273] [MLLIB] [DOCS] Improve documentat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10675#issuecomment-170240595 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49046/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5273] [MLLIB] [DOCS] Improve documentat...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10675#issuecomment-170240593 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5273] [MLLIB] [DOCS] Improve documentat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10675#issuecomment-170240566 **[Test build #49046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49046/consoleFull)** for PR 10675 at commit [`e5032e6`](https://github.com/apache/spark/commit/e5032e6577dd13a20ccb6b7101ecd853b0c64857). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5273] [MLLIB] [DOCS] Improve documentat...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10675#issuecomment-170240020 **[Test build #49046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49046/consoleFull)** for PR 10675 at commit [`e5032e6`](https://github.com/apache/spark/commit/e5032e6577dd13a20ccb6b7101ecd853b0c64857). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12436][SQL] Now InferSchema should retu...
Github user naveenminchu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10474#discussion_r49264010
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/InferSchema.scala
---
@@ -169,7 +169,7 @@ private[json] object InferSchema {
None
}
-case NullType => Some(StringType)
+case NullType => Some(NullType)
--- End diff --
@yhuai Now removed NullType case as suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5273] [MLLIB] [DOCS] Improve documentat...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/10675 [SPARK-5273] [MLLIB] [DOCS] Improve documentation examples for LinearRegression Use a much smaller step size in LinearRegressionWithSGD MLlib examples to achieve a reasonable RMSE. Our training folks hit this exact same issue when concocting an example and had the same solution. You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-5273 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10675.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10675 commit e5032e6577dd13a20ccb6b7101ecd853b0c64857 Author: Sean Owen Date: 2016-01-09T13:14:57Z Use a much smaller step size in LinearRegressionWithSGD MLlib examples to achieve a reasonable RMSE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user jaceklaskowski commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170238716 Yes, it does. I'm using the latest revision + the change. It's a serious issue since standalone Master cannot be started as of today. I do not know how it's supposed to have been fixed, but that's exactly what helped to resolve the issue. Any help appreciated to make it better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12177] [STREAMING] Update KafkaDStreams...
Github user nikit-os commented on the pull request: https://github.com/apache/spark/pull/10294#issuecomment-170237564 Kafka 0.9 introduce new consumer which eliminates the dependence on Zookeeper. It replaces "high-level" and "low-level" consumers and makes working with Kafka much easier. Also new producer and consumer support new security features (such as Kerberos, SSL, unix-like permissions system control). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170230870 I'm not sure about that; I believe Guava is supposed to be `provided` here. Does it fix the issue though? @vanzin how is this one supposed to work re: shading now that we don't need Guava in the public API? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10674#issuecomment-170228387 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12736][CORE][DEPLOY] Standalone Master ...
GitHub user jaceklaskowski opened a pull request: https://github.com/apache/spark/pull/10674 [SPARK-12736][CORE][DEPLOY] Standalone Master cannot be started due t⦠â¦o NoClassDefFoundError: org/spark-project/guava/collect/Maps /cc @srowen @rxin You can merge this pull request into a Git repository by running: $ git pull https://github.com/jaceklaskowski/spark SPARK-12736 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10674.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10674 commit 16c228d9227731e1228e75b419b06bcf29e8765c Author: Jacek Laskowski Date: 2016-01-09T11:34:07Z [SPARK-12736][CORE][DEPLOY] Standalone Master cannot be started due to NoClassDefFoundError: org/spark-project/guava/collect/Maps --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170224808 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49041/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170224807 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170224800 **[Test build #49041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49041/consoleFull)** for PR 10672 at commit [`798441a`](https://github.com/apache/spark/commit/798441ae25936f61c431c01a3d5d3578dd8442c9). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12656] [SQL] Implement Intersect with L...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10630#issuecomment-170220929 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12656] [SQL] Implement Intersect with L...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10630#issuecomment-170220930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49044/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12656] [SQL] Implement Intersect with L...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10630#issuecomment-170220857 **[Test build #49044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49044/consoleFull)** for PR 10630 at commit [`4372170`](https://github.com/apache/spark/commit/4372170f600eb25996c3aa4f09d569312c263686). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12735] Consolidate & move spark-ec2 to ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10673#issuecomment-170220757 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12734][BUILD] Fix Netty exclusion and u...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10672#issuecomment-170220723 LGTM. Other candidates for banning: the servlet API jars in all its packagings --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12340] Fix overflow in various take fun...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10670#issuecomment-170220644 Looks OK to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12340][SQL]fix Int overflow in the Spar...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/10619#issuecomment-170220630 @QiangCai you may want to close this one, and then make a new one that incorporates the original change and also https://github.com/apache/spark/pull/10670 once it's merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10577#issuecomment-170218897 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49045/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
