[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13210 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220821887 I am merging this to master and 2.0. Thank you for reviewing! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220813852 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59087/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220813851 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220813817 **[Test build #59087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59087/consoleFull)** for PR 13210 at commit [`abc12a5`](https://github.com/apache/spark/commit/abc12a5ee606282b069ff0c326a2f32d4ed2fbe2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220811819 **[Test build #59087 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59087/consoleFull)** for PR 13210 at commit [`abc12a5`](https://github.com/apache/spark/commit/abc12a5ee606282b069ff0c326a2f32d4ed2fbe2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220751946 **[Test build #59055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59055/consoleFull)** for PR 13210 at commit [`4e1e486`](https://github.com/apache/spark/commit/4e1e4862197854b3d40b537a7d45787b71399c56). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220751950 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59055/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220751949 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220751874 **[Test build #59055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59055/consoleFull)** for PR 13210 at commit [`4e1e486`](https://github.com/apache/spark/commit/4e1e4862197854b3d40b537a7d45787b71399c56). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220732096 LGTM pending grammar fixes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220706130 **[Test build #3007 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3007/consoleFull)** for PR 13210 at commit [`65d45a9`](https://github.com/apache/spark/commit/65d45a947e905ee14fd8a7556032dd5035182648). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220683834 **[Test build #3007 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3007/consoleFull)** for PR 13210 at commit [`65d45a9`](https://github.com/apache/spark/commit/65d45a947e905ee14fd8a7556032dd5035182648). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220517869 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58936/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220517868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220517778 **[Test build #58936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58936/consoleFull)** for PR 13210 at commit [`65d45a9`](https://github.com/apache/spark/commit/65d45a947e905ee14fd8a7556032dd5035182648). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220511611 cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13210#discussion_r63988111
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -55,10 +55,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoTable =>
throwError("InsertIntoTable is not supported with streaming
DataFrames/Datasets")
-case Aggregate(_, _, child) if child.isStreaming && outputMode ==
Append =>
- throwError(
-"Aggregations are not supported on streaming
DataFrames/Datasets in " +
- "Append output mode. Consider changing output mode to
Update.")
+case Aggregate(_, _, child) if child.isStreaming =>
+ if (outputMode == Append) {
+throwError(
+ "Aggregations are not supported on streaming
DataFrames/Datasets in " +
+"Append output mode. Consider changing output mode to
Update.")
--- End diff --
I didnt get you. IntelliJ seems to catching all the uses of Append object
properly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13210#discussion_r63988091
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -55,10 +55,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoTable =>
throwError("InsertIntoTable is not supported with streaming
DataFrames/Datasets")
-case Aggregate(_, _, child) if child.isStreaming && outputMode ==
Append =>
- throwError(
-"Aggregations are not supported on streaming
DataFrames/Datasets in " +
- "Append output mode. Consider changing output mode to
Update.")
+case Aggregate(_, _, child) if child.isStreaming =>
+ if (outputMode == Append) {
+throwError(
+ "Aggregations are not supported on streaming
DataFrames/Datasets in " +
+"Append output mode. Consider changing output mode to
Update.")
+ }
+ val moreStreamingAggregates = child.find {
+case Aggregate(_, _, grandchild) if grandchild.isStreaming =>
true
+case _ => false
+ }
+ if (moreStreamingAggregates.nonEmpty) {
+throwError("Multiple streaming aggregations are not supported
DataFrames/Datasets")
--- End diff --
will fix. thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/13210#discussion_r63987926
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -55,10 +55,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoTable =>
throwError("InsertIntoTable is not supported with streaming
DataFrames/Datasets")
-case Aggregate(_, _, child) if child.isStreaming && outputMode ==
Append =>
- throwError(
-"Aggregations are not supported on streaming
DataFrames/Datasets in " +
- "Append output mode. Consider changing output mode to
Update.")
+case Aggregate(_, _, child) if child.isStreaming =>
+ if (outputMode == Append) {
+throwError(
+ "Aggregations are not supported on streaming
DataFrames/Datasets in " +
+"Append output mode. Consider changing output mode to
Update.")
+ }
+ val moreStreamingAggregates = child.find {
+case Aggregate(_, _, grandchild) if grandchild.isStreaming =>
true
+case _ => false
+ }
+ if (moreStreamingAggregates.nonEmpty) {
--- End diff --
TreeNode.find returns an Option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/13210#discussion_r63987729
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -55,10 +55,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoTable =>
throwError("InsertIntoTable is not supported with streaming
DataFrames/Datasets")
-case Aggregate(_, _, child) if child.isStreaming && outputMode ==
Append =>
- throwError(
-"Aggregations are not supported on streaming
DataFrames/Datasets in " +
- "Append output mode. Consider changing output mode to
Update.")
+case Aggregate(_, _, child) if child.isStreaming =>
+ if (outputMode == Append) {
+throwError(
+ "Aggregations are not supported on streaming
DataFrames/Datasets in " +
+"Append output mode. Consider changing output mode to
Update.")
+ }
+ val moreStreamingAggregates = child.find {
+case Aggregate(_, _, grandchild) if grandchild.isStreaming =>
true
+case _ => false
+ }
+ if (moreStreamingAggregates.nonEmpty) {
--- End diff --
What is the type of `moreStreamingAggregates`? Option? What do you think
about `foreach`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/13210#discussion_r63987696
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -55,10 +55,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoTable =>
throwError("InsertIntoTable is not supported with streaming
DataFrames/Datasets")
-case Aggregate(_, _, child) if child.isStreaming && outputMode ==
Append =>
- throwError(
-"Aggregations are not supported on streaming
DataFrames/Datasets in " +
- "Append output mode. Consider changing output mode to
Update.")
+case Aggregate(_, _, child) if child.isStreaming =>
+ if (outputMode == Append) {
+throwError(
+ "Aggregations are not supported on streaming
DataFrames/Datasets in " +
+"Append output mode. Consider changing output mode to
Update.")
--- End diff --
I'd suggest changing "Append" to using the Append value and Update too, so
when you refactor the code it'd be easier to catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/13210#discussion_r63987667
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
---
@@ -55,10 +55,19 @@ object UnsupportedOperationChecker {
case _: InsertIntoTable =>
throwError("InsertIntoTable is not supported with streaming
DataFrames/Datasets")
-case Aggregate(_, _, child) if child.isStreaming && outputMode ==
Append =>
- throwError(
-"Aggregations are not supported on streaming
DataFrames/Datasets in " +
- "Append output mode. Consider changing output mode to
Update.")
+case Aggregate(_, _, child) if child.isStreaming =>
+ if (outputMode == Append) {
+throwError(
+ "Aggregations are not supported on streaming
DataFrames/Datasets in " +
+"Append output mode. Consider changing output mode to
Update.")
+ }
+ val moreStreamingAggregates = child.find {
+case Aggregate(_, _, grandchild) if grandchild.isStreaming =>
true
+case _ => false
+ }
+ if (moreStreamingAggregates.nonEmpty) {
+throwError("Multiple streaming aggregations are not supported
DataFrames/Datasets")
--- End diff --
"are not supported DataFrames/Datasets"? It appears as "in" or "for" might
be missing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15428][SQL] Disable multiple streaming ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13210#issuecomment-220509194 **[Test build #58936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58936/consoleFull)** for PR 13210 at commit [`65d45a9`](https://github.com/apache/spark/commit/65d45a947e905ee14fd8a7556032dd5035182648). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
