[GitHub] [spark] dilipbiswal commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
dilipbiswal commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494265327 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei removed a comment on issue #24610: [SPARK-27716][SQL] Complete the transactions support for part of jdbc datasource operations.
turboFei removed a comment on issue #24610: [SPARK-27716][SQL] Complete the transactions support for part of jdbc datasource operations. URL: https://github.com/apache/spark/pull/24610#issuecomment-494261809 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
SparkQA commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494266047 **[Test build #105601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105601/testReport)** for PR 24647 at commit [`e61c3f3`](https://github.com/apache/spark/commit/e61c3f337983f570f089ed3a55db15ff00d4f2d9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
beliefer commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494266375 > > I'm not sure about the contract here, whether providers are required to be stateless. If they're not then this would be a problem for another instance that has state, or if these acquire state at some point. > > While agreeing this point, extracting these calls into method would help readability. I'm not sure Spark community is open to allow minor refactor patch. Thanks for your review. I modified `providingInstance` with private, so as to advance the encapsulation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if functions
AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494266827 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266857 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105593/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
SparkQA commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494266657 **[Test build #105600 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105600/testReport)** for PR 24657 at commit [`6253e16`](https://github.com/apache/spark/commit/6253e161842404e6c9acb22a6e6e1968b2dcadcd). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL]
Optimize uncorrelated predicate subquery
URL: https://github.com/apache/spark/pull/24344#discussion_r285876899
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##
@@ -93,6 +93,21 @@ object SubqueryExpression {
}.isDefined
}
+ /**
+ * Returns true when an expression contains a correlated IN or correlated
EXISTS and
+ * false otherwise. An IN is non-correlated only if the left values are
foldable
+ * and the subquery has no outer references.
+ */
+ def hasCorrelatedInOrExists(e: Expression): Boolean = {
Review comment:
@francis0407 Even if we push down the LHS of IN into subquery, there is no
guarantee how much result we get back, right ? I think @cloud-fan needs to
bless this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
SparkQA commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-494266658 **[Test build #105596 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105596/testReport)** for PR 24643 at commit [`9e3fa8f`](https://github.com/apache/spark/commit/9e3fa8f958aa31e673a448af4dc1c3ed673881c9). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
SparkQA commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-49423 **[Test build #105599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105599/testReport)** for PR 24656 at commit [`00ac82a`](https://github.com/apache/spark/commit/00ac82a5a4f0fe67392c7fc757c78bd28d32c138). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266836 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105594/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-494266874 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105596/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
SparkQA commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266653 **[Test build #105594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105594/testReport)** for PR 24655 at commit [`6f01cee`](https://github.com/apache/spark/commit/6f01cee3855968f1185bd3d78104ca59ef044a21). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if functions
AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494266840 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105595/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
SparkQA commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266656 **[Test build #105593 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105593/testReport)** for PR 24655 at commit [`235d4cf`](https://github.com/apache/spark/commit/235d4cf4ae0c764c648b2ee451976e84e232cc1c). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266829 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-494266865 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-49420 **[Test build #105597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105597/testReport)** for PR 24344 at commit [`5b8f60a`](https://github.com/apache/spark/commit/5b8f60a19a5600e5269de61704547371a3773a7b). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266848 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494266723 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
AmplabJenkins commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-494266745 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105598/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24335: [SPARK-27425][SQL] Add count_if functions
SparkQA commented on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494266652 **[Test build #105595 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105595/testReport)** for PR 24335 at commit [`e4f0465`](https://github.com/apache/spark/commit/e4f04654618b1644678c18c4331418fddb2982e8). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494266719 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494266736 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105600/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494266733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105597/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
SparkQA commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-49421 **[Test build #105598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105598/testReport)** for PR 21586 at commit [`a638a53`](https://github.com/apache/spark/commit/a638a5354a5ecb297bb49453c5a40d1227d10f15). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494266729 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494266724 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105599/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
AmplabJenkins commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-494266732 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494267746 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
AmplabJenkins removed a comment on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-494266732 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494266733 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105597/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-494266874 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105596/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
dilipbiswal commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494267299 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494258025 **[Test build #105597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105597/testReport)** for PR 24344 at commit [`5b8f60a`](https://github.com/apache/spark/commit/5b8f60a19a5600e5269de61704547371a3773a7b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
SparkQA removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494249540 **[Test build #105594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105594/testReport)** for PR 24655 at commit [`6f01cee`](https://github.com/apache/spark/commit/6f01cee3855968f1185bd3d78104ca59ef044a21). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
AmplabJenkins removed a comment on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494266724 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105599/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
SparkQA removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-494252806 **[Test build #105596 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105596/testReport)** for PR 24643 at commit [`9e3fa8f`](https://github.com/apache/spark/commit/9e3fa8f958aa31e673a448af4dc1c3ed673881c9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
dilipbiswal commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494267436 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494266723 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266848 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if functions
AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494266827 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
SparkQA removed a comment on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494261955 **[Test build #105599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105599/testReport)** for PR 24656 at commit [`00ac82a`](https://github.com/apache/spark/commit/00ac82a5a4f0fe67392c7fc757c78bd28d32c138). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
SparkQA removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494263952 **[Test build #105600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105600/testReport)** for PR 24657 at commit [`6253e16`](https://github.com/apache/spark/commit/6253e161842404e6c9acb22a6e6e1968b2dcadcd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
AmplabJenkins removed a comment on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-494266745 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105598/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
SparkQA removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494248027 **[Test build #105593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105593/testReport)** for PR 24655 at commit [`235d4cf`](https://github.com/apache/spark/commit/235d4cf4ae0c764c648b2ee451976e84e232cc1c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
AmplabJenkins removed a comment on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494266719 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266836 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105594/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-494266865 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494266729 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266829 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if functions
SparkQA removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494249556 **[Test build #105595 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105595/testReport)** for PR 24335 at commit [`e4f0465`](https://github.com/apache/spark/commit/e4f04654618b1644678c18c4331418fddb2982e8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494266736 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105600/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
SparkQA removed a comment on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-494258026 **[Test build #105598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105598/testReport)** for PR 21586 at commit [`a638a53`](https://github.com/apache/spark/commit/a638a5354a5ecb297bb49453c5a40d1227d10f15). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if functions
AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494266840 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105595/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494266857 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105593/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer edited a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
beliefer edited a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494214438 > I'm not sure about the contract here, whether providers are required to be stateless. If they're not then this would be a problem for another instance that has state, or if these acquire state at some point. First of all, I am glad to see your reply. I have the same question as you.I investigate all the providers and found every implementation is stateless indeed. I even investigated the providers provided by third parties, such as `com.mongodb.spark.sql.DefaultSource`,`org.elasticsearch.spark.sql.DefaultSource`. Another reason I created this PR is `providingClass.getConstructor().newInstance()` is only called in `org.apache.spark.sql.execution.datasources.DataSource`. The behavior provides a good encapsulation. I modified providingInstance with private, so as to advance the encapsulation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-49426 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494267792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10856/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly.
beliefer commented on a change in pull request #24372: [SPARK-27462][SQL]
Enhance insert into hive table that could choose some columns in target table
flexibly.
URL: https://github.com/apache/spark/pull/24372#discussion_r284222739
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -332,24 +334,58 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
private def preprocess(
insert: InsertIntoTable,
tblName: String,
+ insertedCols: Option[Seq[String]],
partColNames: Seq[String]): InsertIntoTable = {
val normalizedPartSpec = PartitioningUtils.normalizePartitionSpec(
insert.partition, partColNames, tblName, conf.resolver)
val staticPartCols = normalizedPartSpec.filter(_._2.isDefined).keySet
+val selectedCols = if (insertedCols == None) {
+ insert.table.output
+} else {
+ val tableCols = insert.table.output.map(_.name)
+ val noexistsCols = insertedCols.get.filterNot(col =>
tableCols.contains(col))
+ if (noexistsCols.size > 0) {
+throw new AnalysisException(s"Table $tblName does not exists these
columns: $noexistsCols.")
+ }
+ insert.table.output.filter(a => insertedCols.get.contains(a.name))
+}
val expectedColumns = insert.table.output.filterNot(a =>
staticPartCols.contains(a.name))
if (expectedColumns.length != insert.query.schema.length) {
throw new AnalysisException(
-s"$tblName requires that the data to be inserted have the same number
of columns as the " +
- s"target table: target table has ${insert.table.output.size}
column(s) but the " +
- s"inserted data has ${insert.query.output.length +
staticPartCols.size} column(s), " +
+s"$tblName requires that the data to be inserted have the same number
of columns as " +
+ s"the number of columns selected in the target table: the number of
columns selected " +
+ s"has ${expectedColumns.length + staticPartCols.size} column(s) but
the inserted data " +
+ s"has ${insert.query.output.length + staticPartCols.size} column(s),
" +
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
+val tableColumns = insert.table.output.filterNot(a =>
staticPartCols.contains(a.name))
+val tableCols = tableColumns.map(_.name)
+val filledQuery = if (insertedCols == None) {
+ insert.query
+} else {
+ // Because `HiveFileFormat` writes data according to the index of
columns which belongs
+ // target table, in order to align the data, we need to fill in some
empty expressions.
+ val cols = insertedCols.get
+ val project = insert.query.asInstanceOf[Project]
+ val filledProjectList = ArrayBuffer.empty[NamedExpression]
+ var i = 0
+ tableCols.foreach { tableCol =>
+if (cols.contains(tableCol)) {
+ filledProjectList += project.projectList(i)
+ i += 1
+} else {
+ filledProjectList += Alias(Literal(null, NullType), "NULL")()
Review comment:
is there any other considerations? @cloud-fan
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494267784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10855/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer edited a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
beliefer edited a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494214438 > I'm not sure about the contract here, whether providers are required to be stateless. If they're not then this would be a problem for another instance that has state, or if these acquire state at some point. First of all, I am glad to see your reply. I have the same question as you.I investigate all the providers and found every implementation is stateless indeed. I even investigated the providers provided by third parties, such as `com.mongodb.spark.sql.DefaultSource`,`org.elasticsearch.spark.sql.DefaultSource`. Another reason I created this PR is `providingClass.getConstructor().newInstance()` is only called in `org.apache.spark.sql.execution.datasources.DataSource`. The behavior provides a good encapsulation. I modified `providingInstance` with private, so as to advance the encapsulation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494267784 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10855/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
SparkQA commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494268394 **[Test build #105603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105603/testReport)** for PR 24655 at commit [`6f01cee`](https://github.com/apache/spark/commit/6f01cee3855968f1185bd3d78104ca59ef044a21). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494267786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
SparkQA commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494268392 **[Test build #105602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105602/testReport)** for PR 24657 at commit [`6253e16`](https://github.com/apache/spark/commit/6253e161842404e6c9acb22a6e6e1968b2dcadcd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24335: [SPARK-27425][SQL] Add count_if functions
dilipbiswal commented on issue #24335: [SPARK-27425][SQL] Add count_if functions URL: https://github.com/apache/spark/pull/24335#issuecomment-494268144 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494268418 **[Test build #105604 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105604/testReport)** for PR 24344 at commit [`5b8f60a`](https://github.com/apache/spark/commit/5b8f60a19a5600e5269de61704547371a3773a7b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494267750 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10854/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] francis0407 commented on a change in pull request #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
francis0407 commented on a change in pull request #24344: [SPARK-27440][SQL]
Optimize uncorrelated predicate subquery
URL: https://github.com/apache/spark/pull/24344#discussion_r285879099
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##
@@ -93,6 +93,21 @@ object SubqueryExpression {
}.isDefined
}
+ /**
+ * Returns true when an expression contains a correlated IN or correlated
EXISTS and
+ * false otherwise. An IN is non-correlated only if the left values are
foldable
+ * and the subquery has no outer references.
+ */
+ def hasCorrelatedInOrExists(e: Expression): Boolean = {
Review comment:
Push down the LHS of IN into the subquery is the first step. We can also use
a specific aggregation to reduce the result set to an acceptable size.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24627: [SPARK-27748][SS] Kafka consumer/producer password/token redaction.
HeartSaVioR commented on issue #24627: [SPARK-27748][SS] Kafka consumer/producer password/token redaction. URL: https://github.com/apache/spark/pull/24627#issuecomment-494267955 I know this patch just apply redaction to all the places which print configuration as of now, but I'm feeling that Kafka module is too verbose on configuration, especially KafkaConfigUpdater. Assuming debug log level is turned on, you would majorly want to track which consumers are being acquired or evicted, or closed. Logging in KafkaConfigUpdater doesn't help either. Let's revisit the purpose of logging. I guess we are logging to find when consumer/producer is created, and when it's evicted/closed. (either tracking each instance, or counting for each key). Though we use whole configuration as a key of cache, we (as human) cannot check every key-value pairs to find same instance. Some pairs play as a differentiator and we (again, as human) just assume other pairs are not different. So IMHO we don't even need to print out all configurations. Especially, if we really want to track each instance, explicit ID (as #19096 introduces) would work better and much helpful to debug. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494267786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494267792 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10856/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-49426 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494267750 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10854/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494267746 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
dilipbiswal commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494268025 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-49427 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10857/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494269994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins removed a comment on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-49427 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10857/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL]
Optimize uncorrelated predicate subquery
URL: https://github.com/apache/spark/pull/24344#discussion_r285880242
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##
@@ -93,6 +93,21 @@ object SubqueryExpression {
}.isDefined
}
+ /**
+ * Returns true when an expression contains a correlated IN or correlated
EXISTS and
+ * false otherwise. An IN is non-correlated only if the left values are
foldable
+ * and the subquery has no outer references.
+ */
+ def hasCorrelatedInOrExists(e: Expression): Boolean = {
Review comment:
@francis0407 Great.. thanks.. when we do the aggregation, we can still
outperform the existing left semi join approach, right ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded
AmplabJenkins commented on issue #24655: [SPARK-27786][SQL] Fix Sha1, Md5, and Base64 codegen when commons-codec is shaded URL: https://github.com/apache/spark/pull/24655#issuecomment-494269994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile
AmplabJenkins removed a comment on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile URL: https://github.com/apache/spark/pull/24620#issuecomment-494273096 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105605/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
SparkQA commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494272808 **[Test build #105606 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105606/testReport)** for PR 24656 at commit [`00ac82a`](https://github.com/apache/spark/commit/00ac82a5a4f0fe67392c7fc757c78bd28d32c138). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494271545 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105602/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] francis0407 commented on a change in pull request #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
francis0407 commented on a change in pull request #24344: [SPARK-27440][SQL]
Optimize uncorrelated predicate subquery
URL: https://github.com/apache/spark/pull/24344#discussion_r285882617
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##
@@ -93,6 +93,21 @@ object SubqueryExpression {
}.isDefined
}
+ /**
+ * Returns true when an expression contains a correlated IN or correlated
EXISTS and
+ * false otherwise. An IN is non-correlated only if the left values are
foldable
+ * and the subquery has no outer references.
+ */
+ def hasCorrelatedInOrExists(e: Expression): Boolean = {
Review comment:
Yes :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL]
Optimize uncorrelated predicate subquery
URL: https://github.com/apache/spark/pull/24344#discussion_r285884457
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##
@@ -93,6 +93,21 @@ object SubqueryExpression {
}.isDefined
}
+ /**
+ * Returns true when an expression contains a correlated IN or correlated
EXISTS and
+ * false otherwise. An IN is non-correlated only if the left values are
foldable
+ * and the subquery has no outer references.
+ */
+ def hasCorrelatedInOrExists(e: Expression): Boolean = {
+e.find {
+ case InSubquery(values, ListQuery(_, children, _, _)) =>
+values.exists(!_.foldable) || children.nonEmpty
+ case Exists(_, children, _) =>
+children.nonEmpty
+ case _ => false
+}.isDefined
+ }
+
Review comment:
@francis0407 never mind .. i think it wouldn't be a InSubquery..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494272233 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10858/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
dilipbiswal commented on a change in pull request #24344: [SPARK-27440][SQL]
Optimize uncorrelated predicate subquery
URL: https://github.com/apache/spark/pull/24344#discussion_r285883266
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##
@@ -93,6 +93,21 @@ object SubqueryExpression {
}.isDefined
}
+ /**
+ * Returns true when an expression contains a correlated IN or correlated
EXISTS and
+ * false otherwise. An IN is non-correlated only if the left values are
foldable
+ * and the subquery has no outer references.
+ */
+ def hasCorrelatedInOrExists(e: Expression): Boolean = {
+e.find {
+ case InSubquery(values, ListQuery(_, children, _, _)) =>
+values.exists(!_.foldable) || children.nonEmpty
+ case Exists(_, children, _) =>
+children.nonEmpty
+ case _ => false
+}.isDefined
+ }
+
Review comment:
@francis0407 Like wenchen, i can see a source of confusion here. So if we
have a query like
select * from t1 where c1 in (1, 2), we will flag is at correlated, right ?
I understand what you are doing here :-). But, future caller of this function
may not understand the subtle difference (since its an utility function) ? What
do you think ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24657: [MINOR][DOCS] Fix Spark hive example.
SparkQA commented on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494271439 **[Test build #105602 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105602/testReport)** for PR 24657 at commit [`6253e16`](https://github.com/apache/spark/commit/6253e161842404e6c9acb22a6e6e1968b2dcadcd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile
SparkQA commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile URL: https://github.com/apache/spark/pull/24620#issuecomment-494272831 **[Test build #105607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105607/testReport)** for PR 24620 at commit [`32fc1ca`](https://github.com/apache/spark/commit/32fc1ca8ee61e98c0edef2a1a6fb65860d6e6343). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
HeartSaVioR commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494273198 I'm not sure you're understanding the points here. Just assuming we don't want to change the current behavior (but open to performance boost), this patch is different from current for two points of view: 1) This patch will show different behavior if provider provides different instance per constructor call (this is actually obvious) and the instance, or DataSource leverages the fact. 2) This patch will show different behavior if instance is stateful, like calling method changes internal state. That might be what you already checked before (as you're saying all the implementations you've checked are stateless), but it's not guaranteed by interface contract. I'm not saying something is better and something is worse - just would like to let you understand the point what @srowen is saying (and what I've agreed). If this change brings enough value it might be possible to add the contract in interface, but I'm not the one to right to weigh the value. That's why I just only see the chance to refactor (extract the call to method) to simplify the code, nothing else. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
SparkQA removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494268392 **[Test build #105602 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105602/testReport)** for PR 24657 at commit [`6253e16`](https://github.com/apache/spark/commit/6253e161842404e6c9acb22a6e6e1968b2dcadcd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile
AmplabJenkins removed a comment on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile URL: https://github.com/apache/spark/pull/24620#issuecomment-494273093 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile
AmplabJenkins commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile URL: https://github.com/apache/spark/pull/24620#issuecomment-494273093 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example.
AmplabJenkins removed a comment on issue #24657: [MINOR][DOCS] Fix Spark hive example. URL: https://github.com/apache/spark/pull/24657#issuecomment-494271540 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
AmplabJenkins commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494272230 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile
SparkQA commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile URL: https://github.com/apache/spark/pull/24620#issuecomment-494273083 **[Test build #105605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105605/testReport)** for PR 24620 at commit [`0f2eb07`](https://github.com/apache/spark/commit/0f2eb07f689c2981625cd4fb16fce6d49ce8f11e). * This patch **fails to build**. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile
SparkQA commented on issue #24620: [SPARK-27737][SQL][test-hadoop3.2] Upgrade to Hive 2.3.5 for Hive Metastore Client and Hadoop-3.2 profile URL: https://github.com/apache/spark/pull/24620#issuecomment-494270566 **[Test build #105605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105605/testReport)** for PR 24620 at commit [`0f2eb07`](https://github.com/apache/spark/commit/0f2eb07f689c2981625cd4fb16fce6d49ce8f11e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg
zhengruifeng commented on issue #24656: [SPARK-27787][ML] Eliminate uncessary job to compute SSreg URL: https://github.com/apache/spark/pull/24656#issuecomment-494270505 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
