[GitHub] spark issue #17308: [SPARK-19968][SPARK-20737][SS] Use a cached instance of ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17308 **[Test build #77293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77293/testReport)** for PR 17308 at commit [`039d063`](https://github.com/apache/spark/commit/039d063af502586109afb0ecd135390c4b7d2050). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17455: [Spark-20044][Web UI] Support Spark UI behind front-end ...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/17455 Usually new features aren't added to branches after they've been cut, and almost never after an rc has been cut. So given branch-2.2 was cut over a month ago and we're about to cut rc3, I'm pretty sure this will not make it into 2.2.0 sorry. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77292/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77292/testReport)** for PR 17770 at commit [`eb0598e`](https://github.com/apache/spark/commit/eb0598eeeaffe310d39d5bb501ff15fca272e2a7). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedHint(name: String, parameters: Seq[String], child: LogicalPlan)` * `case class ResolvedHint(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77291/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77291/testReport)** for PR 17770 at commit [`1c1cc9d`](https://github.com/apache/spark/commit/1c1cc9d1597d16deab14afb3f7001b13bc705321). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77292/testReport)** for PR 17770 at commit [`eb0598e`](https://github.com/apache/spark/commit/eb0598eeeaffe310d39d5bb501ff15fca272e2a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77290/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77290/testReport)** for PR 17770 at commit [`2bf0590`](https://github.com/apache/spark/commit/2bf059059aabf5c9d25f5606c883cb28261ad535). * This patch **fails to build**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18075 Thank you. Absolutely, it is easier to review this change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77291/testReport)** for PR 17770 at commit [`1c1cc9d`](https://github.com/apache/spark/commit/1c1cc9d1597d16deab14afb3f7001b13bc705321). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18075#discussion_r118165399
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -792,7 +887,18 @@ class CodegenContext {
addMutableState(javaType(expr.dataType), value,
s"$value = ${defaultValue(expr.dataType)};")
- subexprFunctions += s"$fnName($INPUT_ROW);"
+ // Generate the code for this expression tree and wrap it in a
function.
--- End diff --
Is there any reason to move this code block from the original place to here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17770: [SPARK-20392][SQL] Set barrier to prevent re-ente...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/17770#discussion_r118165051
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -174,17 +174,19 @@ class Dataset[T] private[sql](
this(sqlContext.sparkSession, logicalPlan, encoder)
}
- @transient private[sql] val logicalPlan: LogicalPlan = {
+ @transient private val logicalPlan: LogicalPlan = {
--- End diff --
Exposing the logical plan wrapped with the barrier tends to cause problem.
So mark it as `private`.
We have `queryExecution.analyzed` to access the analyzed logical plan.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77290/testReport)** for PR 17770 at commit [`2bf0590`](https://github.com/apache/spark/commit/2bf059059aabf5c9d25f5606c883cb28261ad535). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17455: [Spark-20044][Web UI] Support Spark UI behind front-end ...
Github user agsimeonov commented on the issue: https://github.com/apache/spark/pull/17455 Any chance this can get into v2.2.0? I am using Spark in Docker with nginx reverse proxy and this would help immensely. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18076 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77278/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18076 **[Test build #77278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77278/testReport)** for PR 18076 at commit [`bc66ec5`](https://github.com/apache/spark/commit/bc66ec52adf0f741a0c533b28ca64e3fef9e848e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18079: [SPARK-20841][SQL] Support column aliases for catalog ta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18079 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77280/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18079: [SPARK-20841][SQL] Support column aliases for catalog ta...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18079 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18079: [SPARK-20841][SQL] Support column aliases for catalog ta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18079 **[Test build #77280 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77280/testReport)** for PR 18079 at commit [`902d2a3`](https://github.com/apache/spark/commit/902d2a35740f3c3bc0c97aae56c65d9d25df3a15). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedRelation(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18083: [SPARK-20863] Add metrics/instrumentation to LiveListene...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18083 **[Test build #77289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77289/testReport)** for PR 18083 at commit [`a46c247`](https://github.com/apache/spark/commit/a46c24766fc2d533be82cc709948b37383e68121). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to replace ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77282/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to replace ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16225 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to replace ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16225 **[Test build #77282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77282/testReport)** for PR 16225 at commit [`b5424d9`](https://github.com/apache/spark/commit/b5424d9fea56d2e0fb57ebc27d3d35054da6d22b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18075: [SPARK-18016][SQL][CATALYST] Code Generation: Con...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18075#discussion_r118162963
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -629,7 +736,9 @@ class CodegenContext {
/**
* Splits the generated code of expressions into multiple functions,
because function has
- * 64kb code size limit in JVM
+ * 64kb code size limit in JVM. If the class the function is to be
inlined to would grow beyond
+ * 1600kb, a private, netsted sub-class is declared, and the function is
inlined to it, because
--- End diff --
nit: netsted -> nested?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77286/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/18076 LGTM as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77286/testReport)** for PR 17770 at commit [`c0bee01`](https://github.com/apache/spark/commit/c0bee014eaa268014f5e156498d8cc7d90533ac7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77284/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77284 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77284/testReport)** for PR 17770 at commit [`f3e4208`](https://github.com/apache/spark/commit/f3e4208eb23bee5cfc0e8a33134d58fac5526dbb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18073: [SPARK-20848][SQL] Shutdown the pool after reading parqu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18073 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77279/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18073: [SPARK-20848][SQL] Shutdown the pool after reading parqu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18073 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18073: [SPARK-20848][SQL] Shutdown the pool after reading parqu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18073 **[Test build #77279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77279/testReport)** for PR 18073 at commit [`14e09aa`](https://github.com/apache/spark/commit/14e09aafb8ee0cc9f5f1e95bc41b19eeab6fc7d4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77283/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77283/testReport)** for PR 17770 at commit [`5c53f0f`](https://github.com/apache/spark/commit/5c53f0f4f7488cb69ca8107f2c95e69ea333f11f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18023: [SPARK-12139] [SQL] REGEX Column Specification
Github user janewangfb commented on the issue: https://github.com/apache/spark/pull/18023 @cloud-fan, there are difference between `xyz` and xyz. we always need to extract xyz part from `xyz`, so we will need to pattern match. if we do not get pattern matched, we know it is not regex, then why bother to use UnsolvedRegex, which will need to project list expansion later. Hive supper regex column specification, see https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11974 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77287/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11974 **[Test build #77287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77287/testReport)** for PR 11974 at commit [`46d9c7b`](https://github.com/apache/spark/commit/46d9c7bbbfdd663a19212c2ddc7431ccd6293022). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11974 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18023: [SPARK-12139] [SQL] REGEX Column Specification
Github user janewangfb commented on a diff in the pull request:
https://github.com/apache/spark/pull/18023#discussion_r118160405
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
---
@@ -1230,25 +1230,37 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
}
/**
- * Create a dereference expression. The return type depends on the type
of the parent, this can
- * either be a [[UnresolvedAttribute]] (if the parent is an
[[UnresolvedAttribute]]), or an
- * [[UnresolvedExtractValue]] if the parent is some expression.
+ * Create a dereference expression. The return type depends on the type
of the parent.
+ * If the parent is an [[UnresolvedAttribute]], it can be a
[[UnresolvedAttribute]] or
+ * a [[UnresolvedRegex]] for regex quoted in ``; if the parent is some
other expression,
+ * it can be [[UnresolvedExtractValue]].
*/
override def visitDereference(ctx: DereferenceContext): Expression =
withOrigin(ctx) {
val attr = ctx.fieldName.getText
expression(ctx.base) match {
- case UnresolvedAttribute(nameParts) =>
-UnresolvedAttribute(nameParts :+ attr)
+ case unresolved_attr @ UnresolvedAttribute(nameParts) =>
--- End diff --
this wont work. In your first "case", ctx.fieldName.getStart.getText is
`XYZ`, nameparts is XYZ. and the table part should come from ctx.base.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17770 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77281/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77281/testReport)** for PR 17770 at commit [`f5f0524`](https://github.com/apache/spark/commit/f5f0524ef47cf0f1b2f3a32f28be0251129feabe). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18023: [SPARK-12139] [SQL] REGEX Column Specification
Github user janewangfb commented on a diff in the pull request:
https://github.com/apache/spark/pull/18023#discussion_r118159701
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala
---
@@ -177,6 +177,12 @@ object ParserUtils {
sb.toString()
}
+ /** the column name pattern in quoted regex without qualifier */
+ val escapedIdentifier = "`(.+)`".r
+
+ /** the column name pattern in quoted regex with qualifier */
+ val qualifiedEscapedIdentifier = ("(.+)" + """.""" + "`(.+)`").r
--- End diff --
when the config is on, we need to extract XYZ from `XYZ` pattern, thats why
we need these patterns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18059: [SPARK-20834][SQL]TypeCoercion:loss of precision when wi...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18059 @cloud-fan The current behavior always using `float` , but using `double` can reduce loss of precision,so ,I think using `double` will be better --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17310: [SPARK-18579][SQL] Use ignoreLeadingWhiteSpace and ignor...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17310 This is an API change which we generally don't backport to a patch release. Can you elaborate on why you need it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18059: [SPARK-20834][SQL]TypeCoercion:loss of precision when wi...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18059 I have tested in MYSQL, the result datatype seems to be `decimalType`: mysql> desc test; ++-+--+-+-+---+ | Field | Type| Null | Key | Default | Extra | ++-+--+-+-+---+ | inttest | int(11) | YES | | NULL| | | doubletest | double | YES | | NULL| | | floattest | float | YES | | NULL| | ++-+--+-+-+---+ mysql> select * from test; +---++---+ | inttest | doubletest | floattest | +---++---+ | 123456789 | 3.1415 |3.1416 | +---++---+ mysql> select if(true, inttest, floattest) from test; +--+ | if(true, inttest, floattest) | +--+ |123456789 | +--+ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18073: [SPARK-20848][SQL] Shutdown the pool after readin...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/18073#discussion_r118158352
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormatSuite.scala
---
@@ -26,6 +26,22 @@ import org.apache.spark.sql.test.SharedSQLContext
class ParquetFileFormatSuite extends QueryTest with ParquetTest with
SharedSQLContext {
+ test("Number of threads doesn't grow extremely after parquet file
reading") {
+withTempDir { dir =>
+ val file = dir.toString + "/file"
+ spark.range(1).toDF("a").coalesce(1).write.parquet(file)
+ spark.read.parquet(file)
+ val numThreadBefore = Thread.activeCount
+ (1 to 100).map { _ =>
+spark.read.parquet(file)
+ }
+ val numThreadAfter = Thread.activeCount
+ // Hard to test a correct thread number,
+ // but it shouldn't increase more than a reasonable number.
+ assert(numThreadAfter - numThreadBefore < 20)
--- End diff --
It reduces to few after waiting an enough time. The number returned by
Thread.activeCount is only an estimate. So we may not expect this to be 0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18077: [SPARK-20861][ML][PYTHON] Delegate looping over p...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18077 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18077: [SPARK-20861][ML][PYTHON] Delegate looping over paramMap...
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18077 Merging with master and branch-2.2 which means this will get into 2.2.0 Thanks for the quick fix! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12646: [SPARK-14878][SQL] Trim characters string function suppo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12646 **[Test build #77288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77288/testReport)** for PR 12646 at commit [`1a5747b`](https://github.com/apache/spark/commit/1a5747b32885a818330eee6125a300bdb1dc8346). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77274/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18076 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18076 **[Test build #77274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77274/testReport)** for PR 18076 at commit [`72cee6e`](https://github.com/apache/spark/commit/72cee6eee9fd771e0aebc3cfb6fc6e906b67b351). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18064: [SPARK-20213][SQL] Fix DataFrameWriter operations in SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18064 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77276/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18064: [SPARK-20213][SQL] Fix DataFrameWriter operations in SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18064 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18064: [SPARK-20213][SQL] Fix DataFrameWriter operations in SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18064 **[Test build #77276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77276/testReport)** for PR 18064 at commit [`984bab7`](https://github.com/apache/spark/commit/984bab7bb2297e39088f909c41344a8fc8e06936). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18077: [SPARK-20861] Delegate looping over paramMaps to estimat...
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18077 Other than the tags, this LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18077: [SPARK-20861] Delegate looping over paramMaps to estimat...
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/18077 @MrBago Can you please add the tags "[ML][PYTHON]" to the title? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11974 **[Test build #77287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77287/testReport)** for PR 11974 at commit [`46d9c7b`](https://github.com/apache/spark/commit/46d9c7bbbfdd663a19212c2ddc7431ccd6293022). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77286/testReport)** for PR 17770 at commit [`c0bee01`](https://github.com/apache/spark/commit/c0bee014eaa268014f5e156498d8cc7d90533ac7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/16989 In current change: 1) remove the partial written file when failing 2) remove all shuffle files when `cleanup()`(this is registered as a task completion callback) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18082: [SPARK-20665][SQL][FOLLOW-UP]Move test case to SQLQueryT...
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18082 Done. Should i delete the unit test case from `MathFunctionsSuite.scala`? @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17770: [SPARK-20392][SQL] Set barrier to prevent re-ente...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/17770#discussion_r118155011
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -912,3 +913,17 @@ case class Deduplicate(
override def output: Seq[Attribute] = child.output
}
+
+/** A logical plan for setting a barrier of analysis */
+case class AnalysisBarrier(child: LogicalPlan) extends LeafNode {
+ override def output: Seq[Attribute] = child.output
+ override def analyzed: Boolean = true
+ override def isStreaming: Boolean = child.isStreaming
+ override lazy val canonicalized: LogicalPlan = child.canonicalized
+
+ override def find(f: LogicalPlan => Boolean): Option[LogicalPlan] = if
(f(this)) {
--- End diff --
OK. Agreed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16989 **[Test build #77285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77285/testReport)** for PR 16989 at commit [`222680c`](https://github.com/apache/spark/commit/222680c9d311f2d3fe7265fbf6e834e73cf4c05d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16820: [SPARK-19471] AggregationIterator does not initialize th...
Github user yangw1234 commented on the issue: https://github.com/apache/spark/pull/16820 @gatorsmile Sorry, I totally forget this pr. I will try to address the comment this week (need a little time to re-familiarize the context). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77284 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77284/testReport)** for PR 17770 at commit [`f3e4208`](https://github.com/apache/spark/commit/f3e4208eb23bee5cfc0e8a33134d58fac5526dbb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18078: [SPARK-10643] Make spark-submit download remote files to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18078 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77271/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18078: [SPARK-10643] Make spark-submit download remote files to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18078 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18078: [SPARK-10643] Make spark-submit download remote files to...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18078 **[Test build #77271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77271/testReport)** for PR 18078 at commit [`6e86290`](https://github.com/apache/spark/commit/6e86290a149269b681f3aab3b32f2d829f9d41a1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77283/testReport)** for PR 17770 at commit [`5c53f0f`](https://github.com/apache/spark/commit/5c53f0f4f7488cb69ca8107f2c95e69ea333f11f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18069: [SPARK-20850][SQL]Improve division and multiplica...
Github user heary-cao closed the pull request at: https://github.com/apache/spark/pull/18069 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18069: [SPARK-20850][SQL]Improve division and multiplication mi...
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/18069 @hvanhovell thank you for review it. This is an extreme case. must be with `Longtype` type and `Decimal` type mixing test. However, when test case: spark-sql> select (1234567890123456789012 / 12345678901234567890120) 0.1 spark-sql> select 0.1 * 12345678901234567890120; 1234567890123456789012 the result is ok. For this PR, It is not the best treatment method. close it for a moment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18082: [SPARK-20665][SQL][FOLLOW-UP]Move test case to SQ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18082#discussion_r118153699 --- Diff: sql/core/src/test/resources/sql-tests/inputs/mathFunctions.sql --- @@ -0,0 +1,37 @@ + +-- round and bround +select round(3.1415925, 7); +select round(3.1415925, 8); +select round(3.1415925, 9); +select round(3.1415925, 10); +select round(3.1415925, 100); +select round(3.1415925, 6); +select round(null, 8); + +select bround(3.1415925, 7); +select bround(3.1415925, 8); +select bround(3.1415925, 9); +select bround(3.1415925, 10); +select bround(3.1415925, 100); +select bround(3.1415925, 6); +select bround(null, 8); + +-- math functions --- End diff -- To `-- cot` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18082: [SPARK-20665][SQL][FOLLOW-UP]Move test case to SQ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18082#discussion_r118153601 --- Diff: sql/core/src/test/resources/sql-tests/inputs/mathFunctions.sql --- @@ -0,0 +1,37 @@ + +-- round and bround +select round(3.1415925, 7); +select round(3.1415925, 8); +select round(3.1415925, 9); +select round(3.1415925, 10); +select round(3.1415925, 100); +select round(3.1415925, 6); +select round(null, 8); + +select bround(3.1415925, 7); +select bround(3.1415925, 8); +select bround(3.1415925, 9); +select bround(3.1415925, 10); +select bround(3.1415925, 100); +select bround(3.1415925, 6); +select bround(null, 8); + +-- math functions +select cot(1); +select cot(null); +select cot(0); +select cot(-1); + +-- ceil and ceiling +select ceiling(0); +select ceiling(1); +select ceil(1234567890123456); +select ceil(12345678901234567); +select ceiling(1234567890123456); +select ceiling(12345678901234567); + +-- floor +select floor(0); +select floor(1); +select floor(1234567890123456); +select floor(12345678901234567); --- End diff -- add a new line here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16989 yea let's remove 1) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18059: [SPARK-20834][SQL]TypeCoercion:loss of precision when wi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18059 can you check other databases and see if this is common? Personally I think the current behavior is reasonable, always using double seems not a good choice. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to replace ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16225 **[Test build #77282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77282/testReport)** for PR 16225 at commit [`b5424d9`](https://github.com/apache/spark/commit/b5424d9fea56d2e0fb57ebc27d3d35054da6d22b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17987: [SPARK-19707][SPARK-18922][TESTS][SQL][CORE] Fix test fa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17987 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17987: [SPARK-19707][SPARK-18922][TESTS][SQL][CORE] Fix test fa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17987 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77269/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/16989 @cloud-fan In current change, the shuffle files are deleted twice: 1). After the `ManagedBuffer.release` 2). In the `cleanup()`, the `cleanup()` is already registered as a task completion callback. You mean that it's better to remove 1) ? In my understanding, there's no need to create another task completion callback. We just delete the files in `cleanup()` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17987: [SPARK-19707][SPARK-18922][TESTS][SQL][CORE] Fix test fa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17987 **[Test build #77269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77269/testReport)** for PR 17987 at commit [`1af7324`](https://github.com/apache/spark/commit/1af732442de7d002daf38a13aff72db335509ff2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18073: [SPARK-20848][SQL] Shutdown the pool after readin...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18073#discussion_r118152566
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormatSuite.scala
---
@@ -26,6 +26,22 @@ import org.apache.spark.sql.test.SharedSQLContext
class ParquetFileFormatSuite extends QueryTest with ParquetTest with
SharedSQLContext {
+ test("Number of threads doesn't grow extremely after parquet file
reading") {
+withTempDir { dir =>
+ val file = dir.toString + "/file"
+ spark.range(1).toDF("a").coalesce(1).write.parquet(file)
+ spark.read.parquet(file)
+ val numThreadBefore = Thread.activeCount
+ (1 to 100).map { _ =>
+spark.read.parquet(file)
+ }
+ val numThreadAfter = Thread.activeCount
+ // Hard to test a correct thread number,
+ // but it shouldn't increase more than a reasonable number.
+ assert(numThreadAfter - numThreadBefore < 20)
--- End diff --
after waiting for enough time, can we expect this to be 0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18076 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18078: [SPARK-10643] Make spark-submit download remote files to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18078 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77268/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18078: [SPARK-10643] Make spark-submit download remote files to...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18078 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18079: [SPARK-20841][SQL] Support column aliases for catalog ta...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18079 **[Test build #77280 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77280/testReport)** for PR 18079 at commit [`902d2a3`](https://github.com/apache/spark/commit/902d2a35740f3c3bc0c97aae56c65d9d25df3a15). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17770 **[Test build #77281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77281/testReport)** for PR 17770 at commit [`f5f0524`](https://github.com/apache/spark/commit/f5f0524ef47cf0f1b2f3a32f28be0251129feabe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18078: [SPARK-10643] Make spark-submit download remote files to...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18078 **[Test build #77268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77268/testReport)** for PR 18078 at commit [`4cdeeed`](https://github.com/apache/spark/commit/4cdeeed3f04f0ec62c6909e43ffe2d9824d863f7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16989: [SPARK-19659] Fetch big blocks to disk when shuffle-read...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16989 LGTM, only one comment: https://github.com/apache/spark/pull/16989#discussion_r118151720 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16989: [SPARK-19659] Fetch big blocks to disk when shuff...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16989#discussion_r118151720
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java
---
@@ -126,4 +150,50 @@ private void failRemainingBlocks(String[]
failedBlockIds, Throwable e) {
}
}
}
+
+ private class DownloadCallback implements StreamCallback {
+
+private WritableByteChannel channel = null;
+private File targetFile = null;
+private int chunkIndex;
+
+public DownloadCallback(File targetFile, int chunkIndex) throws
IOException {
+ this.targetFile = targetFile;
+ this.channel = Channels.newChannel(new FileOutputStream(targetFile));
+ this.chunkIndex = chunkIndex;
+}
+
+@Override
+public void onData(String streamId, ByteBuffer buf) throws IOException
{
+ channel.write(buf);
+}
+
+@Override
+public void onComplete(String streamId) throws IOException {
+ channel.close();
+ ManagedBuffer buffer = new FileSegmentManagedBuffer(
+transportConf, targetFile, 0, targetFile.length()) {
+@Override
+public ManagedBuffer release() {
--- End diff --
+1, I think it's simpler to clean these temp files with task completion
callback instead of `ManagedBuffer.release`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18073: [SPARK-20848][SQL] Shutdown the pool after reading parqu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18073 **[Test build #77279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77279/testReport)** for PR 18073 at commit [`14e09aa`](https://github.com/apache/spark/commit/14e09aafb8ee0cc9f5f1e95bc41b19eeab6fc7d4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18076: [SPARK-18406][CORE] Race between end-of-task and complet...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18076 **[Test build #77278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77278/testReport)** for PR 18076 at commit [`bc66ec5`](https://github.com/apache/spark/commit/bc66ec52adf0f741a0c533b28ca64e3fef9e848e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
