[GitHub] NiharS commented on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration
NiharS commented on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration URL: https://github.com/apache/spark/pull/22114#issuecomment-45581 @attilapiros Thanks for your work on #23540 and for pinging me of the fix! This PR was an old change and I appreciate you thinking of it. I believe that #23540 completely fixes the bug seen in the Jenkins tests here. Back when this was failing, I was able to reproduce the error with a 50% success rate by running BroadcastSuite and DistributedSuite back-to-back, on my local machine. I pulled @attilapiros's change and reran the suites on loop, and there were no failures. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-455843908 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins removed a comment on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-455843909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7239/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-455843909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7239/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
AmplabJenkins commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-455843908 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
SparkQA commented on issue #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics URL: https://github.com/apache/spark/pull/22874#issuecomment-455843858 **[Test build #101440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101440/testReport)** for PR 22874 at commit [`26d6250`](https://github.com/apache/spark/commit/26d6250818beb80d76143b0bd184c4f193badb75). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] LantaoJin commented on a change in pull request #22874: [SPARK-25865][CORE] Add GC information to ExecutorMetrics
LantaoJin commented on a change in pull request #22874: [SPARK-25865][CORE] Add
GC information to ExecutorMetrics
URL: https://github.com/apache/spark/pull/22874#discussion_r249267528
##
File path:
core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
##
@@ -99,6 +102,58 @@ case object ProcessTreeMetrics extends ExecutorMetricType {
}
}
+case object GarbageCollectionMetrics extends ExecutorMetricType with Logging {
+ override val names = Seq(
+"MinorGCCount",
+"MinorGCTime",
+"MajorGCCount",
+"MajorGCTime"
+ )
+
+ /* We builtin some common GC collectors which categorized as young
generation and old */
+ private[spark] val YOUNG_GENERATION_BUILTIN_GARBAGE_COLLECTORS = Seq(
+"Copy",
+"PS Scavenge",
+"ParNew",
+"G1 Young Generation"
+ )
+
+ private[spark] val OLD_GENERATION_BUILTIN_GARBAGE_COLLECTORS = Seq(
+"MarkSweepCompact",
+"PS MarkSweep",
+"ConcurrentMarkSweep",
+"G1 Old Generation"
+ )
+
+ private lazy val youngGenerationGarbageCollector: Seq[String] = {
+
SparkEnv.get.conf.get(config.EVENT_LOG_GC_METRICS_YOUNG_GENERATION_GARBAGE_COLLECTORS)
+ }
+
+ private lazy val oldGenerationGarbageCollector: Seq[String] = {
+
SparkEnv.get.conf.get(config.EVENT_LOG_GC_METRICS_OLD_GENERATION_GARBAGE_COLLECTORS)
+ }
+
+ override private[spark] def getMetricValues(memoryManager: MemoryManager):
Array[Long] = {
+val gcMetrics = new Array[Long](names.length) // minorCount, minorTime,
majorCount, majorTime
+if (SparkEnv.get.conf.get(config.EVENT_LOG_GC_METRICS)) {
Review comment:
Yes. I just keep the same structure like ProcfsMetrics. I will remove it
since it still returns 0 when its off.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23595: [MINOR][DOC] Small fix on document
AmplabJenkins removed a comment on issue #23595: [MINOR][DOC] Small fix on document URL: https://github.com/apache/spark/pull/23595#issuecomment-455842850 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23595: [MINOR][DOC] Small fix on document
AmplabJenkins commented on issue #23595: [MINOR][DOC] Small fix on document URL: https://github.com/apache/spark/pull/23595#issuecomment-455842910 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23595: [MINOR][DOC] Small fix on document
AmplabJenkins removed a comment on issue #23595: [MINOR][DOC] Small fix on document URL: https://github.com/apache/spark/pull/23595#issuecomment-455842833 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23595: [MINOR][DOC] Small fix on document
AmplabJenkins commented on issue #23595: [MINOR][DOC] Small fix on document URL: https://github.com/apache/spark/pull/23595#issuecomment-455842850 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23595: [MINOR][DOC] Small fix on document
AmplabJenkins commented on issue #23595: [MINOR][DOC] Small fix on document URL: https://github.com/apache/spark/pull/23595#issuecomment-455842833 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] lys0716 opened a new pull request #23595: [MINOR][DOC] Small fix on document
lys0716 opened a new pull request #23595: [MINOR][DOC] Small fix on document URL: https://github.com/apache/spark/pull/23595 ## What changes were proposed in this pull request? Small fix on the wrong document. ## How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455840348 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101439/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455840347 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455837622 **[Test build #101439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101439/testReport)** for PR 23589 at commit [`d89fd0d`](https://github.com/apache/spark/commit/d89fd0dbd5345ff5f42918241dbccbbddb2f7589). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455840348 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101439/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455840347 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455840314 **[Test build #101439 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101439/testReport)** for PR 23589 at commit [`d89fd0d`](https://github.com/apache/spark/commit/d89fd0dbd5345ff5f42918241dbccbbddb2f7589). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455837670 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7238/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455837668 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455837670 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7238/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455837622 **[Test build #101439 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101439/testReport)** for PR 23589 at commit [`d89fd0d`](https://github.com/apache/spark/commit/d89fd0dbd5345ff5f42918241dbccbbddb2f7589). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455837668 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23588: Diversity
HyukjinKwon commented on issue #23588: Diversity URL: https://github.com/apache/spark/pull/23588#issuecomment-455837547 See all the minor PRs https://github.com/apache/spark/pulls?q=is%3Apr+minor+is%3Aclosed Why do you think the guide doesn't work for this PR specifically? What makes you think this PR complies the doc? Shall we fix the doc then? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] shahidki31 commented on a change in pull request #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
shahidki31 commented on a change in pull request #23589:
[SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation
metrics
URL: https://github.com/apache/spark/pull/23589#discussion_r249264859
##
File path: docs/mllib-evaluation-metrics.md
##
@@ -462,10 +462,10 @@ $$rel_D(r) = \begin{cases}1 & \text{if $r \in D$}, \\ 0
& \text{otherwise}.\end{
Normalized Discounted Cumulative Gain
$NDCG(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{IDCG(D_i,
k)}\sum_{j=0}^{n-1}
- \frac{rel_{D_i}(R_i(j))}{\text{ln}(j+2)}} \\
+ \frac{rel_{D_i}(R_i(j))}{\text{log}_2(j+2)}} \\
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] shahidki31 commented on a change in pull request #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
shahidki31 commented on a change in pull request #23589:
[SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation
metrics
URL: https://github.com/apache/spark/pull/23589#discussion_r249264856
##
File path: docs/mllib-evaluation-metrics.md
##
@@ -439,21 +439,21 @@ $$rel_D(r) = \begin{cases}1 & \text{if $r \in D$}, \\ 0
& \text{otherwise}.\end{
Precision at k
-$p(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{k}
\sum_{j=0}^{\text{min}(\left|D\right|, k) - 1} rel_{D_i}(R_i(j))}$
+$p(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{k}
\sum_{j=0}^{\text{min}(\left|R_i\right|, k) - 1} rel_{D_i}(R_i(j))}$
Review comment:
Thanks. updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] LuciferYang commented on issue #22149: [SPARK-25158][SQL]Executor accidentally exit because ScriptTransformationWriterThread throws TaskKilledException.
LuciferYang commented on issue #22149: [SPARK-25158][SQL]Executor accidentally exit because ScriptTransformationWriterThread throws TaskKilledException. URL: https://github.com/apache/spark/pull/22149#issuecomment-455836922 @cloud-fan should we fix this problems? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23588: Diversity
HyukjinKwon commented on issue #23588: Diversity URL: https://github.com/apache/spark/pull/23588#issuecomment-455835835 Yea so do you think "Diversity" implies what the PR proposes? Take a look for other PRs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kmarekspartz commented on issue #23588: Diversity
kmarekspartz commented on issue #23588: Diversity URL: https://github.com/apache/spark/pull/23588#issuecomment-455833561 > A bug may be reported by creating a JIRA but without creating a pull request > However, trivial changes, where the what should change is virtually the same as the how it should change do not require a JIRA. Example: Fix typos in Foo scaladoc This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue commented on issue #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide
Deegue commented on issue #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide URL: https://github.com/apache/spark/pull/23593#issuecomment-455833261 > Could you attach the screenshot of newly added documentation part? Added a screenshot. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23588: Diversity
HyukjinKwon commented on issue #23588: Diversity URL: https://github.com/apache/spark/pull/23588#issuecomment-455831923 This > The PR title should be of the form [SPARK-][COMPONENT] Title, where SPARK- is the relevant JIRA number, COMPONENT is one of the PR categories shown at spark-prs.appspot.com and Title may be the JIRA’s title or a more specific title describing the PR itself. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23549: [SPARK-26616][MLlib] Expose document frequency in IDFModel
srowen commented on issue #23549: [SPARK-26616][MLlib] Expose document frequency in IDFModel URL: https://github.com/apache/spark/pull/23549#issuecomment-455828442 Looks like a legit failure: ``` ** File "/home/jenkins/workspace/NewSparkPullRequestBuilder/python/pyspark/ml/feature.py", line 971, in __main__.IDF Failed example: model.numDocs Expected: 3L Got: 3 ** 1 of 19 in __main__.IDF ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455825691 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455825696 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101437/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455825696 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101437/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455825691 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
SparkQA removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455808183 **[Test build #101437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101437/testReport)** for PR 23590 at commit [`da1aaac`](https://github.com/apache/spark/commit/da1aaac7cb1b0687ec2e19c5049b96e216f9dce2). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
SparkQA commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455825574 **[Test build #101437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101437/testReport)** for PR 23590 at commit [`da1aaac`](https://github.com/apache/spark/commit/da1aaac7cb1b0687ec2e19c5049b96e216f9dce2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455824334 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101438/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455824333 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455824333 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455824334 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101438/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
SparkQA removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455808189 **[Test build #101438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101438/testReport)** for PR 23416 at commit [`1a8d84b`](https://github.com/apache/spark/commit/1a8d84bce5a85d28b49debc8606471d09a44). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
SparkQA commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455824286 **[Test build #101438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101438/testReport)** for PR 23416 at commit [`1a8d84b`](https://github.com/apache/spark/commit/1a8d84bce5a85d28b49debc8606471d09a44). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455824088 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455824092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101436/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455824092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101436/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455824088 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
SparkQA removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808182 **[Test build #101436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101436/testReport)** for PR 23594 at commit [`c27610c`](https://github.com/apache/spark/commit/c27610cb0bcb6934e59252409a9a7641555eba67). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
SparkQA commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455823986 **[Test build #101436 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101436/testReport)** for PR 23594 at commit [`c27610c`](https://github.com/apache/spark/commit/c27610cb0bcb6934e59252409a9a7641555eba67). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808427 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] attilapiros edited a comment on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration
attilapiros edited a comment on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration URL: https://github.com/apache/spark/pull/22114#issuecomment-455820031 @srowen we should merge this one too. #23540 was started to fix the OOM experienced right here (which was not reproducible locally). So there I just checked whether fixing those leaks are really enough for solving the OOM coming with this change (but I did not want to take away @NiharS's change, just helping with the OOM), this is why I reverted this line after fixing those leaks proved to be sufficient. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] attilapiros commented on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration
attilapiros commented on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration URL: https://github.com/apache/spark/pull/22114#issuecomment-455820031 @srowen we should merge this one too. #23540 was started to fix the OOM experienced right here (which was not reproducible locally). So there I just checked whether fixing those leaks are really enough for solving the OOM coming with this change (but I did not want to take away Nihar's change, just helping with the OOM). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] kmarekspartz commented on issue #23588: Diversity
kmarekspartz commented on issue #23588: Diversity URL: https://github.com/apache/spark/pull/23588#issuecomment-455819321 To clarify, which portion of those guidelines is this not compliant with? There’s no JIRA, but the guidelines are clear that no JIRA is required for contributions like these. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on a change in pull request #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
srowen commented on a change in pull request #23589: [SPARK-26351][mllib]Update
doc and minor correction in the mllib evaluation metrics
URL: https://github.com/apache/spark/pull/23589#discussion_r249256437
##
File path: docs/mllib-evaluation-metrics.md
##
@@ -439,21 +439,21 @@ $$rel_D(r) = \begin{cases}1 & \text{if $r \in D$}, \\ 0
& \text{otherwise}.\end{
Precision at k
-$p(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{k}
\sum_{j=0}^{\text{min}(\left|D\right|, k) - 1} rel_{D_i}(R_i(j))}$
+$p(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{k}
\sum_{j=0}^{\text{min}(\left|R_i\right|, k) - 1} rel_{D_i}(R_i(j))}$
Review comment:
Maybe; it could be the same for all users, or not. The documentation above
this suggests there are equal numbers of recommended and relevant docs for each
user (Q and N) but at least, it will almost never be true that |D_i| is the
same for all users. Q could well be a constant.
But the implementation doesn't assume that and it's not necessary to, so I
might even just remove the references to Q and N, or label them "Q_i" and "N_i"
if you really want to be complete.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on a change in pull request #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
srowen commented on a change in pull request #23589: [SPARK-26351][mllib]Update
doc and minor correction in the mllib evaluation metrics
URL: https://github.com/apache/spark/pull/23589#discussion_r249256455
##
File path: docs/mllib-evaluation-metrics.md
##
@@ -462,10 +462,10 @@ $$rel_D(r) = \begin{cases}1 & \text{if $r \in D$}, \\ 0
& \text{otherwise}.\end{
Normalized Discounted Cumulative Gain
$NDCG(k)=\frac{1}{M} \sum_{i=0}^{M-1} {\frac{1}{IDCG(D_i,
k)}\sum_{j=0}^{n-1}
- \frac{rel_{D_i}(R_i(j))}{\text{ln}(j+2)}} \\
+ \frac{rel_{D_i}(R_i(j))}{\text{log}_2(j+2)}} \\
Review comment:
This isn't how it's computed, though the base won't matter. To avoid
confusion, what about just writing "log" with no base?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide
srowen commented on issue #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide URL: https://github.com/apache/spark/pull/23593#issuecomment-455812028 I don't think a screenshot helps? it's just text, not a UI change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide
dongjoon-hyun commented on issue #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide URL: https://github.com/apache/spark/pull/23593#issuecomment-455811807 Could you attach the screenshot of newly added documentation part? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808669 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808670 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7237/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808670 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7237/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins removed a comment on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808669 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
AmplabJenkins commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808427 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455808243 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455808243 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins removed a comment on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455808245 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7235/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455808231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7236/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455808231 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7236/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
AmplabJenkins commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455808245 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7235/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455808229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
AmplabJenkins removed a comment on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455808229 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
SparkQA commented on issue #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594#issuecomment-455808182 **[Test build #101436 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101436/testReport)** for PR 23594 at commit [`c27610c`](https://github.com/apache/spark/commit/c27610cb0bcb6934e59252409a9a7641555eba67). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
SparkQA commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455808189 **[Test build #101438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101438/testReport)** for PR 23416 at commit [`1a8d84b`](https://github.com/apache/spark/commit/1a8d84bce5a85d28b49debc8606471d09a44). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
SparkQA commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455808183 **[Test build #101437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101437/testReport)** for PR 23590 at commit [`da1aaac`](https://github.com/apache/spark/commit/da1aaac7cb1b0687ec2e19c5049b96e216f9dce2). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk commented on a change in pull request #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
MaxGekk commented on a change in pull request #23594: [SPARK-26657][SQL] Use
Proleptic Gregorian calendar in DayWeek and in WeekOfYear
URL: https://github.com/apache/spark/pull/23594#discussion_r249254823
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##
@@ -231,8 +231,8 @@ class DateExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkEvaluation(WeekOfYear(Cast(Literal(date), DateType, gmtId)), 15)
checkEvaluation(WeekOfYear(Cast(Literal(ts), DateType, gmtId)), 45)
checkEvaluation(WeekOfYear(Cast(Literal("2011-05-06"), DateType, gmtId)),
18)
-checkEvaluation(WeekOfYear(Literal(new Date(toMillis("1582-10-15
13:10:15", 40)
-checkEvaluation(WeekOfYear(Literal(new Date(toMillis("1582-10-04
13:10:15", 39)
+checkEvaluation(WeekOfYear(Cast(Literal("1582-10-15 13:10:15"), DateType,
gmtId)), 41)
Review comment:
See https://www.timeanddate.com/calendar/?year=1582=22=1 , it
shows weeks number too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] MaxGekk opened a new pull request #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear
MaxGekk opened a new pull request #23594: [SPARK-26657][SQL] Use Proleptic Gregorian calendar in DayWeek and in WeekOfYear URL: https://github.com/apache/spark/pull/23594 ## What changes were proposed in this pull request? The expressions `DayWeek`, `DayOfWeek`, `WeekDay` and `WeekOfYear` are changed to use Proleptic Gregorian calendar instead of the hybrid one (Julian+Gregorian). This was achieved by using Java 8 API for date/timestamp manipulation, in particular the `LocalDate` class. Week of year calculation is performed according to ISO-8601. The first week of a week-based-year is the first Monday-based week of the standard ISO year that has at least 4 days in the new year (see https://docs.oracle.com/javase/8/docs/api/java/time/temporal/IsoFields.html). ## How was this patch tested? The changes were tested by `DateExpressionsSuite` and `DateFunctionsSuite`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories.
dongjoon-hyun commented on issue #23416: [SPARK-26463][CORE] Use ConfigEntry for hardcoded configs for scheduler categories. URL: https://github.com/apache/spark/pull/23416#issuecomment-455807713 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever
dongjoon-hyun commented on issue #23590: [SPARK-26665][Core]Fix a bug that BlockTransferService.fetchBlockSync may hang forever URL: https://github.com/apache/spark/pull/23590#issuecomment-455807646 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23582: [SPARK-26661][SQL] Show actual class name of the writing command in CTAS explain
dongjoon-hyun commented on a change in pull request #23582: [SPARK-26661][SQL]
Show actual class name of the writing command in CTAS explain
URL: https://github.com/apache/spark/pull/23582#discussion_r249254525
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
##
@@ -83,10 +84,14 @@ trait CreateHiveTableAsSelectBase extends
DataWritingCommand {
tableDesc: CatalogTable,
tableExists: Boolean): DataWritingCommand
+ // A subclass should override this with the Class of the concrete type
expected to be
+ // returned from `getWritingCommand`.
+ def writingCommandClass: Class[_]
Review comment:
In general, we had better minimize the surface of API in terms of
`parameters` and `return values`. As we know, `Class[_]` has much more
information than `String`(name).
And, Yes. Since I also received the same feedbacks during working on APIs in
the Spark community, I have a strong preference to avoid excessive information
exposure or unnecessary exchanges. Do you have a future plan to use that
function for another purposes?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23582: [SPARK-26661][SQL] Show actual class name of the writing command in CTAS explain
dongjoon-hyun commented on a change in pull request #23582: [SPARK-26661][SQL]
Show actual class name of the writing command in CTAS explain
URL: https://github.com/apache/spark/pull/23582#discussion_r249254525
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
##
@@ -83,10 +84,14 @@ trait CreateHiveTableAsSelectBase extends
DataWritingCommand {
tableDesc: CatalogTable,
tableExists: Boolean): DataWritingCommand
+ // A subclass should override this with the Class of the concrete type
expected to be
+ // returned from `getWritingCommand`.
+ def writingCommandClass: Class[_]
Review comment:
In general, we had better minimize the surface of API in terms of
`parameters` and `return values`. As we know, `Class[_]` has much more
information than `String`(name).
And, Yes. Since I also received the same feedbacks during working on APIs in
the Spark community, I have a strong preference to avoid excessive information
exposure. Do you have a future plan to use that function for another purposes?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] dongjoon-hyun commented on a change in pull request #23582: [SPARK-26661][SQL] Show actual class name of the writing command in CTAS explain
dongjoon-hyun commented on a change in pull request #23582: [SPARK-26661][SQL]
Show actual class name of the writing command in CTAS explain
URL: https://github.com/apache/spark/pull/23582#discussion_r249254525
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
##
@@ -83,10 +84,14 @@ trait CreateHiveTableAsSelectBase extends
DataWritingCommand {
tableDesc: CatalogTable,
tableExists: Boolean): DataWritingCommand
+ // A subclass should override this with the Class of the concrete type
expected to be
+ // returned from `getWritingCommand`.
+ def writingCommandClass: Class[_]
Review comment:
In general, we had better minimize the surface of API in terms of
`parameters` and `return values`. As we know, `Class[_]` has much more
information than `String`(name).
And, Yes. Since I also received the same feedbacks during working on APIs in
the Spark community, I have a strong preference to avoid excessive information
exposure. Is there any future plan to use that function for the other purpose?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration
srowen commented on issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memory for headers without regard for configuration URL: https://github.com/apache/spark/pull/22114#issuecomment-455806841 @attilapiros I think you made and then backed out this change in your https://github.com/apache/spark/pull/23540 pull request. Is it something we should merge now after your change, or did you find we shouldn't make this change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23549: [SPARK-26616][MLlib] Expose document frequency in IDFModel
SparkQA removed a comment on issue #23549: [SPARK-26616][MLlib] Expose document frequency in IDFModel URL: https://github.com/apache/spark/pull/23549#issuecomment-455787359 **[Test build #4522 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4522/testReport)** for PR 23549 at commit [`253a0d6`](https://github.com/apache/spark/commit/253a0d6735c19fe280926d9cba59ba834327178c). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23549: [SPARK-26616][MLlib] Expose document frequency in IDFModel
SparkQA commented on issue #23549: [SPARK-26616][MLlib] Expose document frequency in IDFModel URL: https://github.com/apache/spark/pull/23549#issuecomment-455806719 **[Test build #4522 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4522/testReport)** for PR 23549 at commit [`253a0d6`](https://github.com/apache/spark/commit/253a0d6735c19fe280926d9cba59ba834327178c). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] srowen commented on issue #23580: [SPARK-26660]Add warning logs for large taskBinary size
srowen commented on issue #23580: [SPARK-26660]Add warning logs for large taskBinary size URL: https://github.com/apache/spark/pull/23580#issuecomment-455806418 I don't think we need another config here. Would a user really modify this setting? vs fixing the problem or ignoring the warning? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] Deegue commented on a change in pull request #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide
Deegue commented on a change in pull request #23593: [SPARK-26667][DOC]Add `Scanning Input Table` to Performance Tuning Guide URL: https://github.com/apache/spark/pull/23593#discussion_r249252345 ## File path: docs/sql-performance-tuning.md ## @@ -40,6 +40,41 @@ Configuration of in-memory caching can be done using the `setConf` method on `Sp +## Scanning Input Table + +Spark SQL can increase the speed while scanning tables via tuning hadoop configurations. Review comment: Proofread. I added it in SQL Tuning because we can set it before executing a SQL. I'd appreciate it if you could give more suggestions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455800087 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101435/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455800085 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455800085 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455800087 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101435/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455794661 **[Test build #101435 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101435/testReport)** for PR 23589 at commit [`2003599`](https://github.com/apache/spark/commit/2003599acd27fd129047964d71a61ada41d4b633). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455800026 **[Test build #101435 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101435/testReport)** for PR 23589 at commit [`2003599`](https://github.com/apache/spark/commit/2003599acd27fd129047964d71a61ada41d4b633). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455794230 **[Test build #101434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101434/testReport)** for PR 23589 at commit [`542d687`](https://github.com/apache/spark/commit/542d687252b19b93810121124308e64dd946e19b). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455799705 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101434/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455799705 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/101434/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455799701 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins removed a comment on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455794761 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/7233/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
AmplabJenkins commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455799701 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics
SparkQA commented on issue #23589: [SPARK-26351][mllib]Update doc and minor correction in the mllib evaluation metrics URL: https://github.com/apache/spark/pull/23589#issuecomment-455799609 **[Test build #101434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/101434/testReport)** for PR 23589 at commit [`542d687`](https://github.com/apache/spark/commit/542d687252b19b93810121124308e64dd946e19b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
