[GitHub] spark pull request: [SPARK-2228] change hard coded EVENT_QUEUE_CAP...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1257#issuecomment-47448132 @pwendell why 1 then? Like why not 1000, or 1M? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47448784 Preview of the change:  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1261 [SPARK-2320] Reduce exception/code block font size in web ui You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark ui-pre-size Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1261.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1261 commit 7ab1a69246a1af74601da01b22e2f6f5a6fd62f6 Author: Reynold Xin r...@apache.org Date: 2014-06-29T08:33:04Z [SPARK-2320] Reduce exception/code block font size in web ui --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1262 [SPARK-2299] Consolidate various stageIdTo* hash maps in JobProgressListener This should reduce memory usage for the web ui as well as slightly increase its speed in draining the UI event queue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark ui-consolidate-hashtables Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1262.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1262 commit 63256f592a8ebd2a34e8127979e38fe304c0abf0 Author: Reynold Xin r...@apache.org Date: 2014-06-29T08:26:38Z [SPARK-2320] Reduce pre block font size. commit f959bb8a3551995febd8271b7ba72f4fa2969ea4 Author: Reynold Xin r...@apache.org Date: 2014-06-29T08:31:30Z [SPARK-2299] Consolidate various stageIdTo* hash maps in JobProgressListener to speed it up. commit 7a7b6c41cd8a4ba272bda0a4a2b1402456c09c43 Author: Reynold Xin r...@apache.org Date: 2014-06-29T08:32:07Z Revert css change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-47448815 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47448816 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-47448820 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47448819 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1262#discussion_r14328143
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala ---
@@ -67,28 +61,28 @@ private[ui] class StagePage(parent: JobProgressTab)
extends WebUIPage(stage) {
ul class=unstyled
li
strongTotal task time across all tasks: /strong
-
{UIUtils.formatDuration(listener.stageIdToTime.getOrElse(stageId, 0L) +
activeTime)}
+ {UIUtils.formatDuration(stageData.executorRunTime)}
--- End diff --
Note that I dropped activeTime here (time taken for currently active tasks)
because I'm not sure if the extra data structure required to track this is
worth the benefit (I don't know if anybody really looks at this ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-47449638 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47449641 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16241/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47449639 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2299] Consolidate various stageIdTo* ha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1262#issuecomment-47449640 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16240/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2233] make-distribution script should l...
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/1216#issuecomment-47449894 lgtm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-47453099 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-47453105 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-47453145 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Fix JIRA-983 and support exteranl sort for sor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/931#issuecomment-47453146 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16242/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user willb commented on a diff in the pull request:
https://github.com/apache/spark/pull/143#discussion_r14329726
--- Diff: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ---
@@ -153,6 +153,18 @@ private[spark] object ClosureCleaner extends Logging {
field.setAccessible(true)
field.set(func, outer)
}
+
+if (checkSerializable) {
+ ensureSerializable(func)
+}
+ }
+
+ private def ensureSerializable(func: AnyRef) {
+try {
+ SparkEnv.get.closureSerializer.newInstance().serialize(func)
+} catch {
+ case ex: Exception = throw new SparkException(Task not
serializable: + ex.toString)
--- End diff --
I agree that it is better to wrap the underlying exception but was
following the style of this error in DAGScheduler. I'll make the change and
update that as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47458467 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47458471 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47459527 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16243/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47459526 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user willb commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47459912 Sorry, I missed FailureSuite. I have a fix but ran out of battery before I could push. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47460336 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47460329 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47462979 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47462980 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16244/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
GitHub user marmbrus opened a pull request: https://github.com/apache/spark/pull/1263 [SPARK-2059][SQL] Add analysis checks An initial version of analysis checks. Long term we are going to want something more complete, but this at least prevents us from making it all the way execution with obvious problems. For example, doing a sort on a misspelled attribute is actually enough to kill the DAG scheduler. You can merge this pull request into a Git repository by running: $ git pull https://github.com/marmbrus/spark analysisChecks Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1263.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1263 commit 448c088641af409ffeff59d1c3a326631a6bd599 Author: Michael Armbrust mich...@databricks.com Date: 2014-06-29T18:32:12Z Add analysis checks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1263#issuecomment-47467036 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1263#issuecomment-47467023 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/143#discussion_r14331097
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -533,7 +533,7 @@ abstract class DStream[T: ClassTag] (
* on each RDD of 'this' DStream.
*/
def transform[U: ClassTag](transformFunc: RDD[T] = RDD[U]): DStream[U]
= {
-transform((r: RDD[T], t: Time) =
context.sparkContext.clean(transformFunc(r)))
+transform((r: RDD[T], t: Time) =
context.sparkContext.clean(transformFunc(r), false))
--- End diff --
and for all other instances where that is set to false too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/143#discussion_r14331096
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -533,7 +533,7 @@ abstract class DStream[T: ClassTag] (
* on each RDD of 'this' DStream.
*/
def transform[U: ClassTag](transformFunc: RDD[T] = RDD[U]): DStream[U]
= {
-transform((r: RDD[T], t: Time) =
context.sparkContext.clean(transformFunc(r)))
+transform((r: RDD[T], t: Time) =
context.sparkContext.clean(transformFunc(r), false))
--- End diff --
@willb I think you missed this. Make sure you add comment above this line
to explain the reason why we do not check serializable ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1263#discussion_r14331223 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -20,6 +20,7 @@ package org.apache.spark.sql.catalyst.analysis import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.plans.logical._ import org.apache.spark.sql.catalyst.rules._ +import org.apache.spark.sql.catalyst.errors.TreeNodeException --- End diff -- nitpick sort the import --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1263#discussion_r14331240
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -54,10 +55,22 @@ class Analyzer(catalog: Catalog, registry:
FunctionRegistry, caseSensitive: Bool
ResolveFunctions ::
GlobalAggregates ::
typeCoercionRules :_*),
+Batch(Check Analysis, Once,
+ CheckResolution),
Batch(AnalysisOperators, fixedPoint,
EliminateAnalysisOperators)
)
+ object CheckResolution extends Rule[LogicalPlan] {
+def apply(plan: LogicalPlan): LogicalPlan = {
+ plan.transform {
+case p if p.expressions.filterNot(_.resolved).nonEmpty =
--- End diff --
```if p.expressions.exists(!_.resolved)```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1263#issuecomment-47473283 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16245/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1263#issuecomment-47473282 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1263#issuecomment-47475165 Maybe also add a test case? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47478084 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47478075 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47480391 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47480392 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16246/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47480476 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47480528 LGTM. We could add a `+show stack trace` button later so it doesn't cram all the other columns to the left. (similar to `+show details` on the index page) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47480644 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47480635 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user YanjieGao commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47480949 Hi all, I have modify the files and update the code as your suggestiones.The build has triggered but it didn't merged . I don't know what's the main cause of didn't merge .Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47482971 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16247/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47482970 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1261 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2320] Reduce exception/code block font ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1261#issuecomment-47483890 That's a good idea. Let's do that next. Merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler could ...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1264 [SPARK-2322] Exception in resultHandler could crash DAGScheduler and shutdown SparkContext. This should go into 1.0.1. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-2322 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1264.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1264 commit 5d8d920aebc1530e1c28dce3d9911c6afdab0e6d Author: Reynold Xin r...@apache.org Date: 2014-06-29T23:52:46Z [SPARK-2322] Exception in resultHandler could crash DAGScheduler and shutdown SparkContext. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler could ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47484375 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler could ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47484383 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47484638 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47484629 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47484945 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47484947 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16248/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47484978 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1265 [SPARK-2059][SQL] Add analysis checks This replaces #1263 with a test case. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark sql-analysis-error Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1265.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1265 commit 448c088641af409ffeff59d1c3a326631a6bd599 Author: Michael Armbrust mich...@databricks.com Date: 2014-06-29T18:32:12Z Add analysis checks commit 7371e1babf9e8be61a92cc0fc9171567cdc0f46d Author: Reynold Xin r...@apache.org Date: 2014-06-29T23:58:51Z Merge pull request #1263 from marmbrus/analysisChecks [SPARK-2059][SQL] Add analysis checks commit a639e01952381aeb22467798c52536d1fffd5518 Author: Reynold Xin r...@apache.org Date: 2014-06-30T00:07:56Z Added a test case for unresolved attribute analysis. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1265#issuecomment-47485148 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user marmbrus closed the pull request at: https://github.com/apache/spark/pull/1263 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1265#issuecomment-47485201 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1265#issuecomment-47485210 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/1017#issuecomment-47485261 Ping @rxin, are you concerned about too much logging? I can lower the level if you think this adds too much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1017#issuecomment-47486375 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1017#issuecomment-47486383 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/1245#discussion_r14333421
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -96,15 +98,15 @@ class HashPartitioner(partitions: Int) extends
Partitioner {
* the value of `partitions`.
*/
class RangePartitioner[K : Ordering : ClassTag, V](
-partitions: Int,
+var partitions: Int,
--- End diff --
Hi Reynold, thanks for your comments, will this field `partitions` be used
in executor side, I think this field can be transient according to my
knowledge, am I miss something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47489009 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16251/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1265#issuecomment-47489004 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1265#issuecomment-47489006 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16250/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1017#issuecomment-47489007 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16252/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47489002 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1017#issuecomment-47489005 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47489008 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16249/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47489003 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1264#discussion_r14333837
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -838,7 +839,16 @@ class DAGScheduler(
cleanupStateForJobAndIndependentStages(job,
Some(stage))
listenerBus.post(SparkListenerJobEnd(job.jobId,
JobSucceeded))
}
- job.listener.taskSucceeded(rt.outputId, event.result)
+
+ // taskSucceeded runs some user code that might throw an
exception. Make sure
+ // we are resilient against that.
+ try {
+job.listener.taskSucceeded(rt.outputId, event.result)
--- End diff --
Could we wrap a wider area with this try-catch? For instance, we've also
had problems where Accumulators.add throws an exception, and we similarly don't
want the entirety of Spark to crash. I think any unhandled exception in the
handleTaskCompletion method deserves a task failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1946] Submit tasks after (configured ra...
Github user li-zhihui commented on the pull request: https://github.com/apache/spark/pull/900#issuecomment-47490304 @tgravescs @kayousterhout It will lead to a logic deadlock in yarn-cluster mode, if waitBackendReady is in TaskSchedulerImpl.start. How about move it (waitBackendReady) to postStartHook() ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47491606 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47492104 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47492111 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47492633 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47492785 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47492790 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47493566 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47493571 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user colorant commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47493840 @rxin Moved getBlockLocation method from shuffleBlockManager to HashShuffleBlockMananger to make the interface more general. Does current interface looks reasonable for you? Also still a few shuffle related code could be moved further from block manager to some specific shuffle manager related classes' implementation ( e.g. blockManager.getMultiple). But since they are not tightly related to this shuffleBlockManager generalization works and I am not quite sure whether the other shufflemanager implementation will reuse them or not, so just leave it as it is, and could be done in future PR I guess. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/962 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47495017 I've merged this into master (the Jenkins build finished but got stuck behind another broken build to report success). @sryza I added a comment as you suggested; we can fix this to be more accurate using the file statistics thing you suggested in a later patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47495454 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47495446 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495499 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16256/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495497 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16253/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495494 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47495495 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2288] Hide ShuffleBlockManager behind S...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1241#issuecomment-47495493 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47495500 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16255/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47495496 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47495498 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16254/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1683] Track task read metrics.
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/962#issuecomment-47495559 oops you guys beat me to it! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
