[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919967 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -56,7 +56,7 @@ private[spark] class MemoryStore(blockManager: BlockManager

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919955 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -499,16 +499,16 @@ private[spark] class BlockManagerInfo

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919943 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -225,7 +225,7 @@ class BlockManagerMasterActor(val isLocal: Boolean

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919971 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -176,7 +176,7 @@ private[spark] class MemoryStore(blockManager: BlockManager

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919953 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -499,16 +499,16 @@ private[spark] class BlockManagerInfo

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919947 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -479,18 +479,18 @@ private[spark] class BlockManagerInfo

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919966 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -158,7 +158,7 @@ private[spark] class DiskBlockManager(shuffleBlockManager

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919949 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -479,18 +479,18 @@ private[spark] class BlockManagerInfo

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919946 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -339,7 +339,7 @@ class BlockManagerMasterActor(val isLocal: Boolean

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919951 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -479,18 +479,18 @@ private[spark] class BlockManagerInfo

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919959 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -499,16 +499,16 @@ private[spark] class BlockManagerInfo

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1776#discussion_r15919977 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -325,7 +325,7 @@ private[spark] class MemoryStore(blockManager: BlockManager

[GitHub] spark pull request: [WIP] [SPARK-2655] Change logging level from I...

2014-08-06 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1776#issuecomment-51435412 I went through all the changes and made comments on ones I think that should be kept INFO. A lot of them are too chatty, but some of them are useful to keep based on my

[GitHub] spark pull request: SPARK-2787: Make sort-based shuffle write file...

2014-08-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1799#issuecomment-51552394 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: SPARK-2787: Make sort-based shuffle write file...

2014-08-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1799#issuecomment-51552406 I'm merging this in master & branch-1.1 (since sort-based is disabled by default) --- If your project is set up for it, you can reply to this email and have y

[GitHub] spark pull request: [SPARK-2912] [Spark QA] [WIP] Include commit h...

2014-08-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1816#issuecomment-51565282 BTW not sure if you commented out the actual unit test, but nothing was run in Jenkins ... --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-2849 / 2914] Handle certain Spark confi...

2014-08-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-51565448 Given the complexity of the bash scripts, can we make sure we have some tests for this? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-2911]: provide rdd.parent[T](j) to obta...

2014-08-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1841#issuecomment-51565558 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2911]: provide rdd.parent[T](j) to obta...

2014-08-07 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1841#issuecomment-51568573 Merging in master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2911]: provide rdd.parent[T](j) to obta...

2014-08-07 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1841#discussion_r15979883 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -1233,6 +1233,11 @@ abstract class RDD[T: ClassTag

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1836#discussion_r16006499 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -228,8 +232,19 @@ private[broadcast] object TorrentBroadcast extends

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1836#discussion_r16006529 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -17,14 +17,14 @@ package org.apache.spark.broadcast

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1836#discussion_r16006565 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -228,8 +232,19 @@ private[broadcast] object TorrentBroadcast extends

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1836#discussion_r16006660 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -264,7 +279,19 @@ private[broadcast] object TorrentBroadcast extends

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1836#issuecomment-51636425 This looks good to me other than the minor style problems I pointed out. Thanks for doing it. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1836#discussion_r16007066 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -17,14 +17,14 @@ package org.apache.spark.broadcast

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1836#issuecomment-51644898 Can we also add a unit test for these two? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2897][SPARK-2920]TorrentBroadcast does ...

2014-08-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1836#issuecomment-51670108 Thanks. Merging this in master and branch-1.1 and branch-1.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-2911] apply parent[T](j) to clarify Uni...

2014-08-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1858#issuecomment-51670209 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [Web UI]Make decision order of Worker's WebUI ...

2014-08-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1838#issuecomment-51676244 Thanks. I'm merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark pull request: [SPARK-2911] apply parent[T](j) to clarify Uni...

2014-08-08 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1858#issuecomment-51676330 Thanks. Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2936] Migrate Netty network module from...

2014-08-08 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1865 [SPARK-2936] Migrate Netty network module from Java to Scala The Netty network module was originally written when Scala 2.9.x had a bug that prevents a pure Scala implementation, and a subset of the

[GitHub] spark pull request: Updated Spark SQL README to include the hive-t...

2014-08-09 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1867 Updated Spark SQL README to include the hive-thriftserver module You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark sql-readme

[GitHub] spark pull request: [SPARK-2936] Migrate Netty network module from...

2014-08-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1865#issuecomment-51694967 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2952] Enable logging actor messages at ...

2014-08-09 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1870 [SPARK-2952] Enable logging actor messages at DEBUG level Example messages: ``` 14/08/09 21:37:01 DEBUG BlockManagerMasterActor: [actor] received message RegisterBlockManager(BlockManagerId(0

[GitHub] spark pull request: Turn UpdateBlockInfo into case class.

2014-08-09 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1872 Turn UpdateBlockInfo into case class. This helps us log UpdateBlockInfo properly once #1870 is merged. You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] spark pull request: Updated Spark SQL README to include the hive-t...

2014-08-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1867#issuecomment-51706431 I'm merging this in master and branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark pull request: Updated Spark SQL README to include the hive-t...

2014-08-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1867#issuecomment-51706535 I couldn't merge it on my laptop. @pwendell can you try? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitH

[GitHub] spark pull request: [SPARK-2952] Enable logging actor messages at ...

2014-08-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1870#issuecomment-51707100 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2952] Enable logging actor messages at ...

2014-08-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1870#issuecomment-51707098 @andrewor14 is the SparkSubmit test flaky? I've seen it fail a few times already. ``` [info] - launch simple application with spark-submit *** FAILED ***

[GitHub] spark pull request: Turn UpdateBlockInfo into case class.

2014-08-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1872#issuecomment-51707226 I'm going to merge this one since the test failure is independent of this. --- If your project is set up for it, you can reply to this email and have your reply appe

[GitHub] spark pull request: [SPARK-2953] Allow using short names for io co...

2014-08-09 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1873 [SPARK-2953] Allow using short names for io compression codecs Instead of requiring "org.apache.spark.io.LZ4CompressionCodec", it is easier for users if Spark just accepts "lz4", &

[GitHub] spark pull request: Turn UpdateBlockInfo into case class.

2014-08-10 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1872#issuecomment-51724118 It is using some custom serialization to reduce serialization overhead. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: Remove extra semicolon in Task.scala

2014-08-10 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1876#issuecomment-51724140 Thanks. I've merged this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project doe

[GitHub] spark pull request: [SPARK-2936] Migrate Netty network module from...

2014-08-10 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1865#discussion_r1601 --- Diff: core/src/main/scala/org/apache/spark/network/netty/FileClient.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2936] Migrate Netty network module from...

2014-08-10 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1865#discussion_r16034768 --- Diff: core/src/main/scala/org/apache/spark/network/netty/FileClient.scala --- @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2936] Migrate Netty network module from...

2014-08-10 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1865#discussion_r16034776 --- Diff: core/src/main/scala/org/apache/spark/network/netty/FileServer.scala --- @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: Use transferTo when copy merge files in Extern...

2014-08-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1884#discussion_r16040902 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -744,13 +744,21 @@ private[spark] class ExternalSorter[K, V, C

[GitHub] spark pull request: Use transferTo when copy merge files in Extern...

2014-08-11 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1884#discussion_r16040916 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -744,13 +744,21 @@ private[spark] class ExternalSorter[K, V, C

[GitHub] spark pull request: [SPARK-2952] Enable logging actor messages at ...

2014-08-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1870#issuecomment-51849326 Thanks. Merging in master & branch-1.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2952] Enable logging actor messages at ...

2014-08-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1870#issuecomment-51850434 I thought about that in this PR actually. Since in log4j you can configure what you want, it seems pretty easy to just enable debug logging for a specific package. --- If

[GitHub] spark pull request: [SPARK-2952] Enable logging actor messages at ...

2014-08-11 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1870#issuecomment-51857379 Yea I'm fine turning that into trace. Even better, if we can configure what level to log those on, that'd be great. Do you want to submit a pull request? -

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1907 [WIP] [SPARK-2468] Netty based block server / client module This is a rough draft, but it can't be much worse than the old Netty module since the old one didn't really work :) Com

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16132782 --- Diff: core/src/main/scala/org/apache/spark/network/netty/server.scala --- @@ -0,0 +1,290 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51957233 That's part of the TODO :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16134921 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -262,77 +271,49 @@ object BlockFetcherIterator { readMetrics

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16135674 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client.scala --- @@ -0,0 +1,276 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-51970648 I'm not sure what akka.io would bring here. The main benefit seems to be more native Scala APIs. In this case I am not sure if we care since this is a low level m

[GitHub] spark pull request: [SPARK-2991] Implement RDD lazy transforms for...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1909#issuecomment-51971006 @erikerlandson perhaps also create an umbrella ticket and make all the related tickets a sub-task for the umbrella one? This way it is a lot easier to track them. Cheers

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16139683 --- Diff: core/src/main/scala/org/apache/spark/network/netty/ReferenceCountedBuffer.scala --- @@ -17,15 +17,25 @@ package

[GitHub] spark pull request: [SPARK-2953] Allow using short names for io co...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1873#issuecomment-52009496 I updated the documentation to put the short forms first before fully qualified class names. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: Use transferTo when copy merge files in Extern...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1884#issuecomment-52011123 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: [SPARK-2953] Allow using short names for io co...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1873#issuecomment-52012292 Thanks. Merging in master & branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013277 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-2468] Netty based block server / ...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52013468 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Use transferTo when copy merge files in Extern...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1884#issuecomment-52013620 It's ok the flume test is not related. I am going to merge this in master & branch-1.1. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-1777 (partial)] bugfix: make size of re...

2014-08-12 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1892#issuecomment-52015074 Thanks. Merging in master & branch-1.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1907#issuecomment-52091001 Thanks for the comment, @helena. We use Akka for control plane, but this pull request is for data plane and has different requirements (high throughput for one matter). And

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16202964 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client/ReferenceCountedBuffer.scala --- @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16203036 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyConfig.scala --- @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16204672 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client/LazyInitIterator.scala --- @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16211439 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyConfig.scala --- @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16212159 --- Diff: core/src/main/scala/org/apache/spark/network/netty/client/BlockFetchingClientHandler.scala --- @@ -0,0 +1,63 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-3020: Print completed indices rather tha...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1933#discussion_r16216114 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala --- @@ -38,6 +39,7 @@ private[jobs] object UIData { class StageUIData { var

[GitHub] spark pull request: SPARK-3020: Print completed indices rather tha...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1933#issuecomment-52130911 lgtm other than the todo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2912] [Spark QA] Include commit hash in...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1816#issuecomment-52131421 This is really cool. I will defer to @pwendell for merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [SPARK-2912] [Spark QA] Include commit hash in...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1816#issuecomment-52131437 In the future maybe we should rewrite this in Python so the community can maintain it (bash is great until it gets too complicated). --- If your project is set up for it

[GitHub] spark pull request: SPARK-3020: Print completed indices rather tha...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1933#issuecomment-52132539 Thanks. Merging in master & branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2468] Netty based block server / client...

2014-08-13 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1907#discussion_r16218268 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -262,77 +270,51 @@ object BlockFetcherIterator { readMetrics

[GitHub] spark pull request: [RFC] Disable local execution of Spark jobs by...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52146296 Now I think about it more. LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten the visibili...

2014-08-13 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1938 [SPARK-3027] TaskContext: tighten the visibility and provide Java friendly callback API You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin

[GitHub] spark pull request: [Docs] Add missing tags (minor)

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1936#issuecomment-52147906 Thanks. Merging in master & branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3029] Disable local execution of Spark ...

2014-08-13 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52149427 You need to update the test suites. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1827#discussion_r16223757 --- Diff: core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala --- @@ -207,6 +207,16 @@ class KryoSerializerSuite extends FunSuite with

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1827#discussion_r16223901 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -87,7 +87,8 @@ class KryoSerializer(conf: SparkConf

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten visibility a...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52151103 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1827#issuecomment-52151375 Because this way you get to log the class name :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1827#issuecomment-52151384 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1827#issuecomment-52151742 Yes, back to branch-1.0. (not 1.0.1 since we cannot change a released version, but we can push 1.0.3). --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/1827#discussion_r16225918 --- Diff: core/src/test/scala/org/apache/spark/serializer/KryoSerializerSuite.scala --- @@ -207,6 +207,16 @@ class KryoSerializerSuite extends FunSuite with

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1827#issuecomment-52156475 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten visibility a...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52156734 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3029] Disable local execution of Spark ...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1321#issuecomment-52157116 Merging this in master and branch-1.1. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Minor cleanup of metrics.Source

2014-08-14 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1943 Minor cleanup of metrics.Source - Added override. - Marked some variables as private. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin

[GitHub] spark pull request: SPARK-2893: Do not swallow Exceptions when run...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1827#issuecomment-52161963 I think the test failure is not caused by this. Those two tests have been flaky. I'm going to merge this in master & branch-1.1. Thanks! --- If your project is s

[GitHub] spark pull request: SPARK-3009: Reverted readObject method in Appl...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1947#issuecomment-52221832 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Minor cleanup of metrics.Source

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1943#issuecomment-52221880 Merging in master & branch-1.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten visibility a...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52221998 None of the previous test failures are caused by this PR ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-3027] TaskContext: tighten visibility a...

2014-08-14 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1938#issuecomment-52221970 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3045] [SPARK-3046] Make Serializer inte...

2014-08-14 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/1948 [SPARK-3045] [SPARK-3046] Make Serializer interface Java friendly & use executor's class loader as the default serializer class loader You can merge this pull request into a Git repo

<    9   10   11   12   13   14   15   16   17   18   >