Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27969371
--- Diff:
core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -40,98 +37,127 @@ import org.apache.spark.util.{ActorLogReceive, Utils
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27969584
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -17,33 +17,38 @@
package org.apache.spark.deploy
-import
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27969874
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -17,33 +17,38 @@
package org.apache.spark.deploy
-import
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27969603
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -21,48 +21,50 @@ import java.io.File
import java.io.IOException
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27969598
--- Diff:
core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -40,98 +37,127 @@ import org.apache.spark.util.{ActorLogReceive, Utils
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27968633
--- Diff:
core/src/main/scala/org/apache/spark/deploy/client/AppClient.scala ---
@@ -40,98 +37,127 @@ import org.apache.spark.util.{ActorLogReceive, Utils
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5392#discussion_r27970134
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -79,22 +84,36 @@ private class ClientActor(driverArgs: ClientArguments,
conf
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5392#issuecomment-90920884
I used the following codes to handle errors for the new codes in `Client`
but kept the old codes untouched.
```
logError(e.getMessage, e)
System.exit
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5434#issuecomment-91724400
Rebased on master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5414#discussion_r28048694
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/ui/StreamingJobProgressListenerSuite.scala
---
@@ -0,0 +1,140 @@
+/*
+ * Licensed
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5414#discussion_r28041716
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -135,9 +136,13 @@ class JobScheduler(val ssc
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5414#issuecomment-91172852
Added some tests to StreamingListenerSuite for this issue, and added
StreamingJobProgressListenerSuite for some basic tests of
StreamingJobProgressListener
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5414#discussion_r28048643
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala
---
@@ -88,6 +88,58 @@ class StreamingListenerSuite extends
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5414#issuecomment-91144827
Could you add a StreamingJobProgressListenerSuite ?
OK. I will add some basic tests for StreamingJobProgressListener
---
If your project is set up for it, you
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5414#discussion_r28047175
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala
---
@@ -135,9 +136,13 @@ class JobScheduler(val ssc
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5434#issuecomment-91246879
I added some tests to verify the title titles and column names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5414#discussion_r28048687
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingListenerSuite.scala
---
@@ -88,6 +88,58 @@ class StreamingListenerSuite extends
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/5434
[SPARK-6796][Streaming][WebUI] Add Active Batches and Completed Batches
lists to StreamingPage
This PR adds two lists, `Active Batches` and `Completed Batches`. Here is
the screenshot
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5414#issuecomment-91214133
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/5414#discussion_r28048619
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/ui/StreamingJobProgressListenerSuite.scala
---
@@ -0,0 +1,140 @@
+/*
+ * Licensed
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5434#issuecomment-91113024
cc @tdas and @JoshRosen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6159#issuecomment-103699139
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6227#issuecomment-103710448
@JoshRosen why not round up `size` to a power of 2? I think it's more
possible to reuse the objects.
---
If your project is set up for it, you can reply to this email
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6227#issuecomment-103715302
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6227#issuecomment-103711056
+1. WeakReferences is better for cache usage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6227#discussion_r30663882
--- Diff:
unsafe/src/main/java/org/apache/spark/unsafe/memory/ExecutorMemoryManager.java
---
@@ -44,15 +56,56 @@ public ExecutorMemoryManager
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6227#issuecomment-103712538
I see. LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6234#issuecomment-103598095
LGTM except the minor comment. But I think @srowen will handle it.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/5557#issuecomment-94291451
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6292#issuecomment-104045848
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6294#issuecomment-104036466
Do you need explicit locking? Wouldn't synchronized start, register, stop
etc be enough?
Using `this` will block the `registerReceiver` method when stopping
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/6294
Pr4467
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark pr4467
Alternatively you can review and apply these changes as the patch
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6294#issuecomment-104033650
@viirya could you take a look?
cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6292#discussion_r30750140
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala
---
@@ -132,6 +132,24 @@ class StreamingContextSuite extends
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6306#issuecomment-104085988
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6307#issuecomment-104085750
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6306#issuecomment-104084547
Cannot reproduce it... Just some guess
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6306#issuecomment-104082725
cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/6306
[SPARK-][Streaming] Fix the flaky test in
org.apache.spark.streaming.BasicOperationsSuite
Just added a guard to make sure a batch has completed before moving to the
next batch.
You can merge
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6307#issuecomment-104112034
@tdas as per javadoc:
https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#addShutdownHook(java.lang.Thread)
Shutdown hooks should also finish
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/6659
[SPARK-8112][Streaming]Fix the negative event count issue
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-8112
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6650#issuecomment-109172890
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6614#issuecomment-109766689
@dibbhatt could you resolve the conflicts again? Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6707#discussion_r31934707
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala
---
@@ -64,11 +68,17 @@ private[streaming] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6380#discussion_r31975251
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -527,6 +529,54 @@ class StreamingContext private[streaming
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6380#discussion_r31975006
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -527,6 +529,54 @@ class StreamingContext private[streaming
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6380#discussion_r31975035
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -527,6 +529,54 @@ class StreamingContext private[streaming
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6380#issuecomment-110187199
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6711#issuecomment-110187377
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6672#issuecomment-109330852
cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/6672
[SPARK-4072][Core][WIP]Display Streaming blocks in Streaming UI
Replace #6634
This PR adds `SparkListenerBlockUpdated` to SparkListener so that it can
monitor all block update infos
Github user zsxwing closed the pull request at:
https://github.com/apache/spark/pull/6634
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r31860010
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockStatusListener.scala ---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6672#issuecomment-109586172
New screenshot
 for `char
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6672#issuecomment-114159220
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/3858#issuecomment-114140106
@industrial-sloth I think most of work to add a new type of DStream
includes:
1. Convert parameters from Python types to Java types
2. Convert DStream[T
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/6955
[SPARK-8564][Streaming]Add the Python API for Kinesis
This PR adds the Python API for Kinesis, including a Python example and a
simple unit test.
You can merge this pull request into a Git
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6845#issuecomment-113785985
Do you mean above the axis?
No. Still overlap. But `#batches` is on the top.
---
If your project is set up for it, you can reply to this email and have your
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6845#issuecomment-113783361
Looks a lot of space on the right. Could you just move `#batches` above the
blue bar?
---
If your project is set up for it, you can reply to this email and have your
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/6830#issuecomment-113786084
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/4229#discussion_r32920969
--- Diff: external/mqtt-assembly/pom.xml ---
@@ -0,0 +1,101 @@
+?xml version=1.0 encoding=UTF-8?
+!--
+ ~ Licensed to the Apache Software
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/4229#discussion_r32922157
--- Diff: python/pyspark/streaming/mqtt.py ---
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/4229#discussion_r32920954
--- Diff: python/pyspark/streaming/mqtt.py ---
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/4229#discussion_r32919596
--- Diff: external/mqtt-assembly/pom.xml ---
@@ -0,0 +1,101 @@
+?xml version=1.0 encoding=UTF-8?
+!--
+ ~ Licensed to the Apache Software
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/4229#discussion_r32919438
--- Diff: examples/src/main/python/streaming/mqtt_wordcount.py ---
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r32914064
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -98,6 +99,9 @@ case class SparkListenerExecutorAdded(time: Long
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/7016
[SPARK-8630][Streaming] Prevent from checkpointing QueueInputDStream
This PR throws an exception in `QueueInputDStream.writeObject` so that it
can fail the application when calling
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/7017#issuecomment-115290274
/cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/7017
[SPARK-8634][Streaming][Tests]Fix flaky test StreamingListenerSuite
receiver info reporting
As per the unit test log in
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35754
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/7016#issuecomment-115293536
/cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/7017#issuecomment-115313671
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6955#discussion_r33151330
--- Diff: python/run-tests ---
@@ -126,7 +126,20 @@ function run_streaming_tests() {
KAFKA_ASSEMBLY_JAR=$f
done
-export
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33149671
--- Diff: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ---
@@ -848,6 +908,60 @@ private[spark] object JsonProtocol {
new ExecutorInfo
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/6672#discussion_r33147254
--- Diff: core/src/main/scala/org/apache/spark/ui/storage/StoragePage.scala
---
@@ -30,11 +30,32 @@ private[ui] class StoragePage(parent: StorageTab
801 - 900 of 6049 matches
Mail list logo
