Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17821#discussion_r114419550
--- Diff: core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala
---
@@ -80,8 +91,16 @@ private[deploy] object DeployMessages
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17346#discussion_r114395863
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
---
@@ -53,6 +53,29 @@ object FileStreamSink extends
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17346#discussion_r114396634
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
---
@@ -145,6 +147,41 @@ class FileStreamSinkSuite extends
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17346#discussion_r114396542
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
---
@@ -145,6 +147,41 @@ class FileStreamSinkSuite extends
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17346#discussion_r114397372
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
---
@@ -36,20 +37,27 @@ import
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17346#discussion_r114395114
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
---
@@ -53,6 +53,29 @@ object FileStreamSink extends
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17803
I don't know graphx or mllib. Others look good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17833
By the way, this PR is good to me since it does reduce a little memory
footprint. But we still cannot close
https://issues.apache.org/jira/browse/SPARK-20548 though.
---
If your project is set up
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17833
I think it only cleans `localProperties` in the current thread.
`localProperties` overrides `childValue` and always clones a new Properties for
child threads.
In addition, I think it
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17765
Merging to master and 2.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17765
LGTM!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17821
cc @sameeragarwal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17821
[SPARK-20529][Core]Allow worker and master work with a proxy server
## What changes were proposed in this pull request?
In the current codes, when worker connects to master, master will
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17803#discussion_r114080946
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/streaming/GroupStateTimeout.java
---
@@ -37,7 +37,9 @@
* `map/flatMapGroupsWithState` by
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
Thanks, @mridulm @aramesh117 Merging to master and 2.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
> did this PR was for compression to be enabled for spark streaming usecase.
Streaming checkpoint includes two parts:
- DStream graph and metadata
- RDD checkpoints
Ri
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17790
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17765#discussion_r113825493
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala ---
@@ -500,6 +502,69 @@ class StreamSuite extends StreamTest
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
yes. See
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala#L138
---
If your project is set up for it, you can reply to this email and
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
Streaming checkpoints are on HDFS but don't have an extension :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your pr
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
In addition, I agree that having an extension and separating the codecs are
good ideas. But they should be done in other PRs to not introduce multiple
features in a large PR.
---
If your project
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
> A question I had even with the earlier PR was - should we add the
extension to either the directory or the file indicating compression type ?
Shuffle and cache files don't
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17790
cc @yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17024
@aramesh117 I just opened #17789 to finish the rest work. All credits will
go to you when merging the new PR.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17789
cc @mridulm since you reviewed the initial PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17789
[SPARK-19525][CORE]Add RDD checkpoint compression support
## What changes were proposed in this pull request?
This PR adds RDD checkpoint compression support and add a new config
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17765#discussion_r113592921
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -252,6 +252,7 @@ class StreamExecution
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17765#discussion_r113557507
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -825,6 +833,11 @@ class StreamExecution
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17024
@aramesh117 do you have time to work on this PR recently? We need to merge
this PR ASAP in order to get it into 2.2.0.
---
If your project is set up for it, you can reply to this email and have
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17761
@mridulm this only affect codes calling `runUninterruptibly` which is not a
public API, so won't break any existing codes. The worst case of this PR is
some task needs to wait until network ti
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17763
[SPARK-13747][Core]Add ThreadUtils.awaitReady and disallow Await.ready
## What changes were proposed in this pull request?
Add `ThreadUtils.awaitReady` similar to `ThreadUtils.awaitResult
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17761#discussion_r113273103
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -84,7 +86,20 @@ private[spark] class Executor(
}
// Start
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17761
[SPARK-20461][Core][SS]Use UninterruptibleThread for Executor and fix the
potential hang in CachedKafkaConsumer
## What changes were proposed in this pull request?
This PR changes
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17540
The current `withNewExecutionId` issue is that it doesn't support nested
QueryExecution. I'm wondering if you can really fix this issue without
introducing regression, e.g., track
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17540#discussion_r113087928
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
---
@@ -161,50 +161,51 @@ object FileFormatWriter
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17540#discussion_r113084795
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
---
@@ -161,50 +161,51 @@ object FileFormatWriter
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113080567
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
---
@@ -95,8 +95,10 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113075234
--- Diff:
external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala
---
@@ -199,7 +199,7 @@ private[spark] class KafkaRDD[K, V
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113075321
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala
---
@@ -53,9 +54,27 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113075218
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala
---
@@ -371,7 +371,7 @@ private[kafka010] object
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113075117
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala
---
@@ -125,16 +125,15 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113074969
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -213,46 +203,6 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113074957
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -414,30 +364,9 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113074951
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala
---
@@ -213,46 +203,6 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113074883
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala
---
@@ -53,9 +54,27 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113074531
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
---
@@ -95,8 +95,10 @@ private[kafka010] class
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17752#discussion_r113074386
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala
---
@@ -95,8 +95,10 @@ private[kafka010] class
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17752
[SPARK-20452][SS][Kafka]Fix a potential ConcurrentModificationException for
batch Kafka DataFrame
## What changes were proposed in this pull request?
Cancel a batch Kafka query but one of
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17691
Thanks! Merging to master, 2.2 and 2.1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17691
[MINOR][SS] Fix a missing space in UnsupportedOperationChecker error message
## What changes were proposed in this pull request?
Also went through the same file to ensure other string
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17687
Thanks! Merging to master and 2.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17687
[SPARK-20397][SparkR][SS]Fix flaky test: test_streaming.R.Terminated by
error
## What changes were proposed in this pull request?
Checking a source parameter is asynchronous. When the
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17676
LGTM pending tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111830351
--- Diff:
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala ---
@@ -169,14 +177,23 @@ private[spark] object ReliableCheckpointRDD
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111830180
--- Diff:
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala ---
@@ -133,9 +136,14 @@ private[spark] object ReliableCheckpointRDD extends
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111829095
--- Diff:
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala ---
@@ -27,8 +27,11 @@ import org.apache.hadoop.fs.Path
import
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111833659
--- Diff: core/src/test/scala/org/apache/spark/CheckpointSuite.scala ---
@@ -238,6 +241,42 @@ trait RDDCheckpointTester { self: SparkFunSuite
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111830443
--- Diff: core/src/test/scala/org/apache/spark/CheckpointSuite.scala ---
@@ -251,10 +290,14 @@ class CheckpointSuite extends SparkFunSuite with
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111832087
--- Diff: core/src/test/scala/org/apache/spark/CheckpointSuite.scala ---
@@ -266,13 +309,44 @@ class CheckpointSuite extends SparkFunSuite with
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111829941
--- Diff:
core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala ---
@@ -133,9 +136,14 @@ private[spark] object ReliableCheckpointRDD extends
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111830401
--- Diff: core/src/test/scala/org/apache/spark/CheckpointSuite.scala ---
@@ -21,12 +21,15 @@ import java.io.File
import scala.reflect.ClassTag
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17024#discussion_r111836700
--- Diff: core/src/test/scala/org/apache/spark/CheckpointSuite.scala ---
@@ -266,13 +309,44 @@ class CheckpointSuite extends SparkFunSuite with
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17610
Merging to master and 2.1. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17610
I removed the lock and changed `stopping` to `AtomicBoolean` to ensure
idempotent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17624
nit: could you use a better title? You can add the test name into it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17610
> Isn't it not depending on this being locked in super class methods
invoked in the invocation subtree ?
I don't get it. But I think the stack trace shows why this dead-
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17610
@mridulm yeah, I was thinking to just change `stopping` to a AtomicBoolean
flag. However, it changes the semantics a little, e.g., the second `stop` will
return at once when the first `stop` is
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17463
Could you close this one, please? I submitted #17610 to fix the root issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17610
[SPARK-20131][Core]Use a separate lock for StandaloneSchedulerBackend.stop
## What changes were proposed in this pull request?
`o.a.s.streaming.StreamingContextSuite.SPARK-18560 Receiver
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17599
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17599
[SPARK-17564][Tests]Fix flaky
RequestTimeoutIntegrationSuite.furtherRequestsDelay
## What changes were proposed in this pull request?
This PR fixs the following failure
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17594
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17597
Thanks! Merging to master, 2.1 and 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17597
[SPARK-20285][Tests]Increase the pyspark streaming test timeout to 30
seconds
## What changes were proposed in this pull request?
Saw the following failure locally
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17594#discussion_r110735710
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -304,8 +304,8 @@ class StreamExecution
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17594
Write the log first to fix a race contion in tests
## What changes were proposed in this pull request?
This PR fixes the following failure:
```
sbt.ForkMain$ForkError
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17179
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17216#discussion_r106759443
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -256,6 +259,15 @@ class StreamExecution
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17216#discussion_r106757213
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -256,6 +259,15 @@ class StreamExecution
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17295#discussion_r106749569
--- Diff:
core/src/main/scala/org/apache/spark/security/CryptoStreamUtils.scala ---
@@ -63,12 +83,40 @@ private[spark] object CryptoStreamUtils extends
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106736186
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17216#discussion_r106709281
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
---
@@ -549,9 +581,15 @@ class StreamExecution
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17216#discussion_r106709791
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
---
@@ -29,12 +30,32 @@ class OffsetSeqLogSuite extends
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17327
LGTM. Merging to 2.1. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user zsxwing closed the pull request at:
https://github.com/apache/spark/pull/17284
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17323
cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17323
[SPARK-19986][Tests]Make pyspark.streaming.tests.CheckpointTests more stable
## What changes were proposed in this pull request?
Sometimes, CheckpointTests will hang on a busy machine
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17070
@lw-lin there are conflicts with 2.1. Could you submit a new PR for
branch-2.1?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17070
LGTM. Merging to master and 2.1. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17070#discussion_r106348166
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
---
@@ -195,6 +195,11 @@ class HDFSMetadataLog[T
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/17244
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055705
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -298,12 +368,14 @@ class KeyValueGroupedDataset[K, V] private
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106045620
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/KeyedStateImpl.scala
---
@@ -60,6 +82,45 @@ private[sql] class KeyedStateImpl[S
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106051316
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FlatMapGroupsWithStateExec.scala
---
@@ -0,0 +1,270 @@
+/*
+ * Licensed
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106054568
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
---
@@ -50,6 +50,8 @@ trait StateStore {
/** Get
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055666
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -361,18 +435,20 @@ class KeyValueGroupedDataset[K, V] private
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106045218
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/streaming/KeyedState.scala ---
@@ -92,27 +121,33 @@ import
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055599
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -284,6 +322,38 @@ class KeyValueGroupedDataset[K, V] private[sql
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106036389
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/streaming/KeyedStateTimeout.java
---
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r106055580
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql
1001 - 1100 of 6087 matches
Mail list logo
