GitHub user joseph-torres opened a pull request:
https://github.com/apache/spark/pull/19984
[SPARK-22789] Map-only continuous processing execution
## What changes were proposed in this pull request?
Basic continuous execution, supporting map/flatMap/filter, with commits and
advancement through RPC.
## How was this patch tested?
new unit-ish tests (exercising execution end to end)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joseph-torres/spark continuous-impl
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19984.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19984
----
commit d6bea84447d910e79d5926972d87a80bc5dc2e2e
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-07T22:08:28Z
Refactor StreamExecution into a parent class so continuous processing can
extend it
commit df6b8861173d1e7853952c8f3ffe504975efe204
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T19:31:28Z
address fmt
commit 6f0ce6b1cf1abf602c2b02ce6d31f46f8fa71b7c
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-13T00:09:48Z
slight changes
commit 2b360ab49bcab3c73ea85ce62202e40e950931ef
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-13T00:10:34Z
rm spurious space
commit 1b19f1ce4444f17e7324997649ad8c5f97887912
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-13T00:35:30Z
fix compile
commit 96eba13be9764e63f3d1375d7b51dbfd0675aa98
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T20:48:20Z
harness
commit 2d5efadb9e7662363e3e4a3c66e0f5f73e4935ef
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T21:18:25Z
awaitEpoch impl
commit 578bbb7eb0725b795ac65d1beda436515f4f4eba
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T21:46:09Z
move local[10] to only continuous suite
commit 9051eff6c88838ac61ab45763ed84d593e2d4837
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T21:49:55Z
repeatedly restart
commit 60fa4477591cc264b9ea253f64065d762ce3f96f
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:02:52Z
fix some simple TODOs
commit ea8e76ec75752d134433730ee1a007cce1fdcfe8
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:11:18Z
use runId instead of queryId for endpoint name
commit d0f3cc7701d9eb3e7df571561e751f03c0537f3a
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:19:03Z
more simple todos
commit ba9dbaa1be2f54827a42f3177669082e7d1f99e2
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:27:12Z
remove old state
commit 2cd005f4685e492ae78d6b9c579c80c2370d2f14
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:35:51Z
remove clean shutdown workaround in StreamTest
commit a7fa31fb5375074d888bd0a94e317ad3f1692e5a
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:50:09Z
update ContinuousExecution docs
commit f687432a58acf7337885edfc01adc94188d174d8
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-11T22:59:14Z
add comments to EpochCoordinator
commit 987b011ee78292c3379559910ebe101daf4f9450
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T00:02:54Z
change offset semantic to end of previous epoch
commit 5494fc50ef99b3e584c287b03eaa32b30657a5ce
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T00:18:40Z
document EpochCoordinator
commit d6ef404b85fa6977b5f38a853dca11de5189b3f9
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T02:06:44Z
simplify epoch handling
commit 647bd2745c1c0842002d4f71b61aa34beb0f8b29
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T19:17:58Z
stress tests
commit 053a9f349a4829433a495aa5989f1ca1c8a3256e
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T20:17:22Z
add minBatchesToRetain
commit 7072d21444388fe167fa7e3475b3e95ec9923d5e
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T20:43:33Z
add confs
commit 4083a8f5c6b6ef298726234d54f23a90e971e77e
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-12T21:10:33Z
latency suite not meaningful here
commit 41d391f2027a4e8b3730d15cea7b7fbcdcec27de
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-13T00:04:07Z
more stress::q
commit 402cfa3b10dfb0f37ce8d94336be3b3c01fe9f90
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-13T18:55:23Z
use temp dir
commit e4a1bc19db9ea0233879d270e725ed58d95a34ad
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-14T19:37:36Z
fix against rebase
commit 8887b3c92afe8bb1659f600785af5d97f085f2bb
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-14T21:32:16Z
fix ser/deser
commit 60bf0e33f20134af296d85b5c52729c4063ef2e1
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-14T22:41:21Z
fix rebase compile
commit 749bddc6303321118407ff5c2664528f7160ff65
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-14T23:35:46Z
stop using ProcessingTime in executor
commit 5a15ed5b30cc70e70d199d82e617c67426562ba2
Author: Jose Torres <j...@databricks.com>
Date: 2017-12-14T23:54:52Z
add tests for supported ops
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
