Hello Dan Burkert, Kudu Jenkins, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8552
to look at the new patch set (#3).
Change subject: KUDU-1454 [part 1]: update propagated timestamp on the driver
......................................................................
KUDU-1454 [part 1]: update propagated timestamp on the driver
Currently, Spark uses multiple clients simultaneously, possibly on
different JVMs, for reads and writes. Each client may have the
propagated timestamp properly updated after each read/write. However,
the KuduClient on the driver does not have the correct propagated
timestamp. This is problematic if the user wants to use READ_AT_SNAPSHOT
ReadMode to get a consistent snapshot.
This patch uses Accumulator in Spark to properly update the propagated
timestamp on the driver through comparing the maximum timestamp on each
executor. It also updates back last propagated timestamp to the
executors for further writes. Test is added accordingly and ran in both
'local' and 'local-cluster' mode.
Change-Id: Id0a078ae8ebaa6a859be75c822879291192c5842
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
3 files changed, 98 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/52/8552/3
--
To view, visit http://gerrit.cloudera.org:8080/8552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id0a078ae8ebaa6a859be75c822879291192c5842
Gerrit-Change-Number: 8552
Gerrit-PatchSet: 3
Gerrit-Owner: Hao Hao <[email protected]>
Gerrit-Reviewer: Dan Burkert <[email protected]>
Gerrit-Reviewer: Hao Hao <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <[email protected]>
