Dan Burkert has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8552 )
Change subject: KUDU-1454 [part 1]: update propagated timestamp on the driver ...................................................................... KUDU-1454 [part 1]: update propagated timestamp on the driver Currently, Spark uses multiple clients simultaneously, possibly on different JVMs, for reads and writes. Each client may have the propagated timestamp properly updated after each read/write. However, the KuduClient on the driver does not have the correct propagated timestamp. This is problematic if the user wants to use READ_AT_SNAPSHOT ReadMode to get a consistent snapshot. This patch uses Accumulator in Spark to properly update the propagated timestamp on the driver through comparing the maximum timestamp on each executor. It also updates back last propagated timestamp to the executors for further writes. Test is added accordingly and ran in both 'local' and 'local-cluster' mode. Change-Id: Id0a078ae8ebaa6a859be75c822879291192c5842 Reviewed-on: http://gerrit.cloudera.org:8080/8552 Tested-by: Kudu Jenkins Reviewed-by: Dan Burkert <[email protected]> --- M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala 3 files changed, 98 insertions(+), 5 deletions(-) Approvals: Kudu Jenkins: Verified Dan Burkert: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/8552 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id0a078ae8ebaa6a859be75c822879291192c5842 Gerrit-Change-Number: 8552 Gerrit-PatchSet: 5 Gerrit-Owner: Hao Hao <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Hao Hao <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]>
