Hello Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6081
to look at the new patch set (#3).
Change subject: spark: send authentication credentials to tasks
......................................................................
spark: send authentication credentials to tasks
This changes KuduContext to eagerly connect to the cluster upon first
construction and fetch authentication credentials. The credentials are
stored in a serializable field of the KuduContext so that they will be
available to any executor tasks who need them
Note that this changes the signature of the KuduRDD constructor to just
take in a KuduContext instead of a list of masters and a KuduContext. I
believe this class has always been meant to be constructed from a
KuduContext, so the constructor signature change isn't incompatible. I
changed the constructor to package-private to ensure that this is the
only supported construction mechanism going forward.
I tested this in a secure cluster and was able to run Spark jobs after
the change.
Adding automated testing of this beyond the included unit test would be
a good idea, but I think doing it in a follow-on would be nice to
unblock cluster testing in the meantime.
Change-Id: I7be2c72eb801c3ee10e576ce8705c3b5997dffd9
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M
java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala
4 files changed, 55 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/81/6081/3
--
To view, visit http://gerrit.cloudera.org:8080/6081
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7be2c72eb801c3ee10e576ce8705c3b5997dffd9
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Chris George <chris.geo...@rms.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
