Todd Lipcon has submitted this change and it was merged. Change subject: spark: send authentication credentials to tasks ......................................................................
spark: send authentication credentials to tasks This changes KuduContext to eagerly connect to the cluster upon first construction and fetch authentication credentials. The credentials are stored in a serializable field of the KuduContext so that they will be available to any executor tasks who need them Note that this changes the signature of the KuduRDD constructor to just take in a KuduContext instead of a list of masters and a KuduContext. I believe this class has always been meant to be constructed from a KuduContext, so the constructor signature change isn't incompatible. I changed the constructor to package-private to ensure that this is the only supported construction mechanism going forward. I tested this in a secure cluster and was able to run Spark jobs after the change. Adding automated testing of this beyond the included unit test would be a good idea, but I think doing it in a follow-on would be nice to unblock cluster testing in the meantime. Change-Id: I7be2c72eb801c3ee10e576ce8705c3b5997dffd9 Reviewed-on: http://gerrit.cloudera.org:8080/6081 Reviewed-by: Dan Burkert <[email protected]> Tested-by: Kudu Jenkins --- M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/KuduContextTest.scala 4 files changed, 60 insertions(+), 28 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/6081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I7be2c72eb801c3ee10e576ce8705c3b5997dffd9 Gerrit-PatchSet: 8 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Chris George <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]>
