Dan Burkert has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9834 )
Change subject: [spark] KUDU-2371: Add KuduWriteOptions class and ignoreNull option ...................................................................... [spark] KUDU-2371: Add KuduWriteOptions class and ignoreNull option This patch adds the KuduWriteOptions class to allow configuration of writes to the Kudu table when writing with Spark. This allows extensibility via adding more fields in the future. The instance of this class is passed to functions (insert/delete/upsert/update) in KuduContext. KuduWriteOptions is also supported in DefaultSource APIs. Clients can set up write options in the parameters from SparkSQL. This patch also adds the ignoreNull write option so that users can upsert/update only non-Null columns and leave the rest of the columns unchanged. For example, this feature is useful when users use Spark streaming to process JSON and upsert to Kudu, because missing column values from JSON are set to NULL, resulting in some existing row values being upserted to Null, which is not desired. Change-Id: Ide908ea29f572849eca0ba850ee197c1b22a07c8 Reviewed-on: http://gerrit.cloudera.org:8080/9834 Reviewed-by: Dan Burkert <[email protected]> Tested-by: Kudu Jenkins --- M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala A java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduWriteOptions.scala M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/OperationType.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/TestContext.scala 6 files changed, 237 insertions(+), 26 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9834 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ide908ea29f572849eca0ba850ee197c1b22a07c8 Gerrit-Change-Number: 9834 Gerrit-PatchSet: 11 Gerrit-Owner: Fengling Wang <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Fengling Wang <[email protected]> Gerrit-Reviewer: Hao Hao <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Reviewer: Will Berkeley <[email protected]>
