Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/9004 )
Change subject: KUDU-2254: Detect and warn about misusage of KuduContext ...................................................................... Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/9004/2/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala: http://gerrit.cloudera.org:8080/#/c/9004/2/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala@93 PS2, Line 93: private[kudu] val taskContext = TaskContext.get > does this field now end up serialized? Given this field isn't marked @trans It is hard to be certain that TaskContext is always null in the driver but not the executor based on my knowledge. At least from what I have experiments it is this case (also according to the spark documentation). However, I had a discussion with Dan offline, that this patch may not be necessary. Since creation of KuduContext needs SparkContext, while SparkContext cannot be serializable to executors. That means the users need to create a new SparkContext in the executor to be able to create KuduContext, and SparkContext should be a driver specific entity. -- To view, visit http://gerrit.cloudera.org:8080/9004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I65a7cd11a14fa8a668079b0d1fcf6ed3a34fb652 Gerrit-Change-Number: 9004 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao <hao....@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: Hao Hao <hao....@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Comment-Date: Fri, 12 Jan 2018 00:49:36 +0000 Gerrit-HasComments: Yes
