Hello, my question is the continuation of a problem I described here <http://stackoverflow.com/questions/40834482/delegation-token-can-be-issued-only-with-kerberos-or-web-authentication-when-r> .
I've done some investigation and found out that nameNode.getDelegationToken is called during constructing SparkContext even if delegation token is already presented in token list of current logged user in object of UserGroupInforation class. The problem doesn't occur when waiting time before constructing a new context is less than 10 seconds, because rpc connection to namenode just isn't resetting because of ipc.client.connection.maxidletime property. As a workaround of this problem I do login from keytab before every constructing of SparkContext, which basically just resets token list of current logged user (as well as whole user structure) and the problem seems to be gone. Still I'm not really sure that it is correct way to deal with SparkContext. Having found a reason of the problem, I've got 2 assumptions now: First - SparkContext was designed to be restarted during JVM run and behaviour above is just a bug. Second - it wasn't and I'm just using SparkContext in a wrong manner. Since I haven't found any related bug in Jira and any solution on the internet (as well as too many users facing this error) I tend to think that it is rather a not allowed usage of SparkContext. Is that correct? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Is-restarting-of-SparkContext-allowed-tp20240.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
