Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/16788
Shipping tokens with tasks may have a big issue as discussed with @mridulm
before. Some Spark applications has long running out-band operations to
community with HDFS, like Spark Streaming's WAL, Structured Streaming's
checkpoint. What if token is expired and new token which shipped with task
hasn't fetched, the operations will be failed. So shipping with tasks is not
workable.
Also here only handles Hadoop FileSystem DTs, and only supports default FS,
this is not feature comparable to Spark on YARN.
And your changes will also effect standalone mode, is that your expectation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
