mridulm commented on PR #40064:
URL: https://github.com/apache/spark/pull/40064#issuecomment-1441190260
@Yikf Agree - we only specify two parts for the `JobID` - the `String
jtIdentifier` and `int id`.
We can persist those in the class - and make jobId a `transient lazy val`
which recreates it each time.
Something like this instead:
```
private[this] val jobIdParts: (String, Int) = {
val id = SparkHadoopWriterUtils.createJobID(new Date, 0)
(id.getJtIdentifier, id.getId)
}
@transient lazy private val jobId = new JobID(jobIdParts._1, jobIdParts._2)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
