viirya commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-858400265
> Your rephrase looks good except for one point here. "task (i.e. state store)"? You mean task is kind of a type of state store? is it a typo? I actually expect that it's a mapping between PVC and task Id. A specific statestore is bound to a task, e.g. task 0 is bound with state store 0. This cannot be changed. But where to schedule the task, could be changed generally. This is current situation for HDFS-backed statestore. In other word, task-statestore is moved together if Spark schedule the task to different executor. So actually the mapping between PVC and task id, also means a (implicit) mapping between PVC and statestore of the task. That is why I add "i.e." there. Sorry for if any confusing. > I don't understand this. I assume each statestore must be bound to a specific location. Why we can't schedule the task? For current HDFS-backed statestore, the location can be changed. It is when Spark schedules the task with the statestore to new executor. Once it is changed to different executor, Spark will reload from checkpointed data from HDFS to construct the state store in new executor (location). For resource-specific case (e.g. PVC), the location is fixed generally, because it is bound to specific resource on the executor. But in case like executor lost and the resource is re-mountable. Spark can schedule the task with the statestore to new executor re-mounted with the resource. > `ResourceLocation` sounds too general. Mabye, `RequiredResourceLocation`? `RequiredResourceLocation` sounds good to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
