HeartSaVioR commented on code in PR #47528:
URL: https://github.com/apache/spark/pull/47528#discussion_r1704813193
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TransformWithStateExec.scala:
##########
@@ -439,8 +447,9 @@ case class TransformWithStateExec(
new Path(getStateInfo.checkpointLocation,
s"${getStateInfo.operatorId.toString}")
- val storeNamePath = new Path(stateCheckpointPath, storeName)
- new Path(new Path(storeNamePath, "_metadata"), "schema")
+ val stateSchemaPath = new Path(stateCheckpointPath, "_stateSchema")
Review Comment:
Shall we describe the directory structure for previous version vs proposal,
in the PR description, so that we wouldn't require to look into code? Ideally,
we should have some doc about checkpoint structure (now the complication is
something we can't expect someone to understand without documentation) but I
agree this is a general issue and we want to solve this in general, not from
this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
