L. C. Hsieh created SPARK-46339:
-----------------------------------
Summary: Directory with number name should not be treated as
metadata log
Key: SPARK-46339
URL: https://issues.apache.org/jira/browse/SPARK-46339
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.5.0, 3.4.2, 3.3.3
Reporter: L. C. Hsieh
HDFSMetadataLog takes a metadata path as parameter. When it goes to retrieves
all batches metadata, it calls `CheckpointFileManager.list` to get all files
under the metadata path. However, currently all implementations of
`CheckpointFileManager.list` returns all files/directories under the given
path. So if there is a dictionary with name of batch number (a long value), the
directory will be returned too and cause trouble when HDFSMetadataLog goes to
read it.
Actually, `CheckpointFileManager.list` method clearly defines that it lists the
"files" in a path. That's being said, current implementations don't follow the
doc. We should fix it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
