Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/13513
Just noticed that `FileStreamSource.getBatch(start: Option[Offset], end:
Offset)` is broken in this PR. `start` could be an arbitrary offset.
I think we need to store `batchId` with its file paths together in the
metadata log. `FileStreamSource.getBatch(start: Option[Offset], end: Offset)`
could be very slow when all batches are in the same file because we need to
parse the whole file to get the mapping from `batchId` to `files`. However, in
most cases, `FileStreamSource.getBatch` only queries the latest batch, so if we
don't compact the latest metadata file, we can make it pretty fast by reading
one small file for most of cases. When recovering from failure, the performance
of `FileStreamSource.getBatch` doesn't really matter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
