lidavidm commented on pull request #11145: URL: https://github.com/apache/arrow/pull/11145#issuecomment-918347177
@edponce ScanBatchesUnorderedAsync tags batches with a sequence number, fragment, and whether this is the last batch for the fragment. There is no other end of stream marker. Everything uses this. The problem, and we've gone over this before, is that async generators have an end of stream marker, but when merging multiple generators, you need a inner end of stream marker. We could separate it from the batches, maybe Weston has thoughts. I don't think the occasional empty batch is/should be a big deal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
