trxcllnt commented on issue #41683: URL: https://github.com/apache/arrow/issues/41683#issuecomment-2153895515
@vivek1729 It looks like you've re-implemented [`readAll`](https://github.com/apache/arrow/blob/290e606c4dd937cd34dbccd6f6801ff1ac1d8b9b/js/src/ipc/reader.ts#L710-L715) :slightly_smiling_face:. The `autoDestroy: false` option exists to prevent the RecordBatchReader from [closing the underlying transport](https://github.com/apache/arrow/blob/290e606c4dd937cd34dbccd6f6801ff1ac1d8b9b/js/src/ipc/reader.ts#L422) after reading the EOS bytes from the [IPC stream](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format). > Notice how we create a single instance of the reader The `readAll()` implementation does this as well, though that's more for convenience than anything else. Yielding a new RecordBatchReader instance for each IPC stream would not impact performance in any way (as long as the underlying ReadableStream was not closed after reading the first IPC stream). Here's a more concise version of your example, using `readAll()`: ```js import { RecordBatchReader, Table } from 'apache-arrow'; const resultTables = []; for await (const reader of RecordBatchReader.readAll(responseStream)) { resultTables.push(new Table(await reader.readAll())); } ``` You _must_ exhaust the `reader` in this loop, e.g. by using `readAll()` to collect all the RecordBatches into an Array, or by using an inner `for await (const recordBatch of reader) { ... }` loop to read each batch as it arrives. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
