zeroshade commented on pull request #10603: URL: https://github.com/apache/arrow/pull/10603#issuecomment-869205986
@emkornfield For your suggestion, you're saying to put that logic in the `next()` function? It could work, though I dislike having that check on every time it pulls the next message. What is the actual use case for passing in an external schema? Should we just expect that if there's an external schema passed in that there *shouldn't* be a schema message? This also ties a bit into the discussion going on in the Dev mailing list about dynamic schema changing. If we want to allow changing schemas we'd have to check on each message whether or not the message was a schema message to indicate a new schema for the subsequent messages. Currently the streaming format is defined as always having a schema message followed by the rest, the only case I can think of for passing in an external schema would be to change readers part-way through the same stream which would indicate that there's no expectation of a schema message. Unless there's a context/use case I'm not thinking of (you'd know better than I @emkornfield), I don't like the idea of putting a check in the `next` function for this, but the only other thing i can think of would be to implement a way for the reader to essentially queue up the message it read in the case that it's not a schema message so that the next call to `next()` returns that first message before continuing to read from the stream, which I like better than checking on every call to next. So i kind of rambled a bit there, but I think that made sense? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
