noob-se7en commented on PR #18433: URL: https://github.com/apache/pinot/pull/18433#issuecomment-4515345135
> The multi-topic partition id overloading existing partition id has caused so many issues: > > * We are missing checks here and there > * Stream with custom partition id fails (See #17260) > > I think it is the time to revisit this design decision and avoid the troubles going forward. I can see 2 possible solutions: > > * Make `partition_id` part within the table name `x_y` where x is the topic id and y is the partition id. For single topic name, it remains the same > * Leverage logical table for multiple topics, each topic a separate physical table (Not related to this PR) Big +1 on this. The base design itself needs to be improved, the realtime ingestion code has suffered because of it (correctness + quality wise), and reviews are harder as well (e.g., https://github.com/apache/pinot/pull/18175). There is diminishing interest in reviewing further development on multi-topic due to these issues. Bumping up the issue related to this discussion: https://github.com/apache/pinot/issues/17260 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
