rohityadav1993 opened a new issue, #11954: URL: https://github.com/apache/pinot/issues/11954
[11914](https://github.com/apache/pinot/issues/11914) will add support for adding segments to realtime tables. There are several pros of directly ingesting data to realtime tables via an API similar to `INSERT INTO` statement. As initial thoughts, there have been several minion based improvements that would solve the large number of small sized segment created, e.g. merge & roll up task for realtime tables, upsert compaction for upsert tables. There are several challenges e.g. partitioning the incoming messages for upsert tables or exposing APIs for client to identify the right server to ingest to, interface layer for ingesting messages from an API, etc. This also reduces the feature disparity from ES or [clickhouse](https://clickhouse.com/docs/en/guides/inserting-data) and allows for some of the existing use case migration to Pinot if needed. Starting this conversation to understand the feasibility of such a feature -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
