navina opened a new issue, #9771:
URL: https://github.com/apache/pinot/issues/9771
We noticed that when a partial upsert enabled table is reloaded with a new
column in the schema, the upsert behavior is not honored for the new column.
This can be reproduced deterministically and the likely cause is described
below:
We keep a list of columns which need to be updated in `PartialUpsertHandler`.
```
// _column2Mergers maintains the mapping of merge strategies per columns.
private final Map<String, PartialUpsertMerger> _column2Mergers = new
HashMap<>();
```
This is initialized during `RealtimeTableDataManager#doInit`. However, it is
not updated when we reload the table. This leads to the realtime engine not
being aware of the new column. Hence, it will not handle upsert for this
column. This new column will look the same as if the table didn't not have
upserts.
Thus, as of today, any new column will be ignored in partial upsert. We need
to be enhance partial upsert to handle new columns in the schema.
cc: @KKcorps @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
