sivabalan narayanan created HUDI-5288: -----------------------------------------
Summary: Optimize drop duplicates by avoiding index look up twice Key: HUDI-5288 URL: https://issues.apache.org/jira/browse/HUDI-5288 Project: Apache Hudi Issue Type: Improvement Components: index Reporter: sivabalan narayanan we could potentially optimize DROP_DUPES feature by doing just 1 index look up. as of now, we do an explicit index look up and then drop the matched records and proceed on as a regular commit. why not proceed on as regular commit and then just after index lookup, we can filter for new inserts and proceed. -- This message was sent by Atlassian Jira (v8.20.10#820010)