FrankChen021 commented on issue #19902: URL: https://github.com/apache/druid/issues/19902#issuecomment-5314357036
### Reference benchmark: metadata filter pushdown for `sys.tasks` I added a JMH microbenchmark that compares the metadata-storage portion of the proposed native `sys.tasks` path: - **Pushdown:** pass a `TaskStorageQueryFilter` to `SQLMetadataStorageActionHandler`, which adds the predicates to the metadata SQL. - **Control / previous data flow:** retrieve every task row from metadata storage, deserialize it, and apply the identical predicate in Java. - Both paths use task summary columns rather than task payloads. - Trial setup verifies that both paths return exactly the same task IDs before measurement. - The dataset has 10,000 or 100,000 completed tasks, 1,000 datasources, and 100 task types. - Cases cover the task primary key, indexed datasource, indexed created-time range, and unindexed type column. A short directional smoke run with 10,000 tasks produced: | Filter | Fetch all, then filter | Filter in metadata SQL | Approx. speedup | |---|---:|---:|---:| | `task_id = ...` | 26.764 ms/op | 0.060 ms/op | 446x | | `datasource = ...` | 26.967 ms/op | 0.120 ms/op | 225x | | `type = ...` | 26.453 ms/op | 6.172 ms/op | 4.3x | | `created_time` range | 26.843 ms/op | 0.116 ms/op | 231x | These numbers used abbreviated JMH settings (one 500 ms warmup iteration and two 500 ms measurement iterations), so they are directional rather than publication-grade. The benchmark's normal configuration uses two 2-second warmups and five 2-second measurements at both dataset sizes. This intentionally isolates the metadata-storage benefit. It does **not** include Broker SQL planning, native filter extraction, the `/druid/v2/system` RPC, authorization, or Broker result processing. An end-to-end SQL benchmark should be added separately to measure the complete user-visible path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
