clintropolis opened a new issue #11657: URL: https://github.com/apache/druid/issues/11657
Apache Druid 0.22.0 contains over 400 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from k contributors. [See the complete set of changes for additional details](https://github.com/apache/druid/milestone/43?closed=1). # <a name="22-new-features" href="#22-new-features">#</a> New features ## <a name="22-query" href="#22-query">#</a> Query engine ### <a name="22-multi-distinct" href="#22-multi-distinct">#</a> Support for multiple distinct aggregators in same query _todo_ https://github.com/apache/druid/pull/11014 ### <a name="22-expr-agg" href="#22-expr-agg">#</a> Expression aggregator _todo_ https://github.com/apache/druid/pull/11104 ### <a name="22-array-agg" href="#22-array-agg">#</a> SQL STRING_AGG and ARRAY_AGG aggregator functions _todo_ https://github.com/apache/druid/pull/11157 https://github.com/apache/druid/pull/11241 ### <a name="22-bitwise" href="#22-bitwise">#</a> Bitwise math function expressions and aggregators _todo_ https://github.com/apache/druid/pull/10605 https://github.com/apache/druid/pull/10823 https://github.com/apache/druid/pull/11280 ### <a name="22-sql-routing" href="#22-sql-routing">#</a> SQL query routing improvements _todo_ https://github.com/apache/druid/pull/11566 https://github.com/apache/druid/pull/11495 ### <a name="22-proto-jdbc" href="#22-proto-jdbc">#</a> Avatica protobuf JDBC Support _todo_ https://github.com/apache/druid/pull/10543 ### <a name="22-query-error-log" href="#22-query-error-log">#</a> Improved query error logging _todo_ https://github.com/apache/druid/pull/11519 ## <a name="22-streaming" href="#22-streaming">#</a> Streaming Ingestion ### <a name="22-streaming-task-autoscaling" href="#22-streaming-task-autoscaling">#</a> Task autoscaling for Kafka and Kinesis streaming ingestion _todo_ https://github.com/apache/druid/pull/10524 https://github.com/apache/druid/pull/10985 ### <a name="22-avro-proto-stream" href="#22-avro-proto-stream">#</a> Avro and Protobuf streaming InputFormat and Confluent Schema Registry Support _todo_ https://github.com/apache/druid/pull/11040 https://github.com/apache/druid/pull/11018 https://github.com/apache/druid/pull/10314 https://github.com/apache/druid/pull/10839 ### <a name="22-kafka-groupid" href="#22-kafka-groupid">#</a> Kafka ingestion support for specifying group.id _todo_ https://github.com/apache/druid/pull/11147 ## <a name="22-batch" href="#22-batch">#</a> Native Batch Ingestion ### <a name="22-shuffle-deep-storage" href="#22-shuffle-deep-storage">#</a> Support for using deep storage for intermediary shuffle data _todo_ https://github.com/apache/druid/pull/11507 ### <a name="22-batch-memory" href="#22-batch-memory">#</a> Improved native batch ingestion task memory usage _todo_ https://github.com/apache/druid/pull/11123 https://github.com/apache/druid/pull/11294 ### <a name="22-batch-wait-for-handoff" href="#22-batch-wait-for-handoff">#</a> Allow batch tasks to wait until segment handoff before completion _todo_ https://github.com/apache/druid/pull/10676 ## <a name="22-compact" href="#22-compact">#</a> Data lifecycle management ### <a name="22-compact-granularity" href="#22-compact-granularity">#</a> Support managing segment and query granularity for auto-compaction _todo_ https://github.com/apache/druid/pull/10843 https://github.com/apache/druid/pull/10856 https://github.com/apache/druid/pull/10900 https://github.com/apache/druid/pull/10912 https://github.com/apache/druid/pull/11009 ### <a name="22-compact-drop" href="#22-compact-drop">#</a> Make dropExisting flag for Compaction configurable and add warning documentations _todo_ https://github.com/apache/druid/pull/11070 ### <a name="22-compact-skip-lock" href="#22-compact-skip-lock">#</a> Allow compaction to temporarily skip locked intervals _todo_ https://github.com/apache/druid/pull/11190 ### <a name="22-config-cleanup" href="#22-config-cleanup">#</a> Support for additional automatic metadata cleanup _todo_ https://github.com/apache/druid/pull/11078 https://github.com/apache/druid/pull/11084 https://github.com/apache/druid/pull/11164 https://github.com/apache/druid/pull/11200 https://github.com/apache/druid/pull/11227 https://github.com/apache/druid/pull/11232 https://github.com/apache/druid/pull/11245 ### <a name="22-data-cleanup" href="#22-data-cleanup">#</a> Dropping data _todo_ https://github.com/apache/druid/pull/11025 https://github.com/apache/druid/pull/11501 ## <a name="22-coordinator" href="#22-coordinator">#</a> Coordinator ### <a name="22-coord-timeout" href="#22-coord-timeout">#</a> Control over load/drop timeouts with Apache Zookeeper based segment management _todo_ https://github.com/apache/druid/pull/10213 ### <a name="22-coord-balance" href="#22-coord-balance">#</a> Faster coordinator segment balancing _todo_ https://github.com/apache/druid/pull/11257 ### <a name="22-coord-replication" href="#22-coord-replication">#</a> Improved loadstatus API to optionally compute under-replication based on cluster size _todo_ https://github.com/apache/druid/pull/11056 ### <a name="22-coord-rep-limit" href="#22-coord-rep-limit">#</a> Optional limits on the number of non-primary replicants loaded per coordination cycle _todo_ https://github.com/apache/druid/pull/11135 ## <a name="22-web" href="#22-web">#</a> Web Console ### <a name="22-web-general" href="#22-web-general">#</a> General improvements _todo_ https://github.com/apache/druid/pull/10951 https://github.com/apache/druid/pull/11365 https://github.com/apache/druid/pull/10795 ### <a name="22-web-query" href="#22-web-query">#</a> Query view _todo_ https://github.com/apache/druid/pull/11158 https://github.com/apache/druid/pull/11128 https://github.com/apache/druid/pull/11203 ### <a name="22-web-data" href="#22-web-data">#</a> Data management _todo_ https://github.com/apache/druid/pull/10909 https://github.com/apache/druid/pull/11359 https://github.com/apache/druid/pull/11620 ## <a name="22-metrics" href="#22-metrics">#</a> Metrics ### <a name="22-metrics-prometheus" href="#22-metrics-prometheus">#</a> Prometheus metric emitter _todo_ https://github.com/apache/druid/pull/10412 ### <a name="22-metrics-queue" href="#22-metrics-queue">#</a> New metric ingest/notices/queueSize for supervisor monitoring _todo_ https://github.com/apache/druid/pull/11417 ### <a name="22-metrics-count" href="#22-metrics-count">#</a> New metric query/segments/count for the number of segments which participate in a query _todo_ https://github.com/apache/druid/pull/11394 ## <a name="22-clouds" href="#22-clouds">#</a> Cloud integrations ### <a name="22-clouds-web-ident" href="#22-clouds-web-ident">#</a> AWS Web Identity / IRSA Support _todo_ https://github.com/apache/druid/pull/10541 ### <a name="22-clouds-assume-role" href="#22-clouds-assume-role">#</a> S3 ingestion support for assuming a role _todo_ https://github.com/apache/druid/pull/10995 ### <a name="22-clouds-gcs-lookup" href="#22-clouds-gcs-lookup">#</a> Google Cloud Storage support for URI lookups _todo_ https://github.com/apache/druid/pull/11026 ## <a name="22-odds-and-ends" href="#22-odds-and-ends">#</a> Other changes ### <a name="22-avro-union-by-type" href="#22-avro-union-by-type">#</a> Extracting Avro union fields by type _todo_ https://github.com/apache/druid/pull/10505 ### <a name="22-mysql-mariadb" href="#22-mysql-mariadb">#</a> Support using MariaDb connector with MySQL extensions _todo_ https://github.com/apache/druid/pull/11402 ### <a name="22-dynamic-config" href="#22-dynamic-config">#</a> Add Environment Variable DynamicConfigProvider _todo_ https://github.com/apache/druid/pull/11377 ### <a name="22-dynamic-config-schema-registry" href="#22-dynamic-config-schema-registry">#</a> Add DynamicConfigProvider for Schema Registry _todo_ https://github.com/apache/druid/pull/11362 # <a name="22-security" href="#22-security">#</a> Bug fixes ## <a name="22-security-protos" href="#22-security-protos">#</a> Control of allowed protocols for HTTP and HDFS input sources _todo_ https://github.com/apache/druid/pull/10830 ## <a name="22-security-ldap" href="#22-security-ldap">#</a> Fix expiration logic for LDAP internal credential cache _todo_ https://github.com/apache/druid/pull/11395 # <a name="22-perf-improvements" href="#22-perf-improvements">#</a> Performance improvements ## <a name="22-perf-query" href="#22-perf-query">#</a> Query engine ### <a name="22-perf-general" href="#22-perf-general">#</a> General performance _todo_ https://github.com/apache/druid/pull/11171 https://github.com/apache/druid/pull/10904 ### <a name="22-perf-grouper" href="#22-perf-grouper">#</a> SQL group-by query will now use query granularity when possible _todo_ https://github.com/apache/druid/pull/11379 ### <a name="22-perf-joins" href="#22-perf-joins">#</a> JOIN query enhacements _todo_ https://github.com/apache/druid/pull/11068 https://github.com/apache/druid/pull/10697 https://github.com/apache/druid/pull/10947 ### <a name="22-perf-metadata" href="#22-perf-metadata">#</a> Segment metadata query improvements _todo_ https://github.com/apache/druid/pull/10892 ## <a name="22-perf-vector" href="#22-perf-vector">#</a> Vectorized query engine ### <a name="22-vector-aggs" href="#22-vector-aggs">#</a> DataSketches theta and quantiles aggregators _todo_ https://github.com/apache/druid/pull/11183 https://github.com/apache/druid/pull/10767 ### <a name="22-vector-aggs" href="#22-vector-aggs">#</a> Cardinality aggregator _todo_ https://github.com/apache/druid/pull/11182 ### <a name="22-vector-expr" href="#22-vector-expr">#</a> Expression processing improvements _todo_ https://github.com/apache/druid/pull/10613 https://github.com/apache/druid/pull/11010 https://github.com/apache/druid/pull/11213 ### <a name="22-vector-longs" href="#22-vector-longs">#</a> Improved LONG column scan speeds, especially for 'auto' encoded columns _todo_ https://github.com/apache/druid/pull/11004 https://github.com/apache/druid/pull/11039 ## <a name="22-perf-sql" href="#22-perf-sql">#</a> SQL ### <a name="22-jdbc-frame" href="#22-jdbc-frame">#</a> Druid JDBC server result batch size configuration _todo_ https://github.com/apache/druid/pull/10880 ### <a name="22-sql-parse" href="#22-sql-parse">#</a> SQL parsing improvements _todo_ https://github.com/apache/druid/pull/11041 ### <a name="22-sys-seg" href="#22-sys-seg">#</a> Improved SYS.SEGMENTS performance _todo_ https://github.com/apache/druid/pull/11008 ### <a name="22-sql-broker" href="#22-sql-broker">#</a> Improve broker SQL schema building _todo_ https://github.com/apache/druid/pull/11457 # <a name="22-bug-fixes" href="#22-bug-fixes">#</a> Bug fixes Druid 0.22.0 contains over 80 bug fixes, you can see [the complete list here](https://github.com/apache/druid/pulls?q=label%3ABug+milestone%3A0.22.0+is%3Aclosed+). # <a name="22-upgrading-to-22" href="#22-upgrading-to-22">#</a> Upgrading to 0.22.0 Consider the following changes and updates when upgrading from Druid 0.21.x to 0.22.0. If you're updating from an earlier version than 0.21.0, see the release notes of the relevant intermediate versions. ### <a name="22-dropped-zk-3.4" href="#22-dropped-zk-3.4">#</a> Dropped support for Apache ZooKeeper 3.4 Following up to 0.21, which officially deprecated support for Zookeeper 3.4, [which has been end-of-life for a while](https://mail-archives.apache.org/mod_mbox/zookeeper-user/202004.mbox/%3C41A7EC67-D8F4-4C3A-B2DB-C2741C2EECA3%40apache.org%3E), support for ZooKeeper 3.4 is now removed in 0.22.0. Be sure to upgrade your Zookeeper cluster prior to upgrading your Druid cluster to 0.22.0. https://github.com/apache/druid/issues/10780 https://github.com/apache/druid/pull/11073 DruidInputSource: Fix issues in column projection, timestamp handling. https://github.com/apache/druid/pull/10267 Fix idempotence of segment allocation and task report apis in native batch ingestion https://github.com/apache/druid/pull/11189 SQL timeseries no longer skip empty buckets with all granularity https://github.com/apache/druid/pull/11188 Web console: Remove support for IE11 and other older browsers https://github.com/apache/druid/pull/11357 Update default maxSegmentsInNodeLoadingQueue https://github.com/apache/druid/pull/11540 # <a name="22-dev-notes" href="#22-dev-notes">#</a> Developer notices enrich expression cache key information to support expressions which depend on external state https://github.com/apache/druid/pull/11358 Replace Processing ExecutorService with QueryProcessingPool https://github.com/apache/druid/pull/11382 Make SegmentLoader extensible and customizable https://github.com/apache/druid/pull/11398 # <a name="22-known-issues" href="#22-known-issues">#</a> Known issues For a full list of open issues, please see https://github.com/apache/druid/labels/Bug. # <a name="22-credits" href="#22-credits">#</a> Credits Thanks to everyone who contributed to this release! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
