dependabot[bot] opened a new pull request, #20190: URL: https://github.com/apache/druid/pull/20190
Bumps `delta-kernel.version` from 4.3.1 to 4.4.0. Updates `io.delta:delta-kernel-api` from 4.3.1 to 4.4.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/delta-io/delta/releases";>io.delta:delta-kernel-api's releases</a>.</em></p> <blockquote> <h2>Delta Lake 4.4.0</h2> <p>We are excited to announce the release of <a href="https://github.com/delta-io/delta/releases/tag/v4.4.0";>Delta Lake 4.4.0</a>. This release adds Apache Spark 4.2 support, expands integration with the UC Delta Table API, and adds identity-column and generated-column support to Delta Spark.</p> <h2>Highlights</h2> <ul> <li><strong>[Spark] Apache Spark 4.2 support:</strong> Delta Spark, Delta Connect, and Delta Sharing now support Apache Spark 4.2.0.</li> <li><strong>[Kernel] [Flink]</strong> <strong>UC Delta API integration:</strong> Delta Kernel and the experimental Flink connector can use the UC Delta API for catalog-managed tables.</li> <li><strong>[Spark] Identity columns in SQL DDL:</strong> <code>CREATE TABLE</code> now supports <code>GENERATED ALWAYS AS IDENTITY</code> and <code>GENERATED BY DEFAULT AS IDENTITY</code>.</li> <li><strong>[Spark] Generated columns with Unity Catalog:</strong> Delta tables created through Unity Catalog can now define generated columns.</li> </ul> <h2>Delta Spark</h2> <p>Delta Spark 4.4.0 is built for <a href="https://spark.apache.org/releases/spark-release-4-2-0.html";>Apache Spark 4.2.0</a>, <a href="https://spark.apache.org/releases/spark-release-4.1.0.html";>Apache Spark 4.1.0</a>, and <a href="https://spark.apache.org/releases/spark-release-4-0-1.html";>Apache Spark 4.0.1</a>. As with Apache Spark, Maven artifacts are published for Scala 2.13.</p> <ul> <li>Maven artifacts for Spark 4.2.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.2_2.13/4.4.0/";>delta-spark_4.2_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.2_2.13/4.4.0/";>delta-connect-client_4.2_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.2_2.13/4.4.0/";>delta-connect-common_4.2_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.2_2.13/4.4.0/";>delta-connect-server_4.2_2.13</a></li> </ul> </li> <li>Maven artifacts for Spark 4.1.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.1_2.13/4.4.0/";>delta-spark_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.1_2.13/4.4.0/";>delta-connect-client_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.1_2.13/4.4.0/";>delta-connect-common_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.1_2.13/4.4.0/";>delta-connect-server_4.1_2.13</a></li> </ul> </li> <li>Maven artifacts for Spark 4.0.1: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.0_2.13/4.4.0/";>delta-spark_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.0_2.13/4.4.0/";>delta-connect-client_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.0_2.13/4.4.0/";>delta-connect-common_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.0_2.13/4.4.0/";>delta-connect-server_4.0_2.13</a></li> </ul> </li> <li>Backward-compatibility artifacts without a Spark version in the name default to Spark 4.2.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_2.13/4.4.0/";>delta-spark_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_2.13/4.4.0/";>delta-connect-client_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_2.13/4.4.0/";>delta-connect-common_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_2.13/4.4.0/";>delta-connect-server_2.13</a></li> </ul> </li> <li>Other Maven artifacts: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/4.4.0/";>delta-contribs_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage/4.4.0/";>delta-storage</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/4.4.0/";>delta-storage-s3-dynamodb</a></li> </ul> </li> <li>Python artifact: <ul> <li><a href="https://pypi.org/project/delta-spark/4.4.0/";>delta-spark</a></li> </ul> </li> </ul> <p>The key features of this release are:</p> <ul> <li><strong>Apache Spark 4.2 support</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/7223";>#7223</a>): Delta Spark, Delta Connect, and Delta Sharing are now built and tested with Spark 4.2.0 in addition to Spark 4.0.1 and 4.1.0.</li> <li><strong>Identity columns in Spark SQL DDL</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/7062";>#7062</a>): <code>CREATE TABLE</code> now accepts Spark's <code>GENERATED ALWAYS AS IDENTITY</code> and <code>GENERATED BY DEFAULT AS IDENTITY</code> syntax, including custom start and increment values.</li> <li><strong>Generated columns with Unity Catalog</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/7290";>#7290</a>): Generated-column expressions are now preserved during table creation. Delta computes or validates generated values during writes using the declared expressions.</li> <li><strong><code>SHOW PARTITIONS</code> support</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/6102";>#6102</a>): Users can inspect the partitions of a partitioned Delta table with the standard Spark SQL command.</li> <li><strong><code>VOID</code> column support</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/6965";>#6965</a>, <a href="https://redirect.github.com/delta-io/delta/pull/6966";>#6966</a>): On Spark 4.1 and later, Delta reads now preserve <code>VOID</code> (<code>NullType</code>) columns instead of failing or silently dropping them. This includes time-travel and path-based reads.</li> </ul> <h2>Delta Kernel</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/delta-io/delta/commit/88c008d55b38fb8f49a9a80982004cb5b06a2b59";><code>88c008d</code></a> [Release] Prepare version 4.4.0 for release (<a href="https://redirect.github.com/delta-io/delta/issues/7493";>#7493</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/513f2501810c02c43705b66b8bc872b4c8082726";><code>513f250</code></a> [4.4][Flink] Publish versioned artifacts for Flink 2.0 through 2.3 (<a href="https://redirect.github.com/delta-io/delta/issues/7484";>#7484</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/1931c036125cabf652945e9a026998680bba351b";><code>1931c03</code></a> [4.4][Flink] Copy rows before buffering sink writes (<a href="https://redirect.github.com/delta-io/delta/issues/7475";>#7475</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/d22546f66fe37735f998bfe567b5fea8e117da8f";><code>d22546f</code></a> [Spark] Add AMTIncrementalWriteSuite and deduplicate leaf MDV positions (<a href="https://redirect.github.com/delta-io/delta/issues/7377";>#7377</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/b1bb7e03462ce8a3abb6ad4c5b807e4c0b99a060";><code>b1bb7e0</code></a> [Spark] Add retries to post-commit log listing (<a href="https://redirect.github.com/delta-io/delta/issues/7393";>#7393</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/85bd13693562e31d11523cd154bb764352147830";><code>85bd136</code></a> Extend CDC testing: route DeltaCDCScala/SQLSuite through the V2 changelog (<a href="https://redirect.github.com/delta-io/delta/issues/7";>#7</a>...</li> <li><a href="https://github.com/delta-io/delta/commit/cb97bdb05e6d53e17e7a97227e0a8c6393c5fd76";><code>cb97bdb</code></a> [Flink] Preserve final checkpoint lineage (<a href="https://redirect.github.com/delta-io/delta/issues/7401";>#7401</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/b9b923f3ed60f4cbd7dd96979c783f62e23dba56";><code>b9b923f</code></a> [Kernel] Retry when winning commits are unavailable (<a href="https://redirect.github.com/delta-io/delta/issues/7402";>#7402</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/4df35d4e3af040f332211d21553802d8e60b1a4d";><code>4df35d4</code></a> [Protocol RFC] Nanosecond timestamps feature (<a href="https://redirect.github.com/delta-io/delta/issues/6082";>#6082</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/a9a146961fb1a2ada4fc1dcf05981acd8d654a9e";><code>a9a1469</code></a> [Spark] Rename DeltaSnapshotManager to DeltaV2SnapshotManager (<a href="https://redirect.github.com/delta-io/delta/issues/7391";>#7391</a>)</li> <li>Additional commits viewable in <a href="https://github.com/delta-io/delta/compare/v4.3.1...v4.4.0";>compare view</a></li> </ul> </details> <br /> Updates `io.delta:delta-kernel-defaults` from 4.3.1 to 4.4.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/delta-io/delta/releases";>io.delta:delta-kernel-defaults's releases</a>.</em></p> <blockquote> <h2>Delta Lake 4.4.0</h2> <p>We are excited to announce the release of <a href="https://github.com/delta-io/delta/releases/tag/v4.4.0";>Delta Lake 4.4.0</a>. This release adds Apache Spark 4.2 support, expands integration with the UC Delta Table API, and adds identity-column and generated-column support to Delta Spark.</p> <h2>Highlights</h2> <ul> <li><strong>[Spark] Apache Spark 4.2 support:</strong> Delta Spark, Delta Connect, and Delta Sharing now support Apache Spark 4.2.0.</li> <li><strong>[Kernel] [Flink]</strong> <strong>UC Delta API integration:</strong> Delta Kernel and the experimental Flink connector can use the UC Delta API for catalog-managed tables.</li> <li><strong>[Spark] Identity columns in SQL DDL:</strong> <code>CREATE TABLE</code> now supports <code>GENERATED ALWAYS AS IDENTITY</code> and <code>GENERATED BY DEFAULT AS IDENTITY</code>.</li> <li><strong>[Spark] Generated columns with Unity Catalog:</strong> Delta tables created through Unity Catalog can now define generated columns.</li> </ul> <h2>Delta Spark</h2> <p>Delta Spark 4.4.0 is built for <a href="https://spark.apache.org/releases/spark-release-4-2-0.html";>Apache Spark 4.2.0</a>, <a href="https://spark.apache.org/releases/spark-release-4.1.0.html";>Apache Spark 4.1.0</a>, and <a href="https://spark.apache.org/releases/spark-release-4-0-1.html";>Apache Spark 4.0.1</a>. As with Apache Spark, Maven artifacts are published for Scala 2.13.</p> <ul> <li>Maven artifacts for Spark 4.2.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.2_2.13/4.4.0/";>delta-spark_4.2_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.2_2.13/4.4.0/";>delta-connect-client_4.2_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.2_2.13/4.4.0/";>delta-connect-common_4.2_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.2_2.13/4.4.0/";>delta-connect-server_4.2_2.13</a></li> </ul> </li> <li>Maven artifacts for Spark 4.1.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.1_2.13/4.4.0/";>delta-spark_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.1_2.13/4.4.0/";>delta-connect-client_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.1_2.13/4.4.0/";>delta-connect-common_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.1_2.13/4.4.0/";>delta-connect-server_4.1_2.13</a></li> </ul> </li> <li>Maven artifacts for Spark 4.0.1: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.0_2.13/4.4.0/";>delta-spark_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.0_2.13/4.4.0/";>delta-connect-client_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.0_2.13/4.4.0/";>delta-connect-common_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.0_2.13/4.4.0/";>delta-connect-server_4.0_2.13</a></li> </ul> </li> <li>Backward-compatibility artifacts without a Spark version in the name default to Spark 4.2.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_2.13/4.4.0/";>delta-spark_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_2.13/4.4.0/";>delta-connect-client_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_2.13/4.4.0/";>delta-connect-common_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_2.13/4.4.0/";>delta-connect-server_2.13</a></li> </ul> </li> <li>Other Maven artifacts: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/4.4.0/";>delta-contribs_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage/4.4.0/";>delta-storage</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/4.4.0/";>delta-storage-s3-dynamodb</a></li> </ul> </li> <li>Python artifact: <ul> <li><a href="https://pypi.org/project/delta-spark/4.4.0/";>delta-spark</a></li> </ul> </li> </ul> <p>The key features of this release are:</p> <ul> <li><strong>Apache Spark 4.2 support</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/7223";>#7223</a>): Delta Spark, Delta Connect, and Delta Sharing are now built and tested with Spark 4.2.0 in addition to Spark 4.0.1 and 4.1.0.</li> <li><strong>Identity columns in Spark SQL DDL</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/7062";>#7062</a>): <code>CREATE TABLE</code> now accepts Spark's <code>GENERATED ALWAYS AS IDENTITY</code> and <code>GENERATED BY DEFAULT AS IDENTITY</code> syntax, including custom start and increment values.</li> <li><strong>Generated columns with Unity Catalog</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/7290";>#7290</a>): Generated-column expressions are now preserved during table creation. Delta computes or validates generated values during writes using the declared expressions.</li> <li><strong><code>SHOW PARTITIONS</code> support</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/6102";>#6102</a>): Users can inspect the partitions of a partitioned Delta table with the standard Spark SQL command.</li> <li><strong><code>VOID</code> column support</strong> (<a href="https://redirect.github.com/delta-io/delta/pull/6965";>#6965</a>, <a href="https://redirect.github.com/delta-io/delta/pull/6966";>#6966</a>): On Spark 4.1 and later, Delta reads now preserve <code>VOID</code> (<code>NullType</code>) columns instead of failing or silently dropping them. This includes time-travel and path-based reads.</li> </ul> <h2>Delta Kernel</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/delta-io/delta/commit/88c008d55b38fb8f49a9a80982004cb5b06a2b59";><code>88c008d</code></a> [Release] Prepare version 4.4.0 for release (<a href="https://redirect.github.com/delta-io/delta/issues/7493";>#7493</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/513f2501810c02c43705b66b8bc872b4c8082726";><code>513f250</code></a> [4.4][Flink] Publish versioned artifacts for Flink 2.0 through 2.3 (<a href="https://redirect.github.com/delta-io/delta/issues/7484";>#7484</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/1931c036125cabf652945e9a026998680bba351b";><code>1931c03</code></a> [4.4][Flink] Copy rows before buffering sink writes (<a href="https://redirect.github.com/delta-io/delta/issues/7475";>#7475</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/d22546f66fe37735f998bfe567b5fea8e117da8f";><code>d22546f</code></a> [Spark] Add AMTIncrementalWriteSuite and deduplicate leaf MDV positions (<a href="https://redirect.github.com/delta-io/delta/issues/7377";>#7377</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/b1bb7e03462ce8a3abb6ad4c5b807e4c0b99a060";><code>b1bb7e0</code></a> [Spark] Add retries to post-commit log listing (<a href="https://redirect.github.com/delta-io/delta/issues/7393";>#7393</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/85bd13693562e31d11523cd154bb764352147830";><code>85bd136</code></a> Extend CDC testing: route DeltaCDCScala/SQLSuite through the V2 changelog (<a href="https://redirect.github.com/delta-io/delta/issues/7";>#7</a>...</li> <li><a href="https://github.com/delta-io/delta/commit/cb97bdb05e6d53e17e7a97227e0a8c6393c5fd76";><code>cb97bdb</code></a> [Flink] Preserve final checkpoint lineage (<a href="https://redirect.github.com/delta-io/delta/issues/7401";>#7401</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/b9b923f3ed60f4cbd7dd96979c783f62e23dba56";><code>b9b923f</code></a> [Kernel] Retry when winning commits are unavailable (<a href="https://redirect.github.com/delta-io/delta/issues/7402";>#7402</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/4df35d4e3af040f332211d21553802d8e60b1a4d";><code>4df35d4</code></a> [Protocol RFC] Nanosecond timestamps feature (<a href="https://redirect.github.com/delta-io/delta/issues/6082";>#6082</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/a9a146961fb1a2ada4fc1dcf05981acd8d654a9e";><code>a9a1469</code></a> [Spark] Rename DeltaSnapshotManager to DeltaV2SnapshotManager (<a href="https://redirect.github.com/delta-io/delta/issues/7391";>#7391</a>)</li> <li>Additional commits viewable in <a href="https://github.com/delta-io/delta/compare/v4.3.1...v4.4.0";>compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
