dependabot[bot] opened a new pull request, #20012: URL: https://github.com/apache/druid/pull/20012
Bumps `delta-kernel.version` from 3.2.1 to 4.3.1. Updates `io.delta:delta-kernel-api` from 3.2.1 to 4.3.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/delta-io/delta/releases";>io.delta:delta-kernel-api's releases</a>.</em></p> <blockquote> <h1>Delta Lake 4.3.1</h1> <p>We are pleased to announce the release of <a href="https://github.com/delta-io/delta/releases/tag/v4.3.1";>Delta Lake 4.3.1</a>, a patch release on top of <a href="https://github.com/delta-io/delta/releases/tag/v4.3.0";>4.3.0</a> with targeted bug fixes for OAuth authentication in the Delta REST Catalog, S3A fast listing through <code>FilterFileSystem</code> wrappers, and UC managed-table metadata handling.</p> <h2>Highlights</h2> <ul> <li><strong>[Spark] Fix OAuth case-sensitivity bug in Delta REST Catalog authentication</strong>: Fixes a bug where <code>CaseInsensitiveStringMap.entrySet()</code> lowercased camelCase OAuth config keys (e.g. <code>oauth.clientId</code>), breaking Delta REST Catalog authentication.</li> </ul> <h2>Delta Spark</h2> <p>Delta Spark 4.3.1 is built on <a href="https://spark.apache.org/releases/spark-release-4.1.0.html";>Apache Spark 4.1.0</a> and <a href="https://spark.apache.org/releases/spark-release-4-0-1.html";>Apache Spark 4.0.1</a>. As with Apache Spark, we publish Maven artifacts for Scala 2.13.</p> <ul> <li>Maven artifacts for Spark 4.1.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.1_2.13/4.3.1/";>delta-spark_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.1_2.13/4.3.1/";>delta-connect-client_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.1_2.13/4.3.1/";>delta-connect-common_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.1_2.13/4.3.1/";>delta-connect-server_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/4.3.1/";>delta-contribs_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage/4.3.1/";>delta-storage</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/4.3.1/";>delta-storage-s3-dynamodb</a></li> </ul> </li> <li>Maven artifacts for Spark 4.0.1: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.0_2.13/4.3.1/";>delta-spark_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.0_2.13/4.3.1/";>delta-connect-client_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.0_2.13/4.3.1/";>delta-connect-common_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.0_2.13/4.3.1/";>delta-connect-server_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/4.3.1/";>delta-contribs_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage/4.3.1/";>delta-storage</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/4.3.1/";>delta-storage-s3-dynamodb</a></li> </ul> </li> <li>Backward-compatibility artifacts (no Spark version in the name; default to Spark 4.1.0): <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_2.13/4.3.1/";>delta-spark_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_2.13/4.3.1/";>delta-connect-client_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_2.13/4.3.1/";>delta-connect-common_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_2.13/4.3.1/";>delta-connect-server_2.13</a></li> </ul> </li> <li>Python artifact: <ul> <li><a href="https://pypi.org/project/delta-spark/4.3.1/";>delta-spark</a></li> </ul> </li> </ul> <p>Bug fixes in this release:</p> <ul> <li><strong>Support <code>delta.enabledFastS3AListFrom</code> with OSS UnityCatalog (>=0.4.1).</strong> Because the unitycatalog spark connector introduced CredScopedFileSystem, which relies on S3AFileSystem under the hood, this fix unwraps the internal filesystem of CredScopedFileSystem and handles the casting to enable fast S3A listing.</li> <li><strong>Don't persist <code>is_managed_location</code> into Delta table metadata</strong>: This reserved DSv2 catalog property was leaking into the committed <code>Metadata.configuration</code> at table creation. It is now filtered out at <code>AbstractDeltaCatalog.createTable</code> and synthesized at load for managed tables, matching the behaviour of Spark's <code>V1Table</code>.</li> </ul> <h2>Delta Kernel</h2> <p>The <a href="https://delta.io/blog/delta-kernel/";>Delta Kernel</a> project is a set of <a href="https://github.com/delta-io/delta/tree/master/kernel";>Java</a> libraries for building Delta connectors that read and write Delta tables without needing to understand the <a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md";>Delta protocol</a> directly.</p> <ul> <li>Maven artifacts: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-kernel-api/4.3.1/";>delta-kernel-api</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-kernel-defaults/4.3.1/";>delta-kernel-defaults</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-kernel-unitycatalog/4.3.1/";>delta-kernel-unitycatalog</a></li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/delta-io/delta/commit/54ce02692567fc3e5107a3ae69ebeccdffa7edfe";><code>54ce026</code></a> [Build] Set version to 4.3.1 (<a href="https://redirect.github.com/delta-io/delta/issues/7145";>#7145</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/01d6b1c2c9e72c0efb75f105ec3d013e58618a32";><code>01d6b1c</code></a> [Spark][Test] Add OAuth coverage to UC connector integration tests (<a href="https://redirect.github.com/delta-io/delta/issues/7096";>#7096</a>) (#...</li> <li><a href="https://github.com/delta-io/delta/commit/9319684c8297d2c6633dd4a6e8693ab306c4c276";><code>9319684</code></a> [Storage] Unwrap FilterFileSystem so delta.enableFastS3AListFrom works throug...</li> <li><a href="https://github.com/delta-io/delta/commit/701a26d0732cd6016fc9c7d58d7694be78c1ca2f";><code>701a26d</code></a> [Spark] Don't persist is_managed_location into Delta table metadata (<a href="https://redirect.github.com/delta-io/delta/issues/7126";>#7126</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/69753fc4cf060e0d524ec6d1529059aba96163ed";><code>69753fc</code></a> [Spark][4.3] Fix OAuth case-sensitivity bug in UCTokenBasedRestClientFactory ...</li> <li><a href="https://github.com/delta-io/delta/commit/76c12a2d1d964a31f7bb3e854db87a93421684bf";><code>76c12a2</code></a> Enable changelog support (<a href="https://redirect.github.com/delta-io/delta/issues/7007";>#7007</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/219977bb81cc563315c77aa63f5fe66bcea289bd";><code>219977b</code></a> [4.3] Support arbitrary refs and explicit version override in UC setup script...</li> <li><a href="https://github.com/delta-io/delta/commit/e1004f191a4aa928d43792c04fb9d972e1093f86";><code>e1004f1</code></a> [4.3][Spark] Propagate clustering DomainMetadata through commitLarge (<a href="https://redirect.github.com/delta-io/delta/issues/6992";>#6992</a>) ...</li> <li><a href="https://github.com/delta-io/delta/commit/bd4ba666638f4a85bd5b43f07c0dd2b91ca364c7";><code>bd4ba66</code></a> [4.3][Spark] Carry forward unrecognized delta.* properties on catalog-managed...</li> <li><a href="https://github.com/delta-io/delta/commit/2e411662930eb0edb4d3c9ac237a219adaac06bb";><code>2e41166</code></a> [4.3][build]Fix UC setup cache key collisions and local Spark-version switchi...</li> <li>Additional commits viewable in <a href="https://github.com/delta-io/delta/compare/v3.2.1...v4.3.1";>compare view</a></li> </ul> </details> <br /> Updates `io.delta:delta-kernel-defaults` from 3.2.1 to 4.3.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/delta-io/delta/releases";>io.delta:delta-kernel-defaults's releases</a>.</em></p> <blockquote> <h1>Delta Lake 4.3.1</h1> <p>We are pleased to announce the release of <a href="https://github.com/delta-io/delta/releases/tag/v4.3.1";>Delta Lake 4.3.1</a>, a patch release on top of <a href="https://github.com/delta-io/delta/releases/tag/v4.3.0";>4.3.0</a> with targeted bug fixes for OAuth authentication in the Delta REST Catalog, S3A fast listing through <code>FilterFileSystem</code> wrappers, and UC managed-table metadata handling.</p> <h2>Highlights</h2> <ul> <li><strong>[Spark] Fix OAuth case-sensitivity bug in Delta REST Catalog authentication</strong>: Fixes a bug where <code>CaseInsensitiveStringMap.entrySet()</code> lowercased camelCase OAuth config keys (e.g. <code>oauth.clientId</code>), breaking Delta REST Catalog authentication.</li> </ul> <h2>Delta Spark</h2> <p>Delta Spark 4.3.1 is built on <a href="https://spark.apache.org/releases/spark-release-4.1.0.html";>Apache Spark 4.1.0</a> and <a href="https://spark.apache.org/releases/spark-release-4-0-1.html";>Apache Spark 4.0.1</a>. As with Apache Spark, we publish Maven artifacts for Scala 2.13.</p> <ul> <li>Maven artifacts for Spark 4.1.0: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.1_2.13/4.3.1/";>delta-spark_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.1_2.13/4.3.1/";>delta-connect-client_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.1_2.13/4.3.1/";>delta-connect-common_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.1_2.13/4.3.1/";>delta-connect-server_4.1_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/4.3.1/";>delta-contribs_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage/4.3.1/";>delta-storage</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/4.3.1/";>delta-storage-s3-dynamodb</a></li> </ul> </li> <li>Maven artifacts for Spark 4.0.1: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_4.0_2.13/4.3.1/";>delta-spark_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_4.0_2.13/4.3.1/";>delta-connect-client_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_4.0_2.13/4.3.1/";>delta-connect-common_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_4.0_2.13/4.3.1/";>delta-connect-server_4.0_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-contribs_2.13/4.3.1/";>delta-contribs_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage/4.3.1/";>delta-storage</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-storage-s3-dynamodb/4.3.1/";>delta-storage-s3-dynamodb</a></li> </ul> </li> <li>Backward-compatibility artifacts (no Spark version in the name; default to Spark 4.1.0): <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-spark_2.13/4.3.1/";>delta-spark_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-client_2.13/4.3.1/";>delta-connect-client_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-common_2.13/4.3.1/";>delta-connect-common_2.13</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-connect-server_2.13/4.3.1/";>delta-connect-server_2.13</a></li> </ul> </li> <li>Python artifact: <ul> <li><a href="https://pypi.org/project/delta-spark/4.3.1/";>delta-spark</a></li> </ul> </li> </ul> <p>Bug fixes in this release:</p> <ul> <li><strong>Support <code>delta.enabledFastS3AListFrom</code> with OSS UnityCatalog (>=0.4.1).</strong> Because the unitycatalog spark connector introduced CredScopedFileSystem, which relies on S3AFileSystem under the hood, this fix unwraps the internal filesystem of CredScopedFileSystem and handles the casting to enable fast S3A listing.</li> <li><strong>Don't persist <code>is_managed_location</code> into Delta table metadata</strong>: This reserved DSv2 catalog property was leaking into the committed <code>Metadata.configuration</code> at table creation. It is now filtered out at <code>AbstractDeltaCatalog.createTable</code> and synthesized at load for managed tables, matching the behaviour of Spark's <code>V1Table</code>.</li> </ul> <h2>Delta Kernel</h2> <p>The <a href="https://delta.io/blog/delta-kernel/";>Delta Kernel</a> project is a set of <a href="https://github.com/delta-io/delta/tree/master/kernel";>Java</a> libraries for building Delta connectors that read and write Delta tables without needing to understand the <a href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md";>Delta protocol</a> directly.</p> <ul> <li>Maven artifacts: <ul> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-kernel-api/4.3.1/";>delta-kernel-api</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-kernel-defaults/4.3.1/";>delta-kernel-defaults</a></li> <li><a href="https://repo1.maven.org/maven2/io/delta/delta-kernel-unitycatalog/4.3.1/";>delta-kernel-unitycatalog</a></li> </ul> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/delta-io/delta/commit/54ce02692567fc3e5107a3ae69ebeccdffa7edfe";><code>54ce026</code></a> [Build] Set version to 4.3.1 (<a href="https://redirect.github.com/delta-io/delta/issues/7145";>#7145</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/01d6b1c2c9e72c0efb75f105ec3d013e58618a32";><code>01d6b1c</code></a> [Spark][Test] Add OAuth coverage to UC connector integration tests (<a href="https://redirect.github.com/delta-io/delta/issues/7096";>#7096</a>) (#...</li> <li><a href="https://github.com/delta-io/delta/commit/9319684c8297d2c6633dd4a6e8693ab306c4c276";><code>9319684</code></a> [Storage] Unwrap FilterFileSystem so delta.enableFastS3AListFrom works throug...</li> <li><a href="https://github.com/delta-io/delta/commit/701a26d0732cd6016fc9c7d58d7694be78c1ca2f";><code>701a26d</code></a> [Spark] Don't persist is_managed_location into Delta table metadata (<a href="https://redirect.github.com/delta-io/delta/issues/7126";>#7126</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/69753fc4cf060e0d524ec6d1529059aba96163ed";><code>69753fc</code></a> [Spark][4.3] Fix OAuth case-sensitivity bug in UCTokenBasedRestClientFactory ...</li> <li><a href="https://github.com/delta-io/delta/commit/76c12a2d1d964a31f7bb3e854db87a93421684bf";><code>76c12a2</code></a> Enable changelog support (<a href="https://redirect.github.com/delta-io/delta/issues/7007";>#7007</a>)</li> <li><a href="https://github.com/delta-io/delta/commit/219977bb81cc563315c77aa63f5fe66bcea289bd";><code>219977b</code></a> [4.3] Support arbitrary refs and explicit version override in UC setup script...</li> <li><a href="https://github.com/delta-io/delta/commit/e1004f191a4aa928d43792c04fb9d972e1093f86";><code>e1004f1</code></a> [4.3][Spark] Propagate clustering DomainMetadata through commitLarge (<a href="https://redirect.github.com/delta-io/delta/issues/6992";>#6992</a>) ...</li> <li><a href="https://github.com/delta-io/delta/commit/bd4ba666638f4a85bd5b43f07c0dd2b91ca364c7";><code>bd4ba66</code></a> [4.3][Spark] Carry forward unrecognized delta.* properties on catalog-managed...</li> <li><a href="https://github.com/delta-io/delta/commit/2e411662930eb0edb4d3c9ac237a219adaac06bb";><code>2e41166</code></a> [4.3][build]Fix UC setup cache key collisions and local Spark-version switchi...</li> <li>Additional commits viewable in <a href="https://github.com/delta-io/delta/compare/v3.2.1...v4.3.1";>compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
