Re: [DISCUSS] Add HashiCorp Vault KMS Support to Apache Iceberg

2026-04-30 Thread Yuya Ebihara 
Hi,

I've rewritten the code using the existing dependencies.
The only new dependency added for tests is
org.testcontainers:testcontainers-vault.

I'd like to discuss another topic next:
> Are there any objections to supporting Vault in the Iceberg repository?

BR,
Yuya

On Wed, Apr 29, 2026 at 12:08 AM Jean-Baptiste Onofré 
wrote:

> Hi all,
>
> I have two comments regarding the use of
> io.github.jopenlibs:vault-java-driver:
>
> - The last commit was four months ago, which is acceptable if the project
> is mature and stable.
> - It uses the MIT license (Category A), which is compatible.
>
> That said, since our client requirements do not appear to be overly
> complex, I believe it is worth leveraging our existing HTTP client instead.
> This would allow us to utilize dependencies we already have in the stack.
>
> Regards,
> JB
>
> On Tue, Apr 28, 2026 at 2:47 PM Yuya Ebihara <
> [email protected]> wrote:
>
>> Thanks for the feedback!
>>
>> I've switched to io.github.jopenlibs:vault-java-driver for now.
>>
>> Rewriting it using the existing HTTP client is also an option if
>> minimizing dependencies is important.
>> The usage isn't particularly complex, so I don't expect a high
>> maintenance cost.
>> I believe the Spring Framework takes a similar approach.
>>
>> On Tue, Apr 28, 2026 at 9:29 PM Romain Manni-Bucau 
>> wrote:
>>
>>> Hi,
>>>
>>> why not just reusing http client already there in the stack (for rest
>>> catalog)? for the token a side car can be a simple first step else it is
>>> not crazy to do in plain java without any particular dep
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  | .NET Blog
>>>  | Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>> Javaccino founder (Java/.NET service - contact via linkedin)
>>>
>>>
>>> Le mar. 28 avr. 2026 à 13:45, Steve Loughran  a
>>> écrit :
>>>


 On Tue, 28 Apr 2026 at 01:18, Yuya Ebihara <
 [email protected]> wrote:

>
>
>- Are there any concerns about introducing a dependency on the
>Vault client library?
>
>
> I worry about all dependencies these days -as every library is
 effectively a CVE subscription.

 That bettercloud driver is 7 years old and depends on out of date
 versions of bouncycastle and more

 https://mvnrepository.com/artifact/com.bettercloud/vault-java-driver/5.1.0/dependencies


 Those same people now appear to be working on s different project
 https://mvnrepository.com/artifact/io.github.jopenlibs/vault-java-driver

>>>

Re: [DISCUSS] Add HashiCorp Vault KMS Support to Apache Iceberg

2026-04-28 Thread Jean-Baptiste Onofré 
Hi all,

I have two comments regarding the use of
io.github.jopenlibs:vault-java-driver:

- The last commit was four months ago, which is acceptable if the project
is mature and stable.
- It uses the MIT license (Category A), which is compatible.

That said, since our client requirements do not appear to be overly
complex, I believe it is worth leveraging our existing HTTP client instead.
This would allow us to utilize dependencies we already have in the stack.

Regards,
JB

On Tue, Apr 28, 2026 at 2:47 PM Yuya Ebihara 
wrote:

> Thanks for the feedback!
>
> I've switched to io.github.jopenlibs:vault-java-driver for now.
>
> Rewriting it using the existing HTTP client is also an option if
> minimizing dependencies is important.
> The usage isn't particularly complex, so I don't expect a high
> maintenance cost.
> I believe the Spring Framework takes a similar approach.
>
> On Tue, Apr 28, 2026 at 9:29 PM Romain Manni-Bucau 
> wrote:
>
>> Hi,
>>
>> why not just reusing http client already there in the stack (for rest
>> catalog)? for the token a side car can be a simple first step else it is
>> not crazy to do in plain java without any particular dep
>>
>> Romain Manni-Bucau
>> @rmannibucau  | .NET Blog
>>  | Blog  |
>> Old Blog  | Github
>>  | LinkedIn
>>  | Book
>> 
>> Javaccino founder (Java/.NET service - contact via linkedin)
>>
>>
>> Le mar. 28 avr. 2026 à 13:45, Steve Loughran  a
>> écrit :
>>
>>>
>>>
>>> On Tue, 28 Apr 2026 at 01:18, Yuya Ebihara <
>>> [email protected]> wrote:
>>>


- Are there any concerns about introducing a dependency on the
Vault client library?


 I worry about all dependencies these days -as every library is
>>> effectively a CVE subscription.
>>>
>>> That bettercloud driver is 7 years old and depends on out of date
>>> versions of bouncycastle and more
>>>
>>> https://mvnrepository.com/artifact/com.bettercloud/vault-java-driver/5.1.0/dependencies
>>>
>>>
>>> Those same people now appear to be working on s different project
>>> https://mvnrepository.com/artifact/io.github.jopenlibs/vault-java-driver
>>>
>>

Re: [DISCUSS] Add HashiCorp Vault KMS Support to Apache Iceberg

2026-04-28 Thread Yuya Ebihara 
Thanks for the feedback!

I've switched to io.github.jopenlibs:vault-java-driver for now.

Rewriting it using the existing HTTP client is also an option if minimizing
dependencies is important.
The usage isn't particularly complex, so I don't expect a high maintenance
cost.
I believe the Spring Framework takes a similar approach.

On Tue, Apr 28, 2026 at 9:29 PM Romain Manni-Bucau 
wrote:

> Hi,
>
> why not just reusing http client already there in the stack (for rest
> catalog)? for the token a side car can be a simple first step else it is
> not crazy to do in plain java without any particular dep
>
> Romain Manni-Bucau
> @rmannibucau  | .NET Blog
>  | Blog  |
> Old Blog  | Github
>  | LinkedIn
>  | Book
> 
> Javaccino founder (Java/.NET service - contact via linkedin)
>
>
> Le mar. 28 avr. 2026 à 13:45, Steve Loughran  a
> écrit :
>
>>
>>
>> On Tue, 28 Apr 2026 at 01:18, Yuya Ebihara <
>> [email protected]> wrote:
>>
>>>
>>>
>>>- Are there any concerns about introducing a dependency on the Vault
>>>client library?
>>>
>>>
>>> I worry about all dependencies these days -as every library is
>> effectively a CVE subscription.
>>
>> That bettercloud driver is 7 years old and depends on out of date
>> versions of bouncycastle and more
>>
>> https://mvnrepository.com/artifact/com.bettercloud/vault-java-driver/5.1.0/dependencies
>>
>>
>> Those same people now appear to be working on s different project
>> https://mvnrepository.com/artifact/io.github.jopenlibs/vault-java-driver
>>
>

Re: [DISCUSS] Add HashiCorp Vault KMS Support to Apache Iceberg

2026-04-28 Thread Romain Manni-Bucau 
Hi,

why not just reusing http client already there in the stack (for rest
catalog)? for the token a side car can be a simple first step else it is
not crazy to do in plain java without any particular dep

Romain Manni-Bucau
@rmannibucau  | .NET Blog
 | Blog  | Old
Blog  | Github
 | LinkedIn
 | Book

Javaccino founder (Java/.NET service - contact via linkedin)


Le mar. 28 avr. 2026 à 13:45, Steve Loughran  a écrit :

>
>
> On Tue, 28 Apr 2026 at 01:18, Yuya Ebihara 
> wrote:
>
>>
>>
>>- Are there any concerns about introducing a dependency on the Vault
>>client library?
>>
>>
>> I worry about all dependencies these days -as every library is
> effectively a CVE subscription.
>
> That bettercloud driver is 7 years old and depends on out of date versions
> of bouncycastle and more
>
> https://mvnrepository.com/artifact/com.bettercloud/vault-java-driver/5.1.0/dependencies
>
>
> Those same people now appear to be working on s different project
> https://mvnrepository.com/artifact/io.github.jopenlibs/vault-java-driver
>

Re: [DISCUSS] Add HashiCorp Vault KMS Support to Apache Iceberg

2026-04-28 Thread Steve Loughran 
On Tue, 28 Apr 2026 at 01:18, Yuya Ebihara 
wrote:

>
>
>- Are there any concerns about introducing a dependency on the Vault
>client library?
>
>
> I worry about all dependencies these days -as every library is effectively
a CVE subscription.

That bettercloud driver is 7 years old and depends on out of date versions
of bouncycastle and more
https://mvnrepository.com/artifact/com.bettercloud/vault-java-driver/5.1.0/dependencies


Those same people now appear to be working on s different project
https://mvnrepository.com/artifact/io.github.jopenlibs/vault-java-driver

[DISCUSS] Add HashiCorp Vault KMS Support to Apache Iceberg

2026-04-27 Thread Yuya Ebihara 
Hi all,

I'd like to propose adding HashiCorp Vault as a first-class KMS
implementation in Apache Iceberg's encryption framework.

*Background*

Iceberg's encryption spec supports pluggable KMS backends, but the set of
officially supported providers is still limited.
HashiCorp Vault is one of the most widely adopted secrets management and
encryption-as-a-service platforms in the industry.
The Vault repository  has over 35k
GitHub stars, and it's a common choice in enterprises that prefer
self-hosted or cloud-agnostic key management.

*Motivation*

I recently received a request to support Vault-encrypted Iceberg tables in
Trino.
Rather than implementing this in the Trino repository alone, I believe the
right place for this is upstream in Iceberg itself. Adding it here means:

   1. Any engine benefits immediately - Spark, Flink, Trino, and others can
   all interoperate with Vault-encrypted tables without each needing their
   own implementation.
   2. No duplicated effort - a single, well-tested integration in Iceberg
   avoids the maintenance burden of per-engine implementations diverging over
   time.
   3. Consistency across the ecosystem - users get the same configuration
   interface and behavior regardless of which engine they use.

*Proposal*

I've opened PR #16075  as a
starting point. It introduces:

   - A new iceberg-hashicorp module containing the Vault KMS client
   implementation
   - A new hashicorp value for the kms-type configuration property

*Questions for the Community*

   - Is there any objection to supporting Vault in the Iceberg repository?
   - Are there any concerns about introducing a dependency on the Vault
   client library?

Happy to discuss the design further. Thanks for taking a look.


BR,
Yuya Ebihara