Thanks for volunteering to make a PR for this, Arun! Looking forward to it.
As discussed, please consider securing the extra capability via (new) permission checks. I'd think it might be worth it to also have a feature flag to control the new functionality. Re: External IdP - most of the authentication code is already in `main`. There are a few remaining dangling pieces related to connecting external users to Polaris roles, though, IIRC. Cheers, Dmitri. On Mon, Jul 14, 2025 at 6:34 AM Arun Suri <arun.s...@fivetran.com.invalid> wrote: > Thanks for the detailed response Dimitri and Yufei! > > I agree with making the PR to support user-defined client ID and secret via > the REST API, along with appropriate access checks and possibly introducing > a new permission type/config. I will work on this > > REST fits better with our tooling as it has fewer dependencies and > complications compared to the JAR-based Admin CLI. We also believe building > our migration logic in a neutral way (e.g., using the REST API) is more > robust—no matter how the tools evolve, the API remains the stable contract. > > As for external IdP delegation, it's something we're open to exploring down > the line, though we understand it's still relatively new in Polaris > > Thanks, > > On Sun, Jul 13, 2025 at 1:28 AM Yufei Gu <flyrain...@gmail.com> wrote: > > > The use case of passing secrets via REST is definitely valid, and this > use > > case is not the only one we should be considering. Other concrete > scenarios > > include: > > > > 1. Catalog federation, where Polaris needs to store credentials to > > connect to remote catalogs (e.g., Hive, Glue, Unity Catalog). > > 2. S3-compatible storage without STS support, where Polaris must > persist > > static access keys and secrets to enable read/write operations. > > > > Given these needs, I think it's the right time to formalize our approach > to > > secret management by integrating Polaris with established secret managers > > such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google > > Cloud Secret Manager. > > > > While using the admin tool to inject secrets is a workable short-term > > solution, it’s best treated as a stopgap. > > > > The good news is that the secret management interface was introduced in > the > > Polari core already, > > > > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/secrets/UserSecretsManager.java > > , > > we may just need to provide wrapper implementations for different secret > > managers. > > > > > > Yufei > > > > > > On Thu, Jul 10, 2025 at 12:52 PM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > > > > > Thanks for providing more context, Arun! > > > > > > I do not object to adding user-provided client ID and secret to the > REST > > > API. However, I personally maintain my opinion that this kind of > > operation > > > fits better with the Admin Tool, given the current state of the > project. > > I > > > wonder what other community members think on this topic, too. > > > > > > If we go with updating the current REST API, then limiting access to > > > explicit client ID and secret parameters via access checks will > certainly > > > make sense. We may need a new permission type for this, I guess. > > > > > > Do you have the capacity to make a PR for this? > > > > > > Regarding the Admin Tool, is the difficulty in the fact that it is a > CLI > > > tool that requires a JVM and your existing tooling is based on > HTTP/REST > > > and is not written in java? Just trying to understand the overall use > > case > > > better. > > > > > > Your point about the external vault makes me wonder whether you might > be > > > interested in running an IdP server (e.g. keycloak) in your infra and > > > making Polaris delegate user management to that system. There's some > > > existing support for that, but I'm not sure if anyone tried it > end-to-end > > > without any custom code on the server side (it is certainly possible > with > > > custom code). > > > > > > Thanks, > > > Dmitri. > > > > > > On Thu, Jul 10, 2025 at 3:30 PM Arun Suri <arun.s...@fivetran.com > > .invalid> > > > wrote: > > > > > > > Hey Dmitri and Robert, > > > > > > > > > > > > *To clarify our use case further:* > > > > > > > > > > > > This isn't a one-time migration for us. We're migrating our > *customers* > > > > from > > > > Polaris 0.9 (EclipseLink) to Polaris 1.0 (JDBC) gradually. During > this > > > > process, we’ll be *running both catalog servers in parallel*, with > 1.0 > > > > acting as a *secondary/fallback* catalog. > > > > > > > > Our strategy involves: > > > > > > > > - Registering the same tables in both 0.9 and 1.0 > > > > - Using the *same *clientId* and *clientSecret in both catalogs to > > > > ensure clients can authenticate seamlessly > > > > - Allowing us to *switch traffic between the two catalogs*, and > roll > > > > back instantly if needed > > > > > > > > This setup requires credential continuity — not just for migration, > but > > > to > > > > enable *safe rollback and zero-downtime cutover*. Using different > > > > credentials across catalogs versions would break this flow and > require > > > deep > > > > client coordination to rotate secrets, which is not feasible at > scale. > > > > > > > > > > > > *Regarding your question, Dmitri: *I wonder how your tooling could > > obtain > > > > Principals' secrets from the old > > > > > > > > Polaris instance for use as the new Principal creation request > > parameter > > > > > > > > - We store the credentials in an external Vault as well. So we are > not > > > > reading them from the old Polaris instance, but do have access to > them. > > > > > > > > > > > > We did consider raw table copying, but the differences in schema and > > > > hashing logic between 0.9 and 1.0 make that risky — and harder to > > > > validate/test it completely due to unknown risks. > > > > > > > > So our goal with this proposal is to: > > > > > > > > - Enable a *safe, service-admin only way* to inject known > > credentials > > > > via the API during the transition phase with validations of course > > > > - Keep this functionality configurable. > > > > > > > > We’re not trying to expand Polaris into a full IdP — just to provide > a > > > > secure and practical bridge between versions. So the change seems > fine > > to > > > > us. > > > > > > > > Happy to iterate on the proposal in a future sync > > > > > > > > On Wed, Jul 9, 2025 at 3:49 AM Dmitri Bourlatchkov <di...@apache.org > > > > > > wrote: > > > > > > > > > Hi Arun, > > > > > > > > > > Thank you for starting this discussion! > > > > > > > > > > I did some poking about Keycloak and it looks like Keycloak allows > > > > > user-provided Client IDs. > > > > > > > > > > I think it should be fine for Polaris to accept user-provided > Client > > > IDs > > > > in > > > > > the Principal management API. I suppose we may want to impose some > > > > > restrictions in terms of special characters, but in general a > > > > > previous Polaris Client ID should be valid as an input parameter > when > > > > > creating a new Principal. > > > > > > > > > > I think it should also be fine for Polaris to accept user-provided > > > Client > > > > > Secrets (passwords) when creating Principals. > > > > > > > > > > That said, from my POV using the Admin Tool is still preferable for > > > > > migration use cases. > > > > > > > > > > My main argument in favour of the Admin Tool is that the whole > > > migration > > > > > process is a deployment type of activity when the Polaris service > is > > > > > configured for the first time. Ideally, Polaris data would follow a > > > > backup > > > > > / restore process (not currently implemented) where the old > > instance's > > > > data > > > > > is exported into a file, which is then imported into the new > instance > > > > > before it is started for the first time. > > > > > > > > > > I wonder how your tooling could obtain Principals' secrets from the > > old > > > > > Polaris instance for use as the new Principal creation request > > > parameter. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Jul 8, 2025 at 9:02 AM Arun Suri <arun.s...@fivetran.com > > > > .invalid> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Following up on the suggestion from the discussion here > > > > > > < > > > https://github.com/apache/polaris/issues/1929#issuecomment-3045487786 > > > > > > > > > > > — thank you for the feedback so far. > > > > > > > > > > > > We’re currently migrating our self-hosted Polaris service from > > > version > > > > > 0.9 > > > > > > (EclipseLink-based metastore) to version 1.0 (JDBC-based > > metastore). > > > As > > > > > > part of this transition, we need to preserve the existing > > `clientId` > > > > and > > > > > > `clientSecret` credentials for registered principals. > > > > > > These credentials are already embedded in customer workflows. > > > Rotating > > > > > them > > > > > > during migration would create disruptions and require cross-team > > > > > > coordination with clients — making both rollout and rollback > > > > > significantly > > > > > > more complex. > > > > > > > > > > > > We understand the security implications of allowing arbitrary > > > > credentials > > > > > > to be passed in an API request. That said, we believe this > > capability > > > > can > > > > > > be introduced safely and in a tightly controlled manner. For > > example: > > > > > > > > > > > > - Restricting this functionality to service admin only. > > > > > > - Ensuring all credential transmissions occur only over HTTPS > > > > > > - Clearly documenting that this is strictly for > > *migration/bootstrap > > > > use > > > > > > cases*, not for production use > > > > > > - Disabling this functionality by default in publicly hosted > > > > deployments > > > > > > - Ensuring credentials are never logged (e.g., in observability > > > systems > > > > > > like logs or traces) > > > > > > > > > > > > Our goal is not to weaken the system's security guarantees, but > to > > > > > provide > > > > > > a practical and secure migration path where credential continuity > > is > > > > > > essential. Since there are a lot of DB schema changes involved, > > > Manual > > > > > > insertion into the metastore isn't ideal either, due to potential > > > > > > inconsistencies in hashing or salting logic across versions — > > > > increasing > > > > > > operational risk. > > > > > > > > > > > > *### Proposal* > > > > > > > > > > > > We propose extending the existing `createPrincipal` API to > > optionally > > > > > > accept `clientId` and `clientSecret` fields, with the above > > > safeguards > > > > in > > > > > > place. > > > > > > > > > > > > We would appreciate your feedback on this proposal and are happy > to > > > > > > contribute a patch once there’s alignment. We’re also open to > > > > discussing > > > > > > this during the next Polaris Community Sync if helpful. > > > > > > > > > > > > Arun Suri > > > > > > > > > > > > Senior Software Engineer > > > > > > > > > > > > He/him/his > > > > > > > > > > > > Engineering | Fivetran > > > > > > arun.s...@fivetran.com > > > > > > fivetran.com <//fivetran.com> > > > > > > <http://www.fivetran.com> > > > > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: > > > > twitter] > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > > > > > > > > > [image: > > > > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: > > > > instagram] > > > > > > <https://www.instagram.com/fivetran_ig/> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Arun Suri > > > > > > > > Senior Software Engineer > > > > > > > > He/him/his > > > > > > > > Engineering | Fivetran > > > > arun.s...@fivetran.com > > > > fivetran.com <//fivetran.com> > > > > <http://www.fivetran.com> > > > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: > > twitter] > > > > < > > > > > > > > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > > > > > [image: > > > > linkedin] <https://www.linkedin.com/company/fivetran> [image: > > instagram] > > > > <https://www.instagram.com/fivetran_ig/> > > > > > > > > > > > > -- > Arun Suri > > Senior Software Engineer > > He/him/his > > Engineering | Fivetran > arun.s...@fivetran.com > fivetran.com <//fivetran.com> > <http://www.fivetran.com> > [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] > < > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > [image: > linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] > <https://www.instagram.com/fivetran_ig/> >
