Hi Robert, > There is already a way to rotate credentials [...]
I do not think so. Currently, Polaris allows only the Principal itself to rotate the credential it owns. Admin / root cannot rotate credentials for another Principal. Cheers, Dmitri. On Mon, Jul 21, 2025 at 4:32 AM Robert Stupp <sn...@snazy.de> wrote: > There is already a way to rotate credentials, so I am not sure what > the benefit of a "reset password" workflow would be. > > I do see the disadvantage that 0.9 doesn't have "external IdP" > support, but 1.0 has it. I also see that you have trouble with keeping > both versions alive. However, we have to be extremely careful with > everything related to security. > From what I understand from this thread, all feature requests are > something that IdPs do support, so I do not see a reason to mirror > those kinds of functionality in Polaris. > > The ask for "set client ID and secret" implies the need for the strong > auditing capabilities, password strength, expiry, automatic rotation, > etc > A feature like "reset password" is a full workflow implying the need > to manage expiring "reset password tokens" including sending emails, > managing user identities, etc > > Overall, that's IMHO all not in the scope of a catalog. > > > On Wed, Jul 16, 2025 at 9:24 AM Arun Suri > <arun.s...@fivetran.com.invalid> wrote: > > > > I’m aligned with the phased approach you’ve outlined: supporting password > > resets for existing principals now, while steadily enhancing IdP support > > and making it the default over time > > I will go ahead with making changes based on the discussions > > Thank you for the suggestions > > > > > > On Tue, Jul 15, 2025 at 10:12 PM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > > > > > I support Robert's proposal to leverage external IdPs in Polaris. > > > > > > However, Arun's use case also has merit, IMHO, because it enables > existing > > > deployments to migrate to newer Polaris versions, while keeping the old > > > Polaris Principals. > > > > > > Migrating to an external IdP, while valuable from a security > perspective, > > > is a big deployment change. > > > > > > Perhaps we could implement the password reset request for existing > Polaris > > > Principals, while gradually enhancing support for external IdPs and > making > > > it default at some point. > > > > > > Cheers, > > > Dmitri. > > > > > > On Tue, Jul 15, 2025 at 6:07 AM Robert Stupp <sn...@snazy.de> wrote: > > > > > > > Hi all, > > > > > > > > Writing the following with my "nasty security guy" hat on: > > > > > > > > Generally speaking, storing secrets is a quite sensitive topic that > > > > deserves a lot of attention upfront, during the initial > implementation > > > > and for all changes related to it. What we already have in Polaris is > > > > IMHO strictly speaking not enough, because it lacks the ability to > > > > automatically rotate secrets and targeted logging/auditing > > > > functionality. It's overall okay-ish. With the ability to provide > > > > secrets comes another requirement to ensure that secrets are strong > > > > enough and new permission checks and feature flags, which make the > > > > code more complex. > > > > > > > > The mentioned requirements are the core domain of what IdPs do (for > > > > example the OSS projects Keycloak, Authelia). > > > > > > > > From the Polaris's project PoV I think we should not go down that > > > > route but leverage IdPs to do this for us. Mistakes and oversights in > > > > the domain of secrets are common sources for vulnerabilities. > > > > > > > > On Mon, Jul 14, 2025 at 5:42 PM Dmitri Bourlatchkov < > di...@apache.org> > > > > wrote: > > > > > > > > > > Hi Arun, > > > > > > > > > > After writing my previous email I realised that we might be able to > > > > support > > > > > your use case via REST API by implementing admin-base password > reset > > > > [624]. > > > > > > > > > > The flow would be: > > > > > 1) create a principal using current ID (random client ID/secret) > > > > > 2) set your own client ID / secret via the password reset API (new) > > > > > > > > > > Would this work from your POV? > > > > > > > > > > As for me, it helps to isolate rare admin-level principal > manipulation > > > > from > > > > > the mainstream principal API, which IMHO, is a good idea from a > general > > > > > security perspective. > > > > > > > > > > Also, since we allow user-provided client IDs (after this change), > it > > > > would > > > > > make sense to allow resetting them without re-creating the > principal, > > > > hence > > > > > the new "reset" API. > > > > > > > > > > [624] https://github.com/apache/polaris/issues/624 > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Mon, Jul 14, 2025 at 11:29 AM Dmitri Bourlatchkov < > di...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > Thanks for volunteering to make a PR for this, Arun! Looking > forward > > > > to it. > > > > > > > > > > > > As discussed, please consider securing the extra capability via > (new) > > > > > > permission checks. I'd think it might be worth it to also have a > > > > feature > > > > > > flag to control the new functionality. > > > > > > > > > > > > Re: External IdP - most of the authentication code is already in > > > > `main`. > > > > > > There are a few remaining dangling pieces related to connecting > > > > external > > > > > > users to Polaris roles, though, IIRC. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Jul 14, 2025 at 6:34 AM Arun Suri < > arun.s...@fivetran.com > > > > .invalid> > > > > > > wrote: > > > > > > > > > > > >> Thanks for the detailed response Dimitri and Yufei! > > > > > >> > > > > > >> I agree with making the PR to support user-defined client ID and > > > > secret > > > > > >> via > > > > > >> the REST API, along with appropriate access checks and possibly > > > > > >> introducing > > > > > >> a new permission type/config. I will work on this > > > > > >> > > > > > >> REST fits better with our tooling as it has fewer dependencies > and > > > > > >> complications compared to the JAR-based Admin CLI. We also > believe > > > > > >> building > > > > > >> our migration logic in a neutral way (e.g., using the REST API) > is > > > > more > > > > > >> robust—no matter how the tools evolve, the API remains the > stable > > > > > >> contract. > > > > > >> > > > > > >> As for external IdP delegation, it's something we're open to > > > exploring > > > > > >> down > > > > > >> the line, though we understand it's still relatively new in > Polaris > > > > > >> > > > > > >> Thanks, > > > > > >> > > > > > >> On Sun, Jul 13, 2025 at 1:28 AM Yufei Gu <flyrain...@gmail.com> > > > > wrote: > > > > > >> > > > > > >> > The use case of passing secrets via REST is definitely valid, > and > > > > this > > > > > >> use > > > > > >> > case is not the only one we should be considering. Other > concrete > > > > > >> scenarios > > > > > >> > include: > > > > > >> > > > > > > >> > 1. Catalog federation, where Polaris needs to store > credentials > > > > to > > > > > >> > connect to remote catalogs (e.g., Hive, Glue, Unity > Catalog). > > > > > >> > 2. S3-compatible storage without STS support, where Polaris > > > must > > > > > >> persist > > > > > >> > static access keys and secrets to enable read/write > operations. > > > > > >> > > > > > > >> > Given these needs, I think it's the right time to formalize > our > > > > > >> approach to > > > > > >> > secret management by integrating Polaris with established > secret > > > > > >> managers > > > > > >> > such as HashiCorp Vault, AWS Secrets Manager, Azure Key > Vault, and > > > > > >> Google > > > > > >> > Cloud Secret Manager. > > > > > >> > > > > > > >> > While using the admin tool to inject secrets is a workable > > > > short-term > > > > > >> > solution, it’s best treated as a stopgap. > > > > > >> > > > > > > >> > The good news is that the secret management interface was > > > > introduced in > > > > > >> the > > > > > >> > Polari core already, > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > https://github.com/apache/polaris/blob/main/polaris-core/src/main/java/org/apache/polaris/core/secrets/UserSecretsManager.java > > > > > >> > , > > > > > >> > we may just need to provide wrapper implementations for > different > > > > secret > > > > > >> > managers. > > > > > >> > > > > > > >> > > > > > > >> > Yufei > > > > > >> > > > > > > >> > > > > > > >> > On Thu, Jul 10, 2025 at 12:52 PM Dmitri Bourlatchkov < > > > > di...@apache.org> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > Thanks for providing more context, Arun! > > > > > >> > > > > > > > >> > > I do not object to adding user-provided client ID and > secret to > > > > the > > > > > >> REST > > > > > >> > > API. However, I personally maintain my opinion that this > kind of > > > > > >> > operation > > > > > >> > > fits better with the Admin Tool, given the current state of > the > > > > > >> project. > > > > > >> > I > > > > > >> > > wonder what other community members think on this topic, > too. > > > > > >> > > > > > > > >> > > If we go with updating the current REST API, then limiting > > > access > > > > to > > > > > >> > > explicit client ID and secret parameters via access checks > will > > > > > >> certainly > > > > > >> > > make sense. We may need a new permission type for this, I > guess. > > > > > >> > > > > > > > >> > > Do you have the capacity to make a PR for this? > > > > > >> > > > > > > > >> > > Regarding the Admin Tool, is the difficulty in the fact > that it > > > > is a > > > > > >> CLI > > > > > >> > > tool that requires a JVM and your existing tooling is based > on > > > > > >> HTTP/REST > > > > > >> > > and is not written in java? Just trying to understand the > > > overall > > > > use > > > > > >> > case > > > > > >> > > better. > > > > > >> > > > > > > > >> > > Your point about the external vault makes me wonder whether > you > > > > might > > > > > >> be > > > > > >> > > interested in running an IdP server (e.g. keycloak) in your > > > infra > > > > and > > > > > >> > > making Polaris delegate user management to that system. > There's > > > > some > > > > > >> > > existing support for that, but I'm not sure if anyone tried > it > > > > > >> end-to-end > > > > > >> > > without any custom code on the server side (it is certainly > > > > possible > > > > > >> with > > > > > >> > > custom code). > > > > > >> > > > > > > > >> > > Thanks, > > > > > >> > > Dmitri. > > > > > >> > > > > > > > >> > > On Thu, Jul 10, 2025 at 3:30 PM Arun Suri < > > > arun.s...@fivetran.com > > > > > >> > .invalid> > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Hey Dmitri and Robert, > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > *To clarify our use case further:* > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > This isn't a one-time migration for us. We're migrating > our > > > > > >> *customers* > > > > > >> > > > from > > > > > >> > > > Polaris 0.9 (EclipseLink) to Polaris 1.0 (JDBC) gradually. > > > > During > > > > > >> this > > > > > >> > > > process, we’ll be *running both catalog servers in > parallel*, > > > > with > > > > > >> 1.0 > > > > > >> > > > acting as a *secondary/fallback* catalog. > > > > > >> > > > > > > > > >> > > > Our strategy involves: > > > > > >> > > > > > > > > >> > > > - Registering the same tables in both 0.9 and 1.0 > > > > > >> > > > - Using the *same *clientId* and *clientSecret in both > > > > catalogs > > > > > >> to > > > > > >> > > > ensure clients can authenticate seamlessly > > > > > >> > > > - Allowing us to *switch traffic between the two > catalogs*, > > > > and > > > > > >> roll > > > > > >> > > > back instantly if needed > > > > > >> > > > > > > > > >> > > > This setup requires credential continuity — not just for > > > > migration, > > > > > >> but > > > > > >> > > to > > > > > >> > > > enable *safe rollback and zero-downtime cutover*. Using > > > > different > > > > > >> > > > credentials across catalogs versions would break this > flow and > > > > > >> require > > > > > >> > > deep > > > > > >> > > > client coordination to rotate secrets, which is not > feasible > > > at > > > > > >> scale. > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > *Regarding your question, Dmitri: *I wonder how your > tooling > > > > could > > > > > >> > obtain > > > > > >> > > > Principals' secrets from the old > > > > > >> > > > > > > > > >> > > > Polaris instance for use as the new Principal creation > request > > > > > >> > parameter > > > > > >> > > > > > > > > >> > > > - We store the credentials in an external Vault as well. > So we > > > > are > > > > > >> not > > > > > >> > > > reading them from the old Polaris instance, but do have > access > > > > to > > > > > >> them. > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > We did consider raw table copying, but the differences in > > > > schema and > > > > > >> > > > hashing logic between 0.9 and 1.0 make that risky — and > harder > > > > to > > > > > >> > > > validate/test it completely due to unknown risks. > > > > > >> > > > > > > > > >> > > > So our goal with this proposal is to: > > > > > >> > > > > > > > > >> > > > - Enable a *safe, service-admin only way* to inject > known > > > > > >> > credentials > > > > > >> > > > via the API during the transition phase with > validations of > > > > > >> course > > > > > >> > > > - Keep this functionality configurable. > > > > > >> > > > > > > > > >> > > > We’re not trying to expand Polaris into a full IdP — just > to > > > > > >> provide a > > > > > >> > > > secure and practical bridge between versions. So the > change > > > > seems > > > > > >> fine > > > > > >> > to > > > > > >> > > > us. > > > > > >> > > > > > > > > >> > > > Happy to iterate on the proposal in a future sync > > > > > >> > > > > > > > > >> > > > On Wed, Jul 9, 2025 at 3:49 AM Dmitri Bourlatchkov < > > > > > >> di...@apache.org> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > >> > > > > Hi Arun, > > > > > >> > > > > > > > > > >> > > > > Thank you for starting this discussion! > > > > > >> > > > > > > > > > >> > > > > I did some poking about Keycloak and it looks like > Keycloak > > > > allows > > > > > >> > > > > user-provided Client IDs. > > > > > >> > > > > > > > > > >> > > > > I think it should be fine for Polaris to accept > > > user-provided > > > > > >> Client > > > > > >> > > IDs > > > > > >> > > > in > > > > > >> > > > > the Principal management API. I suppose we may want to > > > impose > > > > some > > > > > >> > > > > restrictions in terms of special characters, but in > general > > > a > > > > > >> > > > > previous Polaris Client ID should be valid as an input > > > > parameter > > > > > >> when > > > > > >> > > > > creating a new Principal. > > > > > >> > > > > > > > > > >> > > > > I think it should also be fine for Polaris to accept > > > > user-provided > > > > > >> > > Client > > > > > >> > > > > Secrets (passwords) when creating Principals. > > > > > >> > > > > > > > > > >> > > > > That said, from my POV using the Admin Tool is still > > > > preferable > > > > > >> for > > > > > >> > > > > migration use cases. > > > > > >> > > > > > > > > > >> > > > > My main argument in favour of the Admin Tool is that the > > > whole > > > > > >> > > migration > > > > > >> > > > > process is a deployment type of activity when the > Polaris > > > > service > > > > > >> is > > > > > >> > > > > configured for the first time. Ideally, Polaris data > would > > > > follow > > > > > >> a > > > > > >> > > > backup > > > > > >> > > > > / restore process (not currently implemented) where the > old > > > > > >> > instance's > > > > > >> > > > data > > > > > >> > > > > is exported into a file, which is then imported into > the new > > > > > >> instance > > > > > >> > > > > before it is started for the first time. > > > > > >> > > > > > > > > > >> > > > > I wonder how your tooling could obtain Principals' > secrets > > > > from > > > > > >> the > > > > > >> > old > > > > > >> > > > > Polaris instance for use as the new Principal creation > > > request > > > > > >> > > parameter. > > > > > >> > > > > > > > > > >> > > > > Cheers, > > > > > >> > > > > Dmitri. > > > > > >> > > > > > > > > > >> > > > > On Tue, Jul 8, 2025 at 9:02 AM Arun Suri < > > > > arun.s...@fivetran.com > > > > > >> > > > .invalid> > > > > > >> > > > > wrote: > > > > > >> > > > > > > > > > >> > > > > > Hi all, > > > > > >> > > > > > > > > > > >> > > > > > Following up on the suggestion from the discussion > here > > > > > >> > > > > > < > > > > > >> > > > > > > > https://github.com/apache/polaris/issues/1929#issuecomment-3045487786 > > > > > >> > > > > > > > > > >> > > > > > — thank you for the feedback so far. > > > > > >> > > > > > > > > > > >> > > > > > We’re currently migrating our self-hosted Polaris > service > > > > from > > > > > >> > > version > > > > > >> > > > > 0.9 > > > > > >> > > > > > (EclipseLink-based metastore) to version 1.0 > (JDBC-based > > > > > >> > metastore). > > > > > >> > > As > > > > > >> > > > > > part of this transition, we need to preserve the > existing > > > > > >> > `clientId` > > > > > >> > > > and > > > > > >> > > > > > `clientSecret` credentials for registered principals. > > > > > >> > > > > > These credentials are already embedded in customer > > > > workflows. > > > > > >> > > Rotating > > > > > >> > > > > them > > > > > >> > > > > > during migration would create disruptions and require > > > > cross-team > > > > > >> > > > > > coordination with clients — making both rollout and > > > rollback > > > > > >> > > > > significantly > > > > > >> > > > > > more complex. > > > > > >> > > > > > > > > > > >> > > > > > We understand the security implications of allowing > > > > arbitrary > > > > > >> > > > credentials > > > > > >> > > > > > to be passed in an API request. That said, we believe > this > > > > > >> > capability > > > > > >> > > > can > > > > > >> > > > > > be introduced safely and in a tightly controlled > manner. > > > For > > > > > >> > example: > > > > > >> > > > > > > > > > > >> > > > > > - Restricting this functionality to service admin > only. > > > > > >> > > > > > - Ensuring all credential transmissions occur only > over > > > > HTTPS > > > > > >> > > > > > - Clearly documenting that this is strictly for > > > > > >> > *migration/bootstrap > > > > > >> > > > use > > > > > >> > > > > > cases*, not for production use > > > > > >> > > > > > - Disabling this functionality by default in publicly > > > hosted > > > > > >> > > > deployments > > > > > >> > > > > > - Ensuring credentials are never logged (e.g., in > > > > observability > > > > > >> > > systems > > > > > >> > > > > > like logs or traces) > > > > > >> > > > > > > > > > > >> > > > > > Our goal is not to weaken the system's security > > > guarantees, > > > > but > > > > > >> to > > > > > >> > > > > provide > > > > > >> > > > > > a practical and secure migration path where credential > > > > > >> continuity > > > > > >> > is > > > > > >> > > > > > essential. Since there are a lot of DB schema changes > > > > involved, > > > > > >> > > Manual > > > > > >> > > > > > insertion into the metastore isn't ideal either, due > to > > > > > >> potential > > > > > >> > > > > > inconsistencies in hashing or salting logic across > > > versions > > > > — > > > > > >> > > > increasing > > > > > >> > > > > > operational risk. > > > > > >> > > > > > > > > > > >> > > > > > *### Proposal* > > > > > >> > > > > > > > > > > >> > > > > > We propose extending the existing `createPrincipal` > API to > > > > > >> > optionally > > > > > >> > > > > > accept `clientId` and `clientSecret` fields, with the > > > above > > > > > >> > > safeguards > > > > > >> > > > in > > > > > >> > > > > > place. > > > > > >> > > > > > > > > > > >> > > > > > We would appreciate your feedback on this proposal > and are > > > > > >> happy to > > > > > >> > > > > > contribute a patch once there’s alignment. We’re also > open > > > > to > > > > > >> > > > discussing > > > > > >> > > > > > this during the next Polaris Community Sync if > helpful. > > > > > >> > > > > > > > > > > >> > > > > > Arun Suri > > > > > >> > > > > > > > > > > >> > > > > > Senior Software Engineer > > > > > >> > > > > > > > > > > >> > > > > > He/him/his > > > > > >> > > > > > > > > > > >> > > > > > Engineering | Fivetran > > > > > >> > > > > > arun.s...@fivetran.com > > > > > >> > > > > > fivetran.com <//fivetran.com> > > > > > >> > > > > > <http://www.fivetran.com> > > > > > >> > > > > > [image: facebook] <https://www.facebook.com/Fivetran/ > > > > > > [image: > > > > > >> > > > twitter] > > > > > >> > > > > > < > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > >> > > > > > > > > > > > >> > > > > > [image: > > > > > >> > > > > > linkedin] <https://www.linkedin.com/company/fivetran> > > > > [image: > > > > > >> > > > instagram] > > > > > >> > > > > > <https://www.instagram.com/fivetran_ig/> > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > -- > > > > > >> > > > Arun Suri > > > > > >> > > > > > > > > >> > > > Senior Software Engineer > > > > > >> > > > > > > > > >> > > > He/him/his > > > > > >> > > > > > > > > >> > > > Engineering | Fivetran > > > > > >> > > > arun.s...@fivetran.com > > > > > >> > > > fivetran.com <//fivetran.com> > > > > > >> > > > <http://www.fivetran.com> > > > > > >> > > > [image: facebook] <https://www.facebook.com/Fivetran/> > > > [image: > > > > > >> > twitter] > > > > > >> > > > < > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > >> > > > > > > > > > >> > > > [image: > > > > > >> > > > linkedin] <https://www.linkedin.com/company/fivetran> > [image: > > > > > >> > instagram] > > > > > >> > > > <https://www.instagram.com/fivetran_ig/> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Arun Suri > > > > > >> > > > > > >> Senior Software Engineer > > > > > >> > > > > > >> He/him/his > > > > > >> > > > > > >> Engineering | Fivetran > > > > > >> arun.s...@fivetran.com > > > > > >> fivetran.com <//fivetran.com> > > > > > >> <http://www.fivetran.com> > > > > > >> [image: facebook] <https://www.facebook.com/Fivetran/> [image: > > > > twitter] > > > > > >> < > > > > > >> > > > > > > > > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > > >> > > > > > > >> [image: > > > > > >> linkedin] <https://www.linkedin.com/company/fivetran> [image: > > > > instagram] > > > > > >> <https://www.instagram.com/fivetran_ig/> > > > > > >> > > > > > > > > > > > > > > > > > > > -- > > Arun Suri > > > > Senior Software Engineer > > > > He/him/his > > > > Engineering | Fivetran > > arun.s...@fivetran.com > > fivetran.com <//fivetran.com> > > <http://www.fivetran.com> > > [image: facebook] <https://www.facebook.com/Fivetran/> [image: twitter] > > < > https://twitter.com/fivetran?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor > > > > [image: > > linkedin] <https://www.linkedin.com/company/fivetran> [image: instagram] > > <https://www.instagram.com/fivetran_ig/> >
