jkolash commented on issue #2514:
URL:
https://github.com/apache/iceberg-python/issues/2514#issuecomment-3456420605
Thanks for responding Kevin,
Thinking about this some more.
In a developer laptop environment I'd ideally want to prevent credentials
from being stored as plaintext and left around so I'd want to prevent or error
out if credentials were even stored in the file and enforce they are always
coming from externally such as a secrets/password manager
In a secure computing environment it may be acceptable to have a process
provision the .pyiceberg.yaml file and lay it down unencrypted before running
the program, but would probably be more ideal to inject from a secerets manager
so it is the same process/code as the developer laptop scenario.
I think my options are
### Write/read + load the pyiceberg config myself entirely without relying
on any of the built in configuration logic.
I won't go with this option, but it is possible.
### Have 2 paired up config files 1 for .pyiceberg.yaml 1 for the secrets
that should be interpolated into .pyiceberg.yaml named catalogs
- I'd likely have .pyiceberg.secrets.yaml that is just a
```
catalog1:
credential: op://vault/secret/field
catalog2:
credential: secretsmanager://name/value/field
```
- Also Have a PR check to inspect all .pyiceberg.yaml files in a git
repository to ensure credential is always not defined
- Also Have a PR check to inspect all .pyiceberg.secrets.yaml files in a git
repository to ensure credential is always a reference not a literal.
This uses
```
secrets = load_my_secrets(catalog_name) # reads from .pyiceberg.secrets.yaml
+ interpolates them at runtime from the secrets manager.
load_catalog(catalog_name, **secrets)
```
### Continue using pyiceberg.catalog._ENV_CATALOG
This is probably the least amount of work, but may break in the future.
- Also Have a PR check to inspect all .pyiceberg.yaml files in a git
repository to ensure credential is always a reference not a literal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
