hampsterx opened a new issue, #50888:
URL: https://github.com/apache/arrow/issues/50888
### Describe the enhancement requested
`GcsOptions::FromServiceAccountCredentials`
(`cpp/src/arrow/filesystem/gcsfs.h:146`) is the
one credential factory on `GcsOptions` with no Python binding, so a caller
holding
[aip/4112] service-account JSON has no way to construct a `GcsFileSystem`
from it.
`python/pyarrow/includes/libarrow_fs.pxd` declares the other four
(`Defaults`, `Anonymous`,
`FromAccessToken`, `FromImpersonatedServiceAccount`, lines 229-240), and the
credential-routing chain inside `GcsFileSystem.__init__`
(`python/pyarrow/_gcsfs.pyx:103-131`,
method at line 88) selects among them. No constructor keyword reaches the
fifth. Line
references are against `d048f71`; behaviour checked on pyarrow 25.0.1.
Two routes are open today to an application holding that JSON in memory, and
each gives
something up:
- Write it to a file and point `GOOGLE_APPLICATION_CREDENTIALS` at it. That
puts the private
key on disk, and it configures the process rather than the filesystem
object, so two
filesystems in one process cannot use different service accounts. The
docstring on this
function names the case it exists for: "Using this function may be useful
when the json
object is obtained from a Cloud Secret Manager or a similar service."
- Mint an access token out of band and pass `access_token` +
`credential_token_expiration`.
Per `FromAccessToken`'s docstring those tokens are time limited and the
caller refreshes
them manually, and the Python API offers no way to replace the token on an
existing
`GcsFileSystem`, so a refresh means constructing a new one.
Concrete case: omniload (panodata/omniload#233) accepts GCS service-account
JSON as either a
path or a base64 blob from a secret store. Its S3 and R2 transports moved to
`pyarrow.fs` for
the per-file open latency (on that sibling transport, 8.68 ms/file through
s3fs against
3.49 ms/file through Arrow, on the reader path that opens one file object
per file). GCS
stayed on gcsfs, because the swap would drop that credential mode.
Proposed shape, following the existing chain:
- Declare `FromServiceAccountCredentials` in `libarrow_fs.pxd` beside the
other four.
- Add a `GcsFileSystem.__init__` keyword taking the aip/4112 JSON as a
string, in the same
base-credential branch as `anonymous` and `access_token` and mutually
exclusive with both.
`service_account_json` reads more clearly than
`service_account_credentials`, since the
value is serialized JSON rather than a credentials object; the C++
accessor calls it
`json_credentials`. It would stay composable with `target_service_account`
by passing the
resulting credentials to `FromImpersonatedServiceAccount`, as the
constructor already does
for the other base credentials.
- Pickling would need a policy decision, and I have not assumed one.
`__reduce__` round-trips
`access_token` today (`_gcsfs.pyx:183`), but a long-lived private key is a
different
proposition and declining to serialize this mode is a reasonable answer.
If it should
round-trip through the existing `__reduce__` design, the pxd would also
need
`GcsCredentials::json_credentials()`, which exists in C++ at `gcsfs.h:47`
and is not
declared there.
#32207 is the same class of binding gap for R, on the sibling impersonation
factory.
Does a constructor keyword fit how you would want this exposed, or would you
rather it
arrive as a separate classmethod?
### Component(s)
Python
[aip/4112]: https://google.aip.dev/auth/4112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
