vinishjail97 commented on PR #897:
URL: https://github.com/apache/incubator-xtable/pull/897#issuecomment-5671116218
Thanks for this @slachiewicz, the size win is significant and it would be
good to land it in 0.5.0.
I built the bundled jar from `main` and from this branch merged onto `main`.
Sizes reproduce your table exactly, 1,097,349,977 bytes before and 423,224,763
after.
To look for gaps in the whitelist I diffed the two class lists (373,577
classes dropped), then scanned the constant pools of the ~77k surviving
platform classes (Hadoop, gRPC, Google Cloud, Hudi, Iceberg, Delta, Spark,
XTable) for references into that dropped set. Exactly 7 references dangle. Five
are unreachable from `RunSync` / `RunCatalogSync`: three Groovy references in
Hive CLI classes, and two in the S3A SSL path, which Hadoop guards with a
`LinkageError` catch. The other two look like real regressions.
### 1. S3A assume-role, once an STS endpoint is configured
```
main this PR
with fix below
A STSClientFactory.builder, endpoint='' BUILDER_OK BUILDER_OK
BUILDER_OK
B STSClientFactory.builder, regional BUILDER_OK
NoClassDefFoundError BUILDER_OK
C FileSystem path, sts.endpoint unset StsException StsException
StsException
D FileSystem path, sts.endpoint regional StsException
NoClassDefFoundError StsException
E FileSystem path, sts.endpoint VPC UnknownHost
NoClassDefFoundError UnknownHost
```
Rows A and B call `org.apache.hadoop.fs.s3a.auth.STSClientFactory.builder`
directly, so no XTable code is involved. `StsException` means the client was
fully built and AWS answered and rejected the test credentials.
Row C is your `AssumedRoleCredentialProvider` case, and it does pass on both
jars. I reproduced that, so the row in your matrix is correct. The difference
only appears once `fs.s3a.assumed.role.sts.endpoint` is set, because
`STSClientFactory` calls `getSTSEndpoint` only when that key is non-empty. That
method uses
`software.amazon.awssdk.thirdparty.org.apache.http.client.utils.URIBuilder`,
which ships only in `bundle`:
```
bundle: 766 entries under software/amazon/awssdk/thirdparty/org/apache/http/
s3: 0 sts: 0 apache-client: 0 third-party-jackson-core: 0
```
Four callers share the trigger: `AssumedRoleCredentialProvider`,
`SessionTokenBinding`, `RoleTokenBinding` and `MarshalledCredentialBinding`.
hadoop-aws 3.4.2 carries the same reference, so a Hadoop bump would not remove
it. A regional STS endpoint is required in opt-in regions, so some deployments
will set this.
### 2. `google-c2p` gRPC resolver on Google Cloud
```
main this PR
with fix below
xds NameResolverProvider registered absent
registered
google-c2p provider registered registered
registered
forced isOnGcp=true enters xDS init "Delegate resolver not
found, enters xDS init
scheme: xds"
off GCP TRANSIENT_FAILURE TRANSIENT_FAILURE
TRANSIENT_FAILURE
```
This is consistent with your reading that only `xds:` targets are affected.
`google-c2p` is the case that slips past it, because the provider stays
registered while its `xds` delegate is gone. I forced the on-GCP branch by
setting `GoogleCloudToProdNameResolver.isOnGcp` reflectively rather than using
a GCE VM. Off GCP all three jars behave identically, which matches your GCS run
passing. Separately,
`GoogleCloudToProdNameResolverProvider$SharedXdsClientPoolProviderBootstrapSetter.setBootstrap`
reaches `io.grpc.xds.InternalSharedXdsClientPoolProvider` through
`invokestatic`, and its only call site has no exception table.
### A fix that keeps most of the win
Restore `grpc-xds`, and bring `bundle` back filtered down to just the
relocated HTTP client:
```xml
<!-- xtable-utilities/pom.xml, artifactSet includes -->
<include>io.grpc:grpc-xds</include>
<include>software.amazon.awssdk:bundle</include>
```
```xml
<!-- pom.xml, inside the existing shade <execution> <filters> -->
<filter>
<artifact>software.amazon.awssdk:bundle</artifact>
<includes>
<include>software/amazon/awssdk/thirdparty/org/apache/http/**</include>
</includes>
</filter>
```
with `bundle` also re-declared as a runtime dependency in
`xtable-utilities`. I built this and every failing row above returns to `main`
behaviour. The jar lands at 437,258,789 bytes, so it still removes about 660 MB
of the original 675 MB.
Happy to send this as a PR against your branch if that is easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
