vinishjail97 commented on code in PR #897:
URL: https://github.com/apache/incubator-xtable/pull/897#discussion_r3833813299
##########
xtable-utilities/pom.xml:
##########
@@ -215,10 +215,30 @@
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
+ <exclusions>
+ <!-- The all-services SDK jar. We name the artifacts S3A uses
instead. -->
+ <exclusion>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bundle</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
+ <!-- What hadoop-aws S3A needs. The all-services `bundle` artifact was
63% of the shaded jar. -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
- <artifactId>bundle</artifactId>
+ <artifactId>s3</artifactId>
Review Comment:
Thank you for working on this @slachiewicz!
I agree with your reading that static analysis cannot see a class named in
configuration, and I would like to help scope what "someone should run this"
means concretely before it merges.
This jar is primarily used when running xtable as a server and not as a
library. It is also the jar that our documentation tells users to run. `java
-jar xtable-utilities/target/xtable-utilities_2.12-<version>-bundled.jar
--datasetConfig my_config.yaml` appears in six places:
- `website/docs/how-to.md:354` — the main quickstart, with `s3://` and
`gs://` base paths at lines 303, 318 and 332
- `website/docs/glue-catalog.md:87` — S3 source plus a Glue Data Catalog
target
- `website/docs/biglake-metastore.md` — GCS source plus BigLake
- `website/docs/unity-catalog.md:51` — S3, GCS and ADLS (`abfss://`) sources
- `website/docs/hms.md` and `website/docs/how-to-catalog-sync.md` — the same
three schemes plus HMS
Would you be willing to run the doc flows above end to end, or say which of
them you can reach? My suggestion, roughly in order of what I would worry about:
1. `how-to.md` against a real S3 bucket, with the shipped defaults
untouched, since that is the path most users take first.
2. `glue-catalog.md`, because it is the only flow that exercises
`software.amazon.awssdk:glue` through this jar rather than through `xtable-aws`.
3. `how-to.md` against GCS. I would flag this one specifically:
`io.grpc:grpc-googleapis` stays in the whitelist and I found it carries a
bytecode reference to `io/grpc/xds/InternalSharedXdsClientPoolProvider`, which
lives in the `grpc-xds` you drop here. `fs.gs.grpc.enable` defaults to `false`
in gcs-connector 2.2.22, so the ordinary path is probably fine, but a run with
it set to `true` would settle it. For context, `grpc-xds` is 40 MB of the
roughly 675 MB saved.
4. `unity-catalog.md` or `hms.md` against ADLS. Azure is untouched by this
PR, so this is a regression check rather than a suspicion.
I am happy to take some of these if that helps, let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
