JonasJ-ap commented on code in PR #7119:
URL: https://github.com/apache/iceberg/pull/7119#discussion_r1137904028
##########
docs/aws.md:
##########
@@ -60,7 +60,7 @@ AWS_SDK_VERSION=2.20.18
AWS_MAVEN_GROUP=software.amazon.awssdk
AWS_PACKAGES=(
"bundle"
- "url-connection-client"
+ "apache-client"
Review Comment:
It seems that `apache-client` is included in the `bundle`. I ran
```bash
jar -tf bundle-2.20.18.jar | egrep "http/apache"
```
and got
```bash
...
software/amazon/awssdk/http/apache/ProxyConfiguration$Builder.class
software/amazon/awssdk/http/apache/ProxyConfiguration$DefaultClientProxyConfigurationBuilder.class
software/amazon/awssdk/http/apache/ProxyConfiguration$1.class
software/amazon/awssdk/http/apache/ProxyConfiguration.class
software/amazon/awssdk/http/apache/ApacheSdkHttpService.class
software/amazon/awssdk/http/apache/ApacheHttpClient$1.class
software/amazon/awssdk/http/apache/ApacheHttpClient$Builder.class
software/amazon/awssdk/http/apache/ApacheHttpClient$DefaultBuilder.class
software/amazon/awssdk/http/apache/ApacheHttpClient$ApacheConnectionManagerFactory$1.class
software/amazon/awssdk/http/apache/ApacheHttpClient$ApacheConnectionManagerFactory.class
software/amazon/awssdk/http/apache/ApacheHttpClient.class
```
I also verified that using the following script
```sh
BRANCH_NAME=s3_credentials
DEPENDENCIES=""
# add AWS dependnecy
AWS_SDK_VERSION=2.17.257
AWS_MAVEN_GROUP=software.amazon.awssdk
AWS_PACKAGES=(
"bundle"
)
for pkg in "${AWS_PACKAGES[@]}"; do
DEPENDENCIES+="$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION,"
done
JARS="iceberg-spark-runtime-3.3_$BRANCH_NAME.jar"
# start Spark SQL client shell
spark-shell --packages=$DEPENDENCIES --jars=$JARS\
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
--conf
spark.sql.catalog.demo.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
--conf spark.sql.catalog.demo.io-impl=org.apache.iceberg.aws.s3.S3FileIO
\
--conf spark.sql.catalog.demo.warehouse=s3://gluetestjonas/warehouse \
--conf spark.sql.catalog.demo.http-client.type=apache \
--conf spark.sql.catalog.demo.client.region=us-east-1
```
I could successfully spawn a spark shell and create/write data to tables in
AWS Glue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
