KodaiD commented on PR #16512:
URL: https://github.com/apache/iceberg/pull/16512#issuecomment-4509157317
$ spark-sql --packages
org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.2,org.xerial:sqlite-jdbc:3.53.1.0\
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=jdbc \
--conf spark.sql.catalog.local.jdbc-driver=org.sqlite.JDBC \
--conf
spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.defaultCatalog=local
WARNING: Using incubator modules: jdk.incubator.vector
:: loading settings :: url =
jar:file:/opt/homebrew/Cellar/apache-spark/4.1.1/libexec/jars/ivy-2.5.3.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/pc-0063_kodai.doki/.ivy2.5.2/cache
The jars for the packages stored in: /Users/pc-0063_kodai.doki/.ivy2.5.2/jars
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13 added as a dependency
org.xerial#sqlite-jdbc added as a dependency
:: resolving dependencies ::
org.apache.spark#spark-submit-parent-4aeb1fa3-1a75-45ea-a606-e9b9163a014e;1.0
confs: [default]
found org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.10.2 in
central
found org.xerial#sqlite-jdbc;3.53.1.0 in central
:: resolution report :: resolve 64ms :: artifacts dl 4ms
:: modules in use:
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.10.2 from
central in [default]
org.xerial#sqlite-jdbc;3.53.1.0 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 2 | 0 |
---------------------------------------------------------------------
:: retrieving ::
org.apache.spark#spark-submit-parent-4aeb1fa3-1a75-45ea-a606-e9b9163a014e
confs: [default]
0 artifacts copied, 2 already retrieved (0kB/4ms)
26/05/21 17:54:02 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
26/05/21 17:54:05 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 2.3.0
26/05/21 17:54:05 WARN ObjectStore: setMetaStoreSchemaVersion called but
recording version is disabled: version = 2.3.0, comment = Set by MetaStore
[email protected]
Spark Web UI available at http://192.168.10.118:4040/
Spark master: local[*], Application Id: local-1779353644121
26/05/21 17:54:06 WARN JdbcCatalog: JDBC catalog is initialized without view
support. To auto-migrate the database's schema and enable view support, set
jdbc.schema-version=V1
spark-sql ()> CREATE DATABASE IF NOT EXISTS local.db;
Time taken: 0.476 seconds
spark-sql ()> CREATE TABLE local.db.sample (id int, name string);
Time taken: 0.241 seconds
spark-sql ()> %
```
```
$ tree warehouse
warehouse
└── db
└── sample
└── metadata
└── 00000-62b8d94c-c1ca-4485-a949-407a547ae344.metadata.json
4 directories, 1 file
$
```
```
$ spark-sql --packages
org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.2,org.apache.iceberg:iceberg-aws-bundle:1.10.2\
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf
spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.rest.type=rest \
--conf spark.sql.catalog.rest.uri=http://localhost:8181/ \
--conf spark.sql.catalog.rest.warehouse=s3://warehouse/ \
--conf spark.sql.catalog.rest.io-impl=org.apache.iceberg.aws.s3.S3FileIO
\
--conf spark.sql.catalog.rest.s3.endpoint=http://localhost:9000/ \
--conf spark.sql.catalog.rest.s3.path-style-access=true \
--conf spark.sql.catalog.rest.s3.access-key-id=admin \
--conf spark.sql.catalog.rest.s3.secret-access-key=password \
--conf spark.sql.catalog.rest.client.region=us-east-1 \
--conf spark.sql.defaultCatalog=rest
WARNING: Using incubator modules: jdk.incubator.vector
:: loading settings :: url =
jar:file:/opt/homebrew/Cellar/apache-spark/4.1.1/libexec/jars/ivy-2.5.3.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/pc-0063_kodai.doki/.ivy2.5.2/cache
The jars for the packages stored in: /Users/pc-0063_kodai.doki/.ivy2.5.2/jars
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13 added as a dependency
org.apache.iceberg#iceberg-aws-bundle added as a dependency
:: resolving dependencies ::
org.apache.spark#spark-submit-parent-233d81ae-ca4a-4d16-b06b-b6f89a3b627c;1.0
confs: [default]
found org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.10.2 in
central
found org.apache.iceberg#iceberg-aws-bundle;1.10.2 in central
:: resolution report :: resolve 58ms :: artifacts dl 5ms
:: modules in use:
org.apache.iceberg#iceberg-aws-bundle;1.10.2 from central in
[default]
org.apache.iceberg#iceberg-spark-runtime-4.0_2.13;1.10.2 from
central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 2 | 0 |
---------------------------------------------------------------------
:: retrieving ::
org.apache.spark#spark-submit-parent-233d81ae-ca4a-4d16-b06b-b6f89a3b627c
confs: [default]
0 artifacts copied, 2 already retrieved (0kB/4ms)
26/05/21 17:55:32 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile:
org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
26/05/21 17:55:35 WARN ObjectStore: Version information not found in
metastore. hive.metastore.schema.verification is not enabled so recording the
schema version 2.3.0
26/05/21 17:55:35 WARN ObjectStore: setMetaStoreSchemaVersion called but
recording version is disabled: version = 2.3.0, comment = Set by MetaStore
[email protected]
Spark Web UI available at http://192.168.10.118:4040/
Spark master: local[*], Application Id: local-1779353733803
spark-sql ()> CREATE DATABASE IF NOT EXISTS rest.db;
Time taken: 0.607 seconds
spark-sql ()> CREATE TABLE rest.db.sample (id int, name string);
Time taken: 0.574 seconds
spark-sql ()>
```
```
$ aws s3 ls s3://warehouse --recursive
2026-05-21 17:55:51 742
db/sample/metadata/00000-120a8a1c-bc36-4db0-924f-fcae53b2fdd6.metadata.json
$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
