#general
@sangarshanan: Hi all ! just wanted to know if it is possible to create and update Iceberg tables without using spark for compute ? using just the java APIs
@dweeks: Yes, you can use the java APIs directly to update tables. You still need a catalog implementation for handling the commits, but everything should be possible through the APIs.
@dweeks: Could you share your use case? Just wondering if you're trying to simply append existing files or you're trying to integrate something else.
@sangarshanan: my usecase is to build a service that can perform CRUD on iceberg tables backed by Hive as catalog and provide an abstraction for users to run it without the overhead of spark
@sangarshanan: this is roughly what I came up with but I got hit with some bugs that I am trying to solve ```Configuration conf = new Configuration(); conf.set("hive.metastore.uris", "
@dweeks: That all looks right. From the table you should be able to use the APIs to manipulate the table. Those are the same APIs that Spark/Trino/Flink use, so you might be able to find examples of the API use there.
@dweeks: The tests are also a good place to look since they exercise all of these code paths.
@aman.rawat: Hi everyone, Is there any high level api/abstraction available to enable row level deletes as part of delete, update or merge operations in table format v2.
@russell.spitzer: Row level deletes are still not supported in the Spark api so there is no way to enabled them there
@aman.rawat: Thanks @russell.spitzer for the update.
@gsreeramkumar: Hi @russell.spitzer, can you pl. throw some light on: 1. what is the overall state of record level deletes? Is it supported in any other engine - or is it just implemented in the core api and is waiting for engine specific adaption? 2. Is there a work stream or ongoing work for implementing this in Spark - where we can come and contribute? Truly appreciate your inputs.
@mohamed.jouini.pro: Hi everyone. I tried to use Iceberg release 0.12.0 with Dynamodb catalog, and this is my SparkSession configuration: ```.set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") .set("spark.sql.catalog.iceberg_dynamo_poc", "org.apache.iceberg.spark.SparkCatalog") .set("spark.sql.catalog.iceberg_dynamo_poc.warehouse", "
@russell.spitzer: Looks like you are missing some things on your classpath, did you make sure to include iceberg-spark3-runtime and the aws SDK bundle and connection client?
@russell.spitzer:
@dweeks: I suspect that the issue is actually due to not having the aws-java-sdk-v2 in your classpath
@dweeks: Dynamo (and most of the native AWS support) uses sdk v2, which is not part of the bundle
@russell.spitzer: oh do we need that in the docs then?
@dweeks: Probably, though maybe we should just bundle it with `iceberg-aws`?
@russell.spitzer: As long as it isn't versioned with the other aws libs I think that's fine
@russell.spitzer: I assumed we didn't include the other libs so that patch releases would be easier to incorporate for end users
@blue: I think the docs cover adding it to the classpath, but it may not be called out very clearly
@russell.spitzer: This is what the doc's list in the startup instructions ```# add Iceberg dependency ICEBERG_VERSION=0.12.0 DEPENDENCIES="org.apache.iceberg:iceberg-spark3-runtime:$ICEBERG_VERSION" # add AWS dependnecy AWS_SDK_VERSION=2.15.40 AWS_MAVEN_GROUP=software.amazon.awssdk AWS_PACKAGES=( "bundle" "url-connection-client" ) for pkg in "${AWS_PACKAGES[@]}"; do DEPENDENCIES+=",$AWS_MAVEN_GROUP:$pkg:$AWS_SDK_VERSION" done```
@russell.spitzer: bundle, url-connection, spark3-runtime
@blue: Yeah, but does that require reading a shell script or is it stated that you need bundle and url-connection-client?
@russell.spitzer: oh sorry I thought you meant it listed aws-java-sdk-v2
@dweeks: no, the sdk v2 is `software.amazon.awssdk`
@russell.spitzer: I thought Daniel was noting that a 4th dependency was also required that is currently not a part of that script
@dweeks: Looks like all that's required is there, but this step may have been missed or it didn't get into the classpath correctly
@russell.spitzer: ah so it is just those 2 other libs, then yes I agree we should pull those out and not write this as a shell script
@dweeks: Yeah, even in the examples it requires a pull from maven, which is not ideal
@russell.spitzer: I still understand though if we want to support changing the patch version
@russell.spitzer: but in this example i would just enumerate everything explictly
@blue: For 3 packages, having a loop doesn't make a ton of sense to me
@russell.spitzer: the loop is just for 2 of them as well :slightly_smiling_face:
@blue: It's a good script for setting up EMR, but probably better to be simple for CLI use
@dweeks: I guess we should probably have @mohamed.jouini.pro verify that this is the issue though
@mohamed.jouini.pro: Please find the entire `pyspark` code ```from pyspark.sql import SparkSession conf = (sc.getConf() .set("spark.jars", "iceberg-spark-runtime-0.12.0.jar,bundle-2.15.40.jar,url-connection-client-2.15.40.jar") .set("spark.jars.packages", "org.apache.spark:spark-avro_2.12:4.0.0") .set("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") .set("spark.sql.catalog.iceberg_dynamo_poc", "org.apache.iceberg.spark.SparkCatalog") .set("spark.sql.catalog.iceberg_dynamo_poc.warehouse", "
@blue: @mohamed.jouini.pro, thanks for that. Have you downloaded the AWS SDK jars as well? I don't think that Spark will download them for you
@russell.spitzer: Yeah if you set it in "jars" it will not download dependencies of those libraries
@dweeks: Hmm, so `bundle-2.15.40.jar,url-connection-client-2.15.40.jar` these are the jars in question.
@russell.spitzer: "Packages" will download the library and all their dependencies
@dweeks: Since their alongside the iceberg runtime, maybe they were already downloaded?
@russell.spitzer: but if they have secondary deps those would be missing, I also think this is one of those ones that spark only warns on if the jars are missing
@mohamed.jouini.pro: jar packages are already in EMR local lib path ```[hadoop@emr~]$ ls -ltr /usr/share/aws/aws-java-sdk/iceberg* -rw-r--r-- 1 root root 19201102 Aug 10 15:49 /usr/share/aws/aws-java-sdk/iceberg-spark3-runtime-0.11.1.jar -rw-r--r-- 1 root root 25809685 Aug 26 13:51 /usr/share/aws/aws-java-sdk/iceberg-spark-runtime-0.12.0.jar [hadoop@emr ~]$ ls -ltr /usr/share/aws/aws-java-sdk/url-connection* -rw-r--r-- 1 root root 21027 Aug 10 15:49 /usr/share/aws/aws-java-sdk/url-connection-client-2.15.40.jar [hadoop@emr ~]$ ls -ltr /usr/share/aws/aws-java-sdk/bundle* -rw-r--r-- 1 root root 257939967 Aug 10 15:49 /usr/share/aws/aws-java-sdk/bundle-2.15.40.jar```
@mohamed.jouini.pro: I can create a table using aws glue catalog, but the lem seems to be related to `DynamoDBCatalog`
@russell.spitzer: Neither of those libs have runtime deps according to maven, although it is amazing to me that bundle is 246 MB!
@mohamed.jouini.pro:
@mohamed.jouini.pro: When reading the error message it looks like Iceberg can't convert table-name to `String` or something like this, In the documentation Iceberg, can use a default dynamodbcatalog `iceberg` if it's not set, but I still get the same output
@russell.spitzer: The error message here ```java.lang.NoSuchMethodError: org.apache.iceberg.aws.AwsProperties.dynamoDbTableName()Ljava/lang/String```
@russell.spitzer: Says it's looking for a method called dynamoDbTableName() which returns a String and is not finding it on the classpath
@russell.spitzer: Specifically this one
@russell.spitzer: Is EMR going to put that older "iceberg-spark3-runtime" jar on the classpath even if you don't add it to jars? Because that could cause the issue since the method doesn't exist in 11.1
@russell.spitzer: ```-rw-r--r-- 1 root root 19201102 Aug 10 15:49 /usr/share/aws/aws-java-sdk/iceberg-spark3-runtime-0.11.1.jar << That one```
@russell.spitzer: Also shouldn't you be using Spark3 0.12 runtime? not Spark-runtime?
@mohamed.jouini.pro: @russell.spitzer, the same errors when using spark3 runtime
@russell.spitzer: yes but are you sure the 0.11.1 jar isn't on the classpath?
@russell.spitzer: That would cause the error
@mohamed.jouini.pro: Ahh, let me remove the old one from aws classpath
@mohamed.jouini.pro: Thx @russell.spitzer, After removing the old runtime jars, I see that Iceberg can create DynamoDB table, but I can't create a table ```spark.sql ("CREATE TABLE iceberg_dynamo_poc.iceberg_dynamo_poc.dynamo1 ( \ id bigint, \ pathId string, \ ) \ USING iceberg " )``` ```org.apache.iceberg.exceptions.NoSuchNamespaceException: Cannot find default warehouse location: namespace iceberg_dynamo_poc does not exist at org.apache.iceberg.aws.dynamodb.DynamoDbCatalog.defaultWarehouseLocation(DynamoDbCatalog.java:158) at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:211) at org.apache.iceberg.CachingCatalog$CachingTableBuilder.lambda$create$0(CachingCatalog.java:212) at org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344) at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)```
@russell.spitzer: ?
@mohamed.jouini.pro: Iceberg has to create the namespace `iceberg_dynamo_poc` if doesn't exist, right ?
@russell.spitzer: You would need to make it if it doesn't exist
@russell.spitzer: looking breifly at the docs, it seems like they don't use a namespace in their table references
@russell.spitzer: just catalog.tableName
@mohamed.jouini.pro: ```spark.sql ("CREATE TABLE iceberg_dynamo_poc.dynamo1 ( \ id bigint, \ pathId string, \ ) \ USING iceberg " ) Py4JJavaError: An error occurred while calling o554.sql. : org.apache.iceberg.exceptions.ValidationException: Table namespace must not be empty: dynamo1```
@russell.spitzer: i believe that is because you didn't set a location? @jackye ^
@mohamed.jouini.pro: I think it should create a namespace, please
@mohamed.jouini.pro: But it should use the `catalog.warehouse` as the default table location
@russell.spitzer: I don't know how they set this up, but in most systems I would imagine you would have to first CREATE DATABASE before putting a table in it, but I don't know how the DynamoCatalog was configured
@dweeks: @russell.spitzer is correct
@jackye: reading the threads now
@dweeks: The namespace needs to exist via a `create database <catalog>.<database>`
@jackye: +1 for what Daniel says, a database needs to be created before table creation in Spark.
@dweeks: I think there is some strangeness in spark in terms of catalogs and databases as well.
@jackye: for ```AWS_PACKAGES=( "bundle" "url-connection-client" )``` the intention was that bundle includes all AWS service packages and can work if you just want to test running on EMR with a bootstrap script, but you can change `bundle` to the list of aws services you use for a script good for production use.
@dweeks: For example, I believe the `use` command can be in context of catalog or database
@mohamed.jouini.pro: Thx, it works after creating the database, it's not the same behavior when working with AWS Glue, I think that namespace is created if doesn't exist Thank you @russell.spitzer, @dweeks @jackye @blue
@jackye: I think you might already have a database of that name in Glue in the past, that’s why you did not need to create it. Otherwise the behavior should be consistent across all Iceberg catalog implementations
@mohamed.jouini.pro: I will test it again with glue
@gsreeramkumar: Hello folks! Is there a way to specify while writing a Row to an Iceberg Table from Spark - *that a specific column is non-existent for that Row*? My question is NOT about `null` - but about `non-existent`? & if yes, is there a way to consume this in Spark? Truly appreciate any help!
