soumilshah1995 commented on issue #11968:
URL: https://github.com/apache/iceberg/issues/11968#issuecomment-3289461163
HI @amogh-jahagirdar
im using spark 4.0.0 I dont see puffin files as well
```
from pyspark.sql import SparkSession
import os
import sys
from pyspark.sql.functions import col, parse_json, lit
os.environ["JAVA_HOME"] =
"/opt/homebrew/Cellar/openjdk@17/17.0.14/libexec/openjdk.jdk/Contents/Home"
SUBMIT_ARGS = (
"--packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0 "
"--repositories https://repo1.maven.org/maven2 "
"pyspark-shell"
)
os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS
os.environ["PYSPARK_PYTHON"] = sys.executable
local_warehouse_path = "/Users/soumilshah/Desktop/warehouse"
spark = (
SparkSession.builder
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.dev", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.dev.type", "hadoop")
.config("spark.sql.catalog.dev.warehouse", local_warehouse_path)
.config("spark.sql.catalog.dev.format-version", "3")
.config("spark.sql.join.preferSortMergeJoin", "false")
.getOrCreate()
)
spark.sql("""
CREATE OR REPLACE TABLE dev.defaulttest_deletion_vectors (
dt DATE,
number INTEGER,
letter STRING
)
USING iceberg
TBLPROPERTIES (
'write.delete.mode'='merge-on-read',
'write.update.mode'='merge-on-read',
'write.merge.mode'='merge-on-read',
'format-version'='3'
)
""")
# Insert sample rows
spark.sql("""
INSERT INTO dev.defaulttest_deletion_vectors VALUES
(CAST('2023-03-01' AS DATE), 1, 'a'),
(CAST('2023-03-02' AS DATE), 2, 'b'),
(CAST('2023-03-03' AS DATE), 3, 'c'),
(CAST('2023-03-04' AS DATE), 4, 'd'),
(CAST('2023-03-05' AS DATE), 5, 'e'),
(CAST('2023-03-06' AS DATE), 6, 'f'),
(CAST('2023-03-07' AS DATE), 7, 'g'),
(CAST('2023-03-08' AS DATE), 8, 'h'),
(CAST('2023-03-09' AS DATE), 9, 'i'),
(CAST('2023-03-10' AS DATE), 10, 'j'),
(CAST('2023-03-11' AS DATE), 11, 'k'),
(CAST('2023-03-12' AS DATE), 12, 'l')
""")
spark.sql("""
DELETE FROM dev.defaulttest_deletion_vectors WHERE number = 5;
""")
<img width="895" height="572" alt="Image"
src="https://github.com/user-attachments/assets/ce359f37-2f6e-4314-825b-6d1ce42aa7fc";
/>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
