Fokko commented on issue #7226:
URL: https://github.com/apache/iceberg/issues/7226#issuecomment-1596258379
Able to reproduce this using:
```sh
rm -rf /tmp/warehouse
rm -rf /tmp/spark-checkpoint
mkdir -p /tmp/warehouse
mkdir -p /tmp/spark-checkpoint
./bin/spark-shell \
--packages org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.0.0 \
--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
--conf spark.sql.catalog.demo=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.demo.type=hadoop \
--conf spark.sql.catalog.demo.warehouse=/tmp/warehouse \
--conf spark.sql.warehouse.dir=/tmp/warehouse \
--conf spark.sql.defaultCatalog=demo
```
```scala
spark.sql("create table test_iceberg_table(ts timestamp, a string, b double)
using iceberg partitioned by (days(ts))")
import java.nio.file.Files
import java.sql.Timestamp
import org.apache.spark.sql.execution.streaming.MemoryStream
import org.apache.spark.sql.streaming.Trigger
import org.apache.spark.sql.{Encoders, SparkSession}
case class Bla(ts: Timestamp, a: String, b: Double)
implicit val sqlContext = spark.sqlContext
implicit val encoder = Encoders.product[Bla]
val memStream = MemoryStream[Bla]
val now = System.currentTimeMillis()
val day = 86400000
memStream.addData(List(
Bla(new Timestamp(now), "test", 12.34),
Bla(new Timestamp(now - 1 * day), "test 1d", 33.34),
Bla(new Timestamp(now - 3 * day), "test 3d", 44.34),
Bla(new Timestamp(now - 2 * day), "test 2d", 55.34),
))
memStream.toDF()
.writeStream
.format("iceberg")
.outputMode("append")
.option("path", "/tmp/warehouse/test_iceberg_table")
.option("checkpointLocation", "/tmp/spark-checkpoint")
.option("fanout-enabled", true)
.trigger(Trigger.Once())
.start()
.awaitTermination()
```
Any idea @aokolnychyi? Not really looking forward to bisecting between
0.13.1 and 1.0.0 :3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
