lucasmo commented on issue #466:
URL:
https://github.com/apache/incubator-xtable/issues/466#issuecomment-2204025995
So this gets me one step closer, but still not working.
The initial load works (I'll add the log at the end). The followup
incremental load fails like this:
```
lfm@ubuntu:~/incubator-xtable$ java -jar
xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar -d
config.yaml
WARNING: Runtime environment or build system does not support multi-release
JARs. This will impact location-based features.
2024-07-02 17:47:27 INFO org.apache.xtable.utilities.RunSync:148 - Running
sync for basePath s3://hidden-s3-bucket/hidden-prefix/ for following table
formats [DELTA]
2024-07-02 17:47:27 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading
HoodieTableMetaClient from s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:47:27 WARN org.apache.hadoop.util.NativeCodeLoader:60 -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2024-07-02 17:47:27 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:136
- Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2024-07-02 17:47:27 WARN org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 -
Directly referencing AWS SDK V1 credential provider
com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential
providers will be removed once S3A is upgraded to SDK V2
2024-07-02 17:47:28 INFO org.apache.hudi.common.table.HoodieTableConfig:276
- Loading table properties from
s3://hidden-s3-bucket/hidden-prefix/.hoodie/hoodie.properties
2024-07-02 17:47:28 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:47:28 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit
timeline for s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:47:28 INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded
instants upto :
Option{val=[20240702171813028__clean__COMPLETED__20240702171815000]}
2024-07-02 17:47:28 INFO
org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View
Manager with storage type :MEMORY
2024-07-02 17:47:28 INFO
org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating
in-memory based Table View
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform
(file:/home/lfm/incubator-xtable/xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar)
to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of
org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-02 17:47:31 INFO
org.apache.spark.sql.delta.storage.DelegatingLogStore:60 - LogStore
`LogStoreAdapter(io.delta.storage.S3SingleDriverLogStore)` is used for scheme
`s3`
2024-07-02 17:47:31 INFO org.apache.spark.sql.delta.DeltaLog:60 - Loading
version 0.
2024-07-02 17:47:32 INFO org.apache.spark.sql.delta.DeltaLogFileIndex:60 -
Created DeltaLogFileIndex(JSON, numFilesInSegment: 1, totalFileSize: 75250570)
2024-07-02 17:47:35 INFO org.apache.spark.sql.delta.Snapshot:60 -
[tableId=4da47f37-d68a-411a-8124-5dfed3c314fd] Created snapshot
Snapshot(path=s3://hidden-s3-bucket/hidden-prefix/_delta_log, version=0,
metadata=Metadata(e65c4be0-428e-4ecd-8fb4-360a8fe22dda,hidden_table_name,,Format(parquet,Map()),{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"hidden_unique_id","type":"long","nullable":true,"metadata":{}},{"name":"op_date","type":"date","nullable":true,"metadata":{}},{"name":"previous_op_dates","type":{"type":"array","elementType":"date","containsNull":true},"nullable":true,"metadata":{}},{"name":"col101","type"
:"timestamp","nullable":true,"metadata":{}},{"name":"col102","type":"string","nullable":true,"metadata":{}},{"name":"col103","type":"boolean","nullable":true,"metadata":{}},{"name":"col104","type":"timestamp","nullable":true,"metadata":{}},{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col107","type":"string","nullable":true,"metadata":{}},{"name":"col108","type":"boolean","nullable":true,"metadata":{}},{"name":"noun_col105","type":"string","nullable":true,"metadata":{}},{"name":"col110","type":"boolean","nullable":true,"metadata":{}},{"name":"col111","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col113","type":"string","nullable":true,"metadata":{}}]},"containsNull":true},"nullable":true,"metadata":{}},{"name":"col112","type":"string","nulla
ble":true,"metadata":{}},{"name":"col113","type":"string","nullable":true,"metadata":{}},{"name":"col114","type":"long","nullable":true,"metadata":{}},{"name":"col115","type":"string","nullable":true,"metadata":{}},{"name":"col116","type":"string","nullable":true,"metadata":{}},{"name":"col117","type":"string","nullable":true,"metadata":{}},{"name":"col118","type":"string","nullable":true,"metadata":{}},{"name":"col119","type":"string","nullable":true,"metadata":{}},{"name":"col120","type":"string","nullable":true,"metadata":{}},{"name":"col121","type":"string","nullable":true,"metadata":{}},{"name":"col122","type":"string","nullable":true,"metadata":{}},{"name":"col123","type":"timestamp","nullable":true,"metadata":{}},{"name":"col124","type":"timestamp","nullable":true,"metadata":{}},{"name":"col125","type":"timestamp","nullable":true,"metadata":{}},{"name":"col126","type":"timestamp","nullable":true,"metadata":{}},{"name":"col127","type":"timestamp","nullable":true,"metadata":{}}
,{"name":"col128","type":"timestamp","nullable":true,"metadata":{}},{"name":"col129","type":"timestamp","nullable":true,"metadata":{}},{"name":"col130","type":"timestamp","nullable":true,"metadata":{}},{"name":"col131","type":"timestamp","nullable":true,"metadata":{}},{"name":"col132","type":"timestamp","nullable":true,"metadata":{}},{"name":"col133","type":"timestamp","nullable":true,"metadata":{}},{"name":"col134","type":"timestamp","nullable":true,"metadata":{}},{"name":"col135","type":"timestamp","nullable":true,"metadata":{}},{"name":"col136","type":"timestamp","nullable":true,"metadata":{}},{"name":"col137","type":"timestamp","nullable":true,"metadata":{}},{"name":"col138","type":"timestamp","nullable":true,"metadata":{}},{"name":"col139","type":"timestamp","nullable":true,"metadata":{}},{"name":"col140","type":"timestamp","nullable":true,"metadata":{}},{"name":"col141","type":"timestamp","nullable":true,"metadata":{}},{"name":"col142","type":"timestamp","nullable":true,"metad
ata":{}},{"name":"col143","type":"timestamp","nullable":true,"metadata":{}},{"name":"col144","type":"timestamp","nullable":true,"metadata":{}},{"name":"col145","type":"timestamp","nullable":true,"metadata":{}},{"name":"col146","type":"timestamp","nullable":true,"metadata":{}},{"name":"col147","type":"timestamp","nullable":true,"metadata":{}},{"name":"col148","type":"timestamp","nullable":true,"metadata":{}},{"name":"col149","type":"timestamp","nullable":true,"metadata":{}},{"name":"col150","type":"timestamp","nullable":true,"metadata":{}},{"name":"col151","type":"string","nullable":true,"metadata":{}},{"name":"col152","type":"string","nullable":true,"metadata":{}},{"name":"col153","type":"string","nullable":true,"metadata":{}},{"name":"col154","type":"string","nullable":true,"metadata":{}},{"name":"col155","type":"string","nullable":true,"metadata":{}},{"name":"col156","type":"string","nullable":true,"metadata":{}},{"name":"col157","type":"string","nullable":true,"metadata":{}},{"na
me":"col158","type":"string","nullable":true,"metadata":{}},{"name":"col159","type":"string","nullable":true,"metadata":{}},{"name":"col160","type":"string","nullable":true,"metadata":{}},{"name":"col161","type":"string","nullable":true,"metadata":{}},{"name":"op_date_utc","type":"date","nullable":true,"metadata":{}},{"name":"col163","type":"timestamp","nullable":true,"metadata":{}},{"name":"col164","type":"timestamp","nullable":true,"metadata":{}},{"name":"col165","type":"timestamp","nullable":true,"metadata":{}},{"name":"col166","type":"timestamp","nullable":true,"metadata":{}},{"name":"col167","type":"timestamp","nullable":true,"metadata":{}},{"name":"col168","type":"timestamp","nullable":true,"metadata":{}},{"name":"col169","type":"timestamp","nullable":true,"metadata":{}},{"name":"col170","type":"timestamp","nullable":true,"metadata":{}},{"name":"col171","type":"timestamp","nullable":true,"metadata":{}},{"name":"col172","type":"timestamp","nullable":true,"metadata":{}},{"name":
"col173","type":"timestamp","nullable":true,"metadata":{}},{"name":"col174","type":"timestamp","nullable":true,"metadata":{}},{"name":"col175","type":"timestamp","nullable":true,"metadata":{}},{"name":"col176","type":"timestamp","nullable":true,"metadata":{}},{"name":"col177","type":"timestamp","nullable":true,"metadata":{}},{"name":"col178","type":"timestamp","nullable":true,"metadata":{}},{"name":"col179","type":"timestamp","nullable":true,"metadata":{}},{"name":"col180","type":"timestamp","nullable":true,"metadata":{}},{"name":"col181","type":"timestamp","nullable":true,"metadata":{}},{"name":"col182","type":"timestamp","nullable":true,"metadata":{}},{"name":"col183","type":"boolean","nullable":true,"metadata":{}},{"name":"col184","type":"date","nullable":true,"metadata":{}},{"name":"scheduled_hidden_unique_ids","type":{"type":"array","elementType":"long","containsNull":true},"nullable":true,"metadata":{}},{"name":"col186","type":{"type":"array","elementType":"string","containsNu
ll":true},"nullable":true,"metadata":{}},{"name":"col187","type":"long","nullable":true,"metadata":{}},{"name":"col188","type":"long","nullable":true,"metadata":{}},{"name":"col189","type":"long","nullable":true,"metadata":{}},{"name":"col190","type":"long","nullable":true,"metadata":{}},{"name":"col191","type":"long","nullable":true,"metadata":{}},{"name":"col192","type":"long","nullable":true,"metadata":{}},{"name":"col193","type":"timestamp","nullable":true,"metadata":{}},{"name":"col194","type":"timestamp","nullable":true,"metadata":{}},{"name":"col195","type":"timestamp","nullable":true,"metadata":{}},{"name":"col196","type":"timestamp","nullable":true,"metadata":{}},{"name":"col197","type":"timestamp","nullable":true,"metadata":{}},{"name":"col198","type":"timestamp","nullable":true,"metadata":{}},{"name":"col199","type":"timestamp","nullable":true,"metadata":{}},{"name":"col200","type":"timestamp","nullable":true,"metadata":{}},{"name":"col201","type":"timestamp","nullable":t
rue,"metadata":{}},{"name":"col202","type":"timestamp","nullable":true,"metadata":{}},{"name":"col203","type":"timestamp","nullable":true,"metadata":{}},{"name":"col204","type":"timestamp","nullable":true,"metadata":{}},{"name":"col205","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col206","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col184_local","type":"date","nullable":true,"metadata":{}},{"name":"col206_internal","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col209","type":"string","nullable":true,"metadata":{}},{"name":"col210","type":{"type":"array","elementType":"integer","containsNull":true},"nullable":true,"metadata":{}},{"name":"col211","type":{"type":"array","elementType":"timestamp","containsNull":true},"nullable":true,"metadata":{}},{"name":"col212","type":"string","nullable":true,"metadat
a":{}},{"name":"col213","type":"integer","nullable":true,"metadata":{}},{"name":"event_millis","type":"long","nullable":true,"metadata":{}},{"name":"_hoodie_is_deleted","type":"boolean","nullable":true,"metadata":{}},{"name":"xtable_partition_col_DAY_op_date","type":"date","nullable":true,"metadata":{"delta.generationExpression":"CAST(op_date
as DATE)"}}]},List(xtable_partition_col_DAY_op_date),Map(XTABLE_METADATA ->
{"lastInstantSynced":"2024-07-02T17:18:13.028Z","instantsToConsiderForNextSync":[],"version":0},
delta.logRetentionDuration -> interval 168 hours),Some(1719940693028)),
logSegment=LogSegment(s3://hidden-s3-bucket/hidden-prefix/_delta_log,0,WrappedArray(S3AFileStatus{path=s3://hidden-s3-bucket/hidden-prefix/_delta_log/00000000000000000000.json;
isDirectory=false; length=75250570; replication=1; blocksize=33554432;
modification_time=1719942187000; access_time=0; owner=lfm; group=lfm;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
isErasureCoded=fal
se} isEmptyDirectory=FALSE eTag=414a8c48671a5c1a268e391487db2926-2
versionId=null),None,1719942187000), checksumOpt=None)
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:95)
at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:72)
at
org.apache.avro.specific.SpecificData.lambda$getClass$2(SpecificData.java:259)
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at org.apache.avro.util.MapUtil.computeIfAbsent(MapUtil.java:42)
at org.apache.avro.specific.SpecificData.getClass(SpecificData.java:257)
at
org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:508)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:237)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
at
org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:355)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:186)
at
org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:248)
at
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:180)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:161)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:154)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:263)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:248)
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:209)
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeHoodieCleanMetadata(TimelineMetadataUtils.java:173)
at
org.apache.xtable.hudi.HudiConversionSource.isAffectedByCleanupProcess(HudiConversionSource.java:164)
at
org.apache.xtable.hudi.HudiConversionSource.isIncrementalSyncSafeFrom(HudiConversionSource.java:148)
at
org.apache.xtable.conversion.ConversionController.isIncrementalSyncSufficient(ConversionController.java:244)
at
org.apache.xtable.conversion.ConversionController.lambda$getFormatsToSyncIncrementally$4(ConversionController.java:175)
at
java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176)
at
java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1764)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.xtable.conversion.ConversionController.getFormatsToSyncIncrementally(ConversionController.java:178)
at
org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:109)
at org.apache.xtable.utilities.RunSync.main(RunSync.java:169)
Caused by: java.lang.IllegalStateException: Recursive update
at
java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1760)
at org.apache.avro.util.MapUtil.computeIfAbsent(MapUtil.java:42)
at org.apache.avro.specific.SpecificData.getClass(SpecificData.java:257)
at
org.apache.avro.specific.SpecificData.getForSchema(SpecificData.java:164)
at
org.apache.avro.specific.SpecificDatumWriter.<init>(SpecificDatumWriter.java:47)
at
org.apache.hudi.avro.model.HoodieCleanPartitionMetadata.<clinit>(HoodieCleanPartitionMetadata.java:532)
... 38 more
```
The first load succeeds like this:
```
lfm@ubuntu:~/incubator-xtable$ java -jar
xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar -d
config.yaml
WARNING: Runtime environment or build system does not support multi-release
JARs. This will impact location-based features.
2024-07-02 17:40:30 INFO org.apache.xtable.utilities.RunSync:148 - Running
sync for basePath s3://hidden-s3-bucket/hidden-prefix/ for following table
formats [DELTA]
2024-07-02 17:40:30 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading
HoodieTableMetaClient from s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:40:30 WARN org.apache.hadoop.util.NativeCodeLoader:60 -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2024-07-02 17:40:30 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:136
- Cannot locate configuration: tried
hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2024-07-02 17:40:31 WARN org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 -
Directly referencing AWS SDK V1 credential provider
com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential
providers will be removed once S3A is upgraded to SDK V2
2024-07-02 17:40:31 INFO org.apache.hudi.common.table.HoodieTableConfig:276
- Loading table properties from
s3://hidden-s3-bucket/hidden-prefix/.hoodie/hoodie.properties
2024-07-02 17:40:31 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table
of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from
s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:40:31 INFO
org.apache.hudi.common.table.HoodieTableMetaClient:155 - Loading Active commit
timeline for s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:40:32 INFO
org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded
instants upto :
Option{val=[20240702171813028__clean__COMPLETED__20240702171815000]}
2024-07-02 17:40:32 INFO
org.apache.hudi.common.table.view.FileSystemViewManager:243 - Creating View
Manager with storage type :MEMORY
2024-07-02 17:40:32 INFO
org.apache.hudi.common.table.view.FileSystemViewManager:255 - Creating
in-memory based Table View
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform
(file:/home/lfm/incubator-xtable/xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar)
to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of
org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-07-02 17:40:33 INFO
org.apache.spark.sql.delta.storage.DelegatingLogStore:60 - LogStore
`LogStoreAdapter(io.delta.storage.S3SingleDriverLogStore)` is used for scheme
`s3`
2024-07-02 17:40:34 INFO org.apache.spark.sql.delta.DeltaLog:60 - Creating
initial snapshot without metadata, because the directory is empty
2024-07-02 17:40:34 INFO org.apache.spark.sql.delta.InitialSnapshot:60 -
[tableId=697e4b0d-05a7-45d0-9038-7a0bebf69cba] Created snapshot
InitialSnapshot(path=s3://hidden-s3-bucket/hidden-prefix/_delta_log,
version=-1,
metadata=Metadata(91d449f4-8540-4208-a0be-b2c134bde849,null,null,Format(parquet,Map()),null,List(),Map(),Some(1719942034877)),
logSegment=LogSegment(s3://hidden-s3-bucket/hidden-prefix/_delta_log,-1,List(),None,-1),
checksumOpt=None)
2024-07-02 17:40:34 INFO
org.apache.xtable.conversion.ConversionController:240 - No previous
InternalTable sync for target. Falling back to snapshot sync.
2024-07-02 17:40:35 INFO
org.apache.hudi.common.table.TableSchemaResolver:317 - Reading schema from
s3://hidden-s3-bucket/hidden-prefix/op_date=2024-06-29/c061e248-2981-463e-8055-b8e4d59c2a24-0_2-22-137_20240702171726140.parquet
2024-07-02 17:40:35 INFO org.apache.hudi.common.table.HoodieTableConfig:276
- Loading table properties from
s3://hidden-s3-bucket/hidden-prefix/.hoodie/hoodie.properties
2024-07-02 17:40:49 INFO
org.apache.hudi.common.table.view.FileSystemViewManager:165 - Creating InMemory
based view for basePath s3://hidden-s3-bucket/hidden-prefix
2024-07-02 17:40:49 INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView:259 - Took 1 ms
to read 0 instants, 0 replaced file groups
2024-07-02 17:40:49 INFO org.apache.hudi.common.util.ClusteringUtils:147 -
Found 0 files in pending clustering operations
2024-07-02 17:40:49 INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building
file system view for partition (op_date=2021-10-29)
2024-07-02 17:40:49 INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building
file system view for partition (op_date=2019-08-07)
2024-07-02 17:40:49 INFO
org.apache.hudi.common.table.view.AbstractTableFileSystemView:429 - Building
file system view for partition (op_date=2023-08-12)
[ ... several thousand partitions ... ]
2024-07-02 17:42:58 INFO org.apache.spark.sql.delta.DeltaLog:60 - No delta
log found for the Delta table at s3://hidden-s3-bucket/hidden-prefix/_delta_log
2024-07-02 17:42:58 INFO org.apache.spark.sql.delta.InitialSnapshot:60 -
[tableId=91d449f4-8540-4208-a0be-b2c134bde849] Created snapshot
InitialSnapshot(path=s3://hidden-s3-bucket/hidden-prefix/_delta_log,
version=-1,
metadata=Metadata(e65c4be0-428e-4ecd-8fb4-360a8fe22dda,null,null,Format(parquet,Map()),null,List(),Map(),Some(1719942178350)),
logSegment=LogSegment(s3://hidden-s3-bucket/hidden-prefix/_delta_log,-1,List(),None,-1),
checksumOpt=None)
2024-07-02 17:43:05 INFO
org.apache.spark.sql.delta.OptimisticTransaction:60 -
[tableId=e65c4be0,txnId=b9564248] Updated metadata from - to
Metadata(e65c4be0-428e-4ecd-8fb4-360a8fe22dda,hidden_table_name,,Format(parquet,Map()),{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"hidden_unique_id","type":"long","nullable":true,"metadata":{}},{"name":"op_date","type":"date","nullable":true,"metadata":{}},{"name":"previous_op_dates","type":{"type":"array","elementType":"date","containsNull":true},"nullable":true,"metadata":{}},{"name":"col101","type":"timestamp","nullable":true,"metadata":{}},{"name":"col102","type":"str
ing","nullable":true,"metadata":{}},{"name":"col103","type":"boolean","nullable":true,"metadata":{}},{"name":"col104","type":"timestamp","nullable":true,"metadata":{}},{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col107","type":"string","nullable":true,"metadata":{}},{"name":"col108","type":"boolean","nullable":true,"metadata":{}},{"name":"noun_col105","type":"string","nullable":true,"metadata":{}},{"name":"col110","type":"boolean","nullable":true,"metadata":{}},{"name":"col111","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col113","type":"string","nullable":true,"metadata":{}}]},"containsNull":true},"nullable":true,"metadata":{}},{"name":"col112","type":"string","nullable":true,"metadata":{}},{"name":"col113","type":"string","nullable":tru
e,"metadata":{}},{"name":"col114","type":"long","nullable":true,"metadata":{}},{"name":"col115","type":"string","nullable":true,"metadata":{}},{"name":"col116","type":"string","nullable":true,"metadata":{}},{"name":"col117","type":"string","nullable":true,"metadata":{}},{"name":"col118","type":"string","nullable":true,"metadata":{}},{"name":"col119","type":"string","nullable":true,"metadata":{}},{"name":"col120","type":"string","nullable":true,"metadata":{}},{"name":"col121","type":"string","nullable":true,"metadata":{}},{"name":"col122","type":"string","nullable":true,"metadata":{}},{"name":"col123","type":"timestamp","nullable":true,"metadata":{}},{"name":"col124","type":"timestamp","nullable":true,"metadata":{}},{"name":"col125","type":"timestamp","nullable":true,"metadata":{}},{"name":"col126","type":"timestamp","nullable":true,"metadata":{}},{"name":"col127","type":"timestamp","nullable":true,"metadata":{}},{"name":"col128","type":"timestamp","nullable":true,"metadata":{}},{"na
me":"col129","type":"timestamp","nullable":true,"metadata":{}},{"name":"col130","type":"timestamp","nullable":true,"metadata":{}},{"name":"col131","type":"timestamp","nullable":true,"metadata":{}},{"name":"col132","type":"timestamp","nullable":true,"metadata":{}},{"name":"col133","type":"timestamp","nullable":true,"metadata":{}},{"name":"col134","type":"timestamp","nullable":true,"metadata":{}},{"name":"col135","type":"timestamp","nullable":true,"metadata":{}},{"name":"col136","type":"timestamp","nullable":true,"metadata":{}},{"name":"col137","type":"timestamp","nullable":true,"metadata":{}},{"name":"col138","type":"timestamp","nullable":true,"metadata":{}},{"name":"col139","type":"timestamp","nullable":true,"metadata":{}},{"name":"col140","type":"timestamp","nullable":true,"metadata":{}},{"name":"col141","type":"timestamp","nullable":true,"metadata":{}},{"name":"col142","type":"timestamp","nullable":true,"metadata":{}},{"name":"col143","type":"timestamp","nullable":true,"metadata":
{}},{"name":"col144","type":"timestamp","nullable":true,"metadata":{}},{"name":"col145","type":"timestamp","nullable":true,"metadata":{}},{"name":"col146","type":"timestamp","nullable":true,"metadata":{}},{"name":"col147","type":"timestamp","nullable":true,"metadata":{}},{"name":"col148","type":"timestamp","nullable":true,"metadata":{}},{"name":"col149","type":"timestamp","nullable":true,"metadata":{}},{"name":"col150","type":"timestamp","nullable":true,"metadata":{}},{"name":"col151","type":"string","nullable":true,"metadata":{}},{"name":"col152","type":"string","nullable":true,"metadata":{}},{"name":"col153","type":"string","nullable":true,"metadata":{}},{"name":"col154","type":"string","nullable":true,"metadata":{}},{"name":"col155","type":"string","nullable":true,"metadata":{}},{"name":"col156","type":"string","nullable":true,"metadata":{}},{"name":"col157","type":"string","nullable":true,"metadata":{}},{"name":"col158","type":"string","nullable":true,"metadata":{}},{"name":"col
159","type":"string","nullable":true,"metadata":{}},{"name":"col160","type":"string","nullable":true,"metadata":{}},{"name":"col161","type":"string","nullable":true,"metadata":{}},{"name":"op_date_utc","type":"date","nullable":true,"metadata":{}},{"name":"col163","type":"timestamp","nullable":true,"metadata":{}},{"name":"col164","type":"timestamp","nullable":true,"metadata":{}},{"name":"col165","type":"timestamp","nullable":true,"metadata":{}},{"name":"col166","type":"timestamp","nullable":true,"metadata":{}},{"name":"col167","type":"timestamp","nullable":true,"metadata":{}},{"name":"col168","type":"timestamp","nullable":true,"metadata":{}},{"name":"col169","type":"timestamp","nullable":true,"metadata":{}},{"name":"col170","type":"timestamp","nullable":true,"metadata":{}},{"name":"col171","type":"timestamp","nullable":true,"metadata":{}},{"name":"col172","type":"timestamp","nullable":true,"metadata":{}},{"name":"col173","type":"timestamp","nullable":true,"metadata":{}},{"name":"col1
74","type":"timestamp","nullable":true,"metadata":{}},{"name":"col175","type":"timestamp","nullable":true,"metadata":{}},{"name":"col176","type":"timestamp","nullable":true,"metadata":{}},{"name":"col177","type":"timestamp","nullable":true,"metadata":{}},{"name":"col178","type":"timestamp","nullable":true,"metadata":{}},{"name":"col179","type":"timestamp","nullable":true,"metadata":{}},{"name":"col180","type":"timestamp","nullable":true,"metadata":{}},{"name":"col181","type":"timestamp","nullable":true,"metadata":{}},{"name":"col182","type":"timestamp","nullable":true,"metadata":{}},{"name":"col183","type":"boolean","nullable":true,"metadata":{}},{"name":"col184","type":"date","nullable":true,"metadata":{}},{"name":"scheduled_hidden_unique_ids","type":{"type":"array","elementType":"long","containsNull":true},"nullable":true,"metadata":{}},{"name":"col186","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col187","type":"long",
"nullable":true,"metadata":{}},{"name":"col188","type":"long","nullable":true,"metadata":{}},{"name":"col189","type":"long","nullable":true,"metadata":{}},{"name":"col190","type":"long","nullable":true,"metadata":{}},{"name":"col191","type":"long","nullable":true,"metadata":{}},{"name":"col192","type":"long","nullable":true,"metadata":{}},{"name":"col193","type":"timestamp","nullable":true,"metadata":{}},{"name":"col194","type":"timestamp","nullable":true,"metadata":{}},{"name":"col195","type":"timestamp","nullable":true,"metadata":{}},{"name":"col196","type":"timestamp","nullable":true,"metadata":{}},{"name":"col197","type":"timestamp","nullable":true,"metadata":{}},{"name":"col198","type":"timestamp","nullable":true,"metadata":{}},{"name":"col199","type":"timestamp","nullable":true,"metadata":{}},{"name":"col200","type":"timestamp","nullable":true,"metadata":{}},{"name":"col201","type":"timestamp","nullable":true,"metadata":{}},{"name":"col202","type":"timestamp","nullable":true,"
metadata":{}},{"name":"col203","type":"timestamp","nullable":true,"metadata":{}},{"name":"col204","type":"timestamp","nullable":true,"metadata":{}},{"name":"col205","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col206","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col184_local","type":"date","nullable":true,"metadata":{}},{"name":"col206_internal","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col209","type":"string","nullable":true,"metadata":{}},{"name":"col210","type":{"type":"array","elementType":"integer","containsNull":true},"nullable":true,"metadata":{}},{"name":"col211","type":{"type":"array","elementType":"timestamp","containsNull":true},"nullable":true,"metadata":{}},{"name":"col212","type":"string","nullable":true,"metadata":{}},{"name":"col213","type":"integer","nullable":true,"metadata":{}},
{"name":"event_millis","type":"long","nullable":true,"metadata":{}},{"name":"_hoodie_is_deleted","type":"boolean","nullable":true,"metadata":{}},{"name":"xtable_partition_col_DAY_op_date","type":"date","nullable":true,"metadata":{"delta.generationExpression":"CAST(op_date
as DATE)"}}]},List(xtable_partition_col_DAY_op_date),Map(XTABLE_METADATA ->
{"lastInstantSynced":"2024-07-02T17:18:13.028Z","instantsToConsiderForNextSync":[],"version":0},
delta.logRetentionDuration -> interval 168 hours),Some(1719940693028))
2024-07-02 17:43:05 INFO
org.apache.spark.sql.delta.OptimisticTransaction:60 -
[tableId=e65c4be0,txnId=b9564248] Attempting to commit version 0 with 4145
actions with Serializable isolation level
2024-07-02 17:43:07 INFO org.apache.spark.sql.delta.DeltaLog:60 - Creating
a new snapshot v0 for commit version 0
2024-07-02 17:43:07 INFO org.apache.spark.sql.delta.DeltaLog:60 - Loading
version 0.
2024-07-02 17:43:07 INFO org.apache.spark.sql.delta.DeltaLogFileIndex:60 -
Created DeltaLogFileIndex(JSON, numFilesInSegment: 1, totalFileSize: 75250570)
2024-07-02 17:43:11 INFO org.apache.spark.sql.delta.Snapshot:60 -
[tableId=e65c4be0-428e-4ecd-8fb4-360a8fe22dda] Created snapshot
Snapshot(path=s3://hidden-s3-bucket/hidden-prefix/_delta_log, version=0,
metadata=Metadata(e65c4be0-428e-4ecd-8fb4-360a8fe22dda,hidden_table_name,,Format(parquet,Map()),{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"hidden_unique_id","type":"long","nullable":true,"metadata":{}},{"name":"op_date","type":"date","nullable":true,"metadata":{}},{"name":"previous_op_dates","type":{"type":"array","elementType":"date","containsNull":true},"nullable":true,"metadata":{}},{"name":"col101","type"
:"timestamp","nullable":true,"metadata":{}},{"name":"col102","type":"string","nullable":true,"metadata":{}},{"name":"col103","type":"boolean","nullable":true,"metadata":{}},{"name":"col104","type":"timestamp","nullable":true,"metadata":{}},{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col107","type":"string","nullable":true,"metadata":{}},{"name":"col108","type":"boolean","nullable":true,"metadata":{}},{"name":"noun_col105","type":"string","nullable":true,"metadata":{}},{"name":"col110","type":"boolean","nullable":true,"metadata":{}},{"name":"col111","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col113","type":"string","nullable":true,"metadata":{}}]},"containsNull":true},"nullable":true,"metadata":{}},{"name":"col112","type":"string","nulla
ble":true,"metadata":{}},{"name":"col113","type":"string","nullable":true,"metadata":{}},{"name":"col114","type":"long","nullable":true,"metadata":{}},{"name":"col115","type":"string","nullable":true,"metadata":{}},{"name":"col116","type":"string","nullable":true,"metadata":{}},{"name":"col117","type":"string","nullable":true,"metadata":{}},{"name":"col118","type":"string","nullable":true,"metadata":{}},{"name":"col119","type":"string","nullable":true,"metadata":{}},{"name":"col120","type":"string","nullable":true,"metadata":{}},{"name":"col121","type":"string","nullable":true,"metadata":{}},{"name":"col122","type":"string","nullable":true,"metadata":{}},{"name":"col123","type":"timestamp","nullable":true,"metadata":{}},{"name":"col124","type":"timestamp","nullable":true,"metadata":{}},{"name":"col125","type":"timestamp","nullable":true,"metadata":{}},{"name":"col126","type":"timestamp","nullable":true,"metadata":{}},{"name":"col127","type":"timestamp","nullable":true,"metadata":{}}
,{"name":"col128","type":"timestamp","nullable":true,"metadata":{}},{"name":"col129","type":"timestamp","nullable":true,"metadata":{}},{"name":"col130","type":"timestamp","nullable":true,"metadata":{}},{"name":"col131","type":"timestamp","nullable":true,"metadata":{}},{"name":"col132","type":"timestamp","nullable":true,"metadata":{}},{"name":"col133","type":"timestamp","nullable":true,"metadata":{}},{"name":"col134","type":"timestamp","nullable":true,"metadata":{}},{"name":"col135","type":"timestamp","nullable":true,"metadata":{}},{"name":"col136","type":"timestamp","nullable":true,"metadata":{}},{"name":"col137","type":"timestamp","nullable":true,"metadata":{}},{"name":"col138","type":"timestamp","nullable":true,"metadata":{}},{"name":"col139","type":"timestamp","nullable":true,"metadata":{}},{"name":"col140","type":"timestamp","nullable":true,"metadata":{}},{"name":"col141","type":"timestamp","nullable":true,"metadata":{}},{"name":"col142","type":"timestamp","nullable":true,"metad
ata":{}},{"name":"col143","type":"timestamp","nullable":true,"metadata":{}},{"name":"col144","type":"timestamp","nullable":true,"metadata":{}},{"name":"col145","type":"timestamp","nullable":true,"metadata":{}},{"name":"col146","type":"timestamp","nullable":true,"metadata":{}},{"name":"col147","type":"timestamp","nullable":true,"metadata":{}},{"name":"col148","type":"timestamp","nullable":true,"metadata":{}},{"name":"col149","type":"timestamp","nullable":true,"metadata":{}},{"name":"col150","type":"timestamp","nullable":true,"metadata":{}},{"name":"col151","type":"string","nullable":true,"metadata":{}},{"name":"col152","type":"string","nullable":true,"metadata":{}},{"name":"col153","type":"string","nullable":true,"metadata":{}},{"name":"col154","type":"string","nullable":true,"metadata":{}},{"name":"col155","type":"string","nullable":true,"metadata":{}},{"name":"col156","type":"string","nullable":true,"metadata":{}},{"name":"col157","type":"string","nullable":true,"metadata":{}},{"na
me":"col158","type":"string","nullable":true,"metadata":{}},{"name":"col159","type":"string","nullable":true,"metadata":{}},{"name":"col160","type":"string","nullable":true,"metadata":{}},{"name":"col161","type":"string","nullable":true,"metadata":{}},{"name":"op_date_utc","type":"date","nullable":true,"metadata":{}},{"name":"col163","type":"timestamp","nullable":true,"metadata":{}},{"name":"col164","type":"timestamp","nullable":true,"metadata":{}},{"name":"col165","type":"timestamp","nullable":true,"metadata":{}},{"name":"col166","type":"timestamp","nullable":true,"metadata":{}},{"name":"col167","type":"timestamp","nullable":true,"metadata":{}},{"name":"col168","type":"timestamp","nullable":true,"metadata":{}},{"name":"col169","type":"timestamp","nullable":true,"metadata":{}},{"name":"col170","type":"timestamp","nullable":true,"metadata":{}},{"name":"col171","type":"timestamp","nullable":true,"metadata":{}},{"name":"col172","type":"timestamp","nullable":true,"metadata":{}},{"name":
"col173","type":"timestamp","nullable":true,"metadata":{}},{"name":"col174","type":"timestamp","nullable":true,"metadata":{}},{"name":"col175","type":"timestamp","nullable":true,"metadata":{}},{"name":"col176","type":"timestamp","nullable":true,"metadata":{}},{"name":"col177","type":"timestamp","nullable":true,"metadata":{}},{"name":"col178","type":"timestamp","nullable":true,"metadata":{}},{"name":"col179","type":"timestamp","nullable":true,"metadata":{}},{"name":"col180","type":"timestamp","nullable":true,"metadata":{}},{"name":"col181","type":"timestamp","nullable":true,"metadata":{}},{"name":"col182","type":"timestamp","nullable":true,"metadata":{}},{"name":"col183","type":"boolean","nullable":true,"metadata":{}},{"name":"col184","type":"date","nullable":true,"metadata":{}},{"name":"scheduled_hidden_unique_ids","type":{"type":"array","elementType":"long","containsNull":true},"nullable":true,"metadata":{}},{"name":"col186","type":{"type":"array","elementType":"string","containsNu
ll":true},"nullable":true,"metadata":{}},{"name":"col187","type":"long","nullable":true,"metadata":{}},{"name":"col188","type":"long","nullable":true,"metadata":{}},{"name":"col189","type":"long","nullable":true,"metadata":{}},{"name":"col190","type":"long","nullable":true,"metadata":{}},{"name":"col191","type":"long","nullable":true,"metadata":{}},{"name":"col192","type":"long","nullable":true,"metadata":{}},{"name":"col193","type":"timestamp","nullable":true,"metadata":{}},{"name":"col194","type":"timestamp","nullable":true,"metadata":{}},{"name":"col195","type":"timestamp","nullable":true,"metadata":{}},{"name":"col196","type":"timestamp","nullable":true,"metadata":{}},{"name":"col197","type":"timestamp","nullable":true,"metadata":{}},{"name":"col198","type":"timestamp","nullable":true,"metadata":{}},{"name":"col199","type":"timestamp","nullable":true,"metadata":{}},{"name":"col200","type":"timestamp","nullable":true,"metadata":{}},{"name":"col201","type":"timestamp","nullable":t
rue,"metadata":{}},{"name":"col202","type":"timestamp","nullable":true,"metadata":{}},{"name":"col203","type":"timestamp","nullable":true,"metadata":{}},{"name":"col204","type":"timestamp","nullable":true,"metadata":{}},{"name":"col205","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col206","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col184_local","type":"date","nullable":true,"metadata":{}},{"name":"col206_internal","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col209","type":"string","nullable":true,"metadata":{}},{"name":"col210","type":{"type":"array","elementType":"integer","containsNull":true},"nullable":true,"metadata":{}},{"name":"col211","type":{"type":"array","elementType":"timestamp","containsNull":true},"nullable":true,"metadata":{}},{"name":"col212","type":"string","nullable":true,"metadat
a":{}},{"name":"col213","type":"integer","nullable":true,"metadata":{}},{"name":"event_millis","type":"long","nullable":true,"metadata":{}},{"name":"_hoodie_is_deleted","type":"boolean","nullable":true,"metadata":{}},{"name":"xtable_partition_col_DAY_op_date","type":"date","nullable":true,"metadata":{"delta.generationExpression":"CAST(op_date
as DATE)"}}]},List(xtable_partition_col_DAY_op_date),Map(XTABLE_METADATA ->
{"lastInstantSynced":"2024-07-02T17:18:13.028Z","instantsToConsiderForNextSync":[],"version":0},
delta.logRetentionDuration -> interval 168 hours),Some(1719940693028)),
logSegment=LogSegment(s3://hidden-s3-bucket/hidden-prefix/_delta_log,0,WrappedArray(S3AFileStatus{path=s3://hidden-s3-bucket/hidden-prefix/_delta_log/00000000000000000000.json;
isDirectory=false; length=75250570; replication=1; blocksize=33554432;
modification_time=1719942187000; access_time=0; owner=lfm; group=lfm;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
isErasureCoded=fal
se} isEmptyDirectory=FALSE eTag=414a8c48671a5c1a268e391487db2926-2
versionId=null),None,1719942187000), checksumOpt=None)
2024-07-02 17:43:11 INFO org.apache.spark.sql.delta.DeltaLog:60 - Updated
snapshot to Snapshot(path=s3://hidden-s3-bucket/hidden-prefix/_delta_log,
version=0,
metadata=Metadata(e65c4be0-428e-4ecd-8fb4-360a8fe22dda,hidden_table_name,,Format(parquet,Map()),{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"hidden_unique_id","type":"long","nullable":true,"metadata":{}},{"name":"op_date","type":"date","nullable":true,"metadata":{}},{"name":"previous_op_dates","type":{"type":"array","elementType":"date","containsNull":true},"nullable":true,"metadata":{}},{"name":"col101","type":"timestamp","nullable":true,"metadata":{}},
{"name":"col102","type":"string","nullable":true,"metadata":{}},{"name":"col103","type":"boolean","nullable":true,"metadata":{}},{"name":"col104","type":"timestamp","nullable":true,"metadata":{}},{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col107","type":"string","nullable":true,"metadata":{}},{"name":"col108","type":"boolean","nullable":true,"metadata":{}},{"name":"noun_col105","type":"string","nullable":true,"metadata":{}},{"name":"col110","type":"boolean","nullable":true,"metadata":{}},{"name":"col111","type":{"type":"array","elementType":{"type":"struct","fields":[{"name":"col105","type":"string","nullable":true,"metadata":{}},{"name":"col106","type":"string","nullable":true,"metadata":{}},{"name":"col113","type":"string","nullable":true,"metadata":{}}]},"containsNull":true},"nullable":true,"metadata":{}},{"name":"col112","type":"string","nullable":true,"metadata":{}},{"name":"col113","t
ype":"string","nullable":true,"metadata":{}},{"name":"col114","type":"long","nullable":true,"metadata":{}},{"name":"col115","type":"string","nullable":true,"metadata":{}},{"name":"col116","type":"string","nullable":true,"metadata":{}},{"name":"col117","type":"string","nullable":true,"metadata":{}},{"name":"col118","type":"string","nullable":true,"metadata":{}},{"name":"col119","type":"string","nullable":true,"metadata":{}},{"name":"col120","type":"string","nullable":true,"metadata":{}},{"name":"col121","type":"string","nullable":true,"metadata":{}},{"name":"col122","type":"string","nullable":true,"metadata":{}},{"name":"col123","type":"timestamp","nullable":true,"metadata":{}},{"name":"col124","type":"timestamp","nullable":true,"metadata":{}},{"name":"col125","type":"timestamp","nullable":true,"metadata":{}},{"name":"col126","type":"timestamp","nullable":true,"metadata":{}},{"name":"col127","type":"timestamp","nullable":true,"metadata":{}},{"name":"col128","type":"timestamp","nullab
le":true,"metadata":{}},{"name":"col129","type":"timestamp","nullable":true,"metadata":{}},{"name":"col130","type":"timestamp","nullable":true,"metadata":{}},{"name":"col131","type":"timestamp","nullable":true,"metadata":{}},{"name":"col132","type":"timestamp","nullable":true,"metadata":{}},{"name":"col133","type":"timestamp","nullable":true,"metadata":{}},{"name":"col134","type":"timestamp","nullable":true,"metadata":{}},{"name":"col135","type":"timestamp","nullable":true,"metadata":{}},{"name":"col136","type":"timestamp","nullable":true,"metadata":{}},{"name":"col137","type":"timestamp","nullable":true,"metadata":{}},{"name":"col138","type":"timestamp","nullable":true,"metadata":{}},{"name":"col139","type":"timestamp","nullable":true,"metadata":{}},{"name":"col140","type":"timestamp","nullable":true,"metadata":{}},{"name":"col141","type":"timestamp","nullable":true,"metadata":{}},{"name":"col142","type":"timestamp","nullable":true,"metadata":{}},{"name":"col143","type":"timestamp"
,"nullable":true,"metadata":{}},{"name":"col144","type":"timestamp","nullable":true,"metadata":{}},{"name":"col145","type":"timestamp","nullable":true,"metadata":{}},{"name":"col146","type":"timestamp","nullable":true,"metadata":{}},{"name":"col147","type":"timestamp","nullable":true,"metadata":{}},{"name":"col148","type":"timestamp","nullable":true,"metadata":{}},{"name":"col149","type":"timestamp","nullable":true,"metadata":{}},{"name":"col150","type":"timestamp","nullable":true,"metadata":{}},{"name":"col151","type":"string","nullable":true,"metadata":{}},{"name":"col152","type":"string","nullable":true,"metadata":{}},{"name":"col153","type":"string","nullable":true,"metadata":{}},{"name":"col154","type":"string","nullable":true,"metadata":{}},{"name":"col155","type":"string","nullable":true,"metadata":{}},{"name":"col156","type":"string","nullable":true,"metadata":{}},{"name":"col157","type":"string","nullable":true,"metadata":{}},{"name":"col158","type":"string","nullable":true
,"metadata":{}},{"name":"col159","type":"string","nullable":true,"metadata":{}},{"name":"col160","type":"string","nullable":true,"metadata":{}},{"name":"col161","type":"string","nullable":true,"metadata":{}},{"name":"op_date_utc","type":"date","nullable":true,"metadata":{}},{"name":"col163","type":"timestamp","nullable":true,"metadata":{}},{"name":"col164","type":"timestamp","nullable":true,"metadata":{}},{"name":"col165","type":"timestamp","nullable":true,"metadata":{}},{"name":"col166","type":"timestamp","nullable":true,"metadata":{}},{"name":"col167","type":"timestamp","nullable":true,"metadata":{}},{"name":"col168","type":"timestamp","nullable":true,"metadata":{}},{"name":"col169","type":"timestamp","nullable":true,"metadata":{}},{"name":"col170","type":"timestamp","nullable":true,"metadata":{}},{"name":"col171","type":"timestamp","nullable":true,"metadata":{}},{"name":"col172","type":"timestamp","nullable":true,"metadata":{}},{"name":"col173","type":"timestamp","nullable":true,
"metadata":{}},{"name":"col174","type":"timestamp","nullable":true,"metadata":{}},{"name":"col175","type":"timestamp","nullable":true,"metadata":{}},{"name":"col176","type":"timestamp","nullable":true,"metadata":{}},{"name":"col177","type":"timestamp","nullable":true,"metadata":{}},{"name":"col178","type":"timestamp","nullable":true,"metadata":{}},{"name":"col179","type":"timestamp","nullable":true,"metadata":{}},{"name":"col180","type":"timestamp","nullable":true,"metadata":{}},{"name":"col181","type":"timestamp","nullable":true,"metadata":{}},{"name":"col182","type":"timestamp","nullable":true,"metadata":{}},{"name":"col183","type":"boolean","nullable":true,"metadata":{}},{"name":"col184","type":"date","nullable":true,"metadata":{}},{"name":"scheduled_hidden_unique_ids","type":{"type":"array","elementType":"long","containsNull":true},"nullable":true,"metadata":{}},{"name":"col186","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"n
ame":"col187","type":"long","nullable":true,"metadata":{}},{"name":"col188","type":"long","nullable":true,"metadata":{}},{"name":"col189","type":"long","nullable":true,"metadata":{}},{"name":"col190","type":"long","nullable":true,"metadata":{}},{"name":"col191","type":"long","nullable":true,"metadata":{}},{"name":"col192","type":"long","nullable":true,"metadata":{}},{"name":"col193","type":"timestamp","nullable":true,"metadata":{}},{"name":"col194","type":"timestamp","nullable":true,"metadata":{}},{"name":"col195","type":"timestamp","nullable":true,"metadata":{}},{"name":"col196","type":"timestamp","nullable":true,"metadata":{}},{"name":"col197","type":"timestamp","nullable":true,"metadata":{}},{"name":"col198","type":"timestamp","nullable":true,"metadata":{}},{"name":"col199","type":"timestamp","nullable":true,"metadata":{}},{"name":"col200","type":"timestamp","nullable":true,"metadata":{}},{"name":"col201","type":"timestamp","nullable":true,"metadata":{}},{"name":"col202","type":"
timestamp","nullable":true,"metadata":{}},{"name":"col203","type":"timestamp","nullable":true,"metadata":{}},{"name":"col204","type":"timestamp","nullable":true,"metadata":{}},{"name":"col205","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col206","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col184_local","type":"date","nullable":true,"metadata":{}},{"name":"col206_internal","type":{"type":"array","elementType":"string","containsNull":true},"nullable":true,"metadata":{}},{"name":"col209","type":"string","nullable":true,"metadata":{}},{"name":"col210","type":{"type":"array","elementType":"integer","containsNull":true},"nullable":true,"metadata":{}},{"name":"col211","type":{"type":"array","elementType":"timestamp","containsNull":true},"nullable":true,"metadata":{}},{"name":"col212","type":"string","nullable":true,"metadata":{}},{"name":"col213","type":"integer","nu
llable":true,"metadata":{}},{"name":"event_millis","type":"long","nullable":true,"metadata":{}},{"name":"_hoodie_is_deleted","type":"boolean","nullable":true,"metadata":{}},{"name":"xtable_partition_col_DAY_op_date","type":"date","nullable":true,"metadata":{"delta.generationExpression":"CAST(op_date
as DATE)"}}]},List(xtable_partition_col_DAY_op_date),Map(XTABLE_METADATA ->
{"lastInstantSynced":"2024-07-02T17:18:13.028Z","instantsToConsiderForNextSync":[],"version":0},
delta.logRetentionDuration -> interval 168 hours),Some(1719940693028)),
logSegment=LogSegment(s3://hidden-s3-bucket/hidden-prefix/_delta_log,0,WrappedArray(S3AFileStatus{path=s3://hidden-s3-bucket/hidden-prefix/_delta_log/00000000000000000000.json;
isDirectory=false; length=75250570; replication=1; blocksize=33554432;
modification_time=1719942187000; access_time=0; owner=lfm; group=lfm;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true;
isErasureCoded=false} isEmptyDirectory=FALSE eTag=414a8c48671a
5c1a268e391487db2926-2 versionId=null),None,1719942187000), checksumOpt=None)
2024-07-02 17:43:11 INFO org.apache.spark.sql.delta.Snapshot:60 -
[tableId=e65c4be0-428e-4ecd-8fb4-360a8fe22dda] DELTA: Compute snapshot for
version: 0
2024-07-02 17:43:12 WARN org.apache.spark.sql.catalyst.util.package:72 -
Truncated the string representation of a plan since it was too large. This
behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
2024-07-02 17:43:23 INFO org.apache.spark.sql.delta.Snapshot:60 -
[tableId=e65c4be0-428e-4ecd-8fb4-360a8fe22dda] DELTA: Done
2024-07-02 17:43:23 INFO
org.apache.spark.sql.delta.OptimisticTransaction:60 -
[tableId=e65c4be0,txnId=b9564248] Committed delta #0 to
s3://hidden-s3-bucket/hidden-prefix/_delta_log
2024-07-02 17:43:23 INFO
org.apache.xtable.conversion.ConversionController:133 - Sync is successful for
the following formats DELTA
```
Note this is the writer's configuration:
```python
full_cow_hudi_options = {
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.datasource.write.table.type": "COPY_ON_WRITE",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.partitionpath.field": "op_date",
"hoodie.datasource.write.precombine.field": "event_millis",
"hoodie.datasource.write.recordkey.field": "hidden_unique_id",
"hoodie.datasource.write.payload.class":
"org.apache.hudi.common.model.DefaultHoodieRecordPayload",
"hoodie.payload.ordering.field": "event_millis",
"hoodie.insert.shuffle.parallelism": 2,
"hoodie.upsert.shuffle.parallelism": 2,
"hoodie.metadata.enable": "false",
}
df.write.format("hudi").options(**full_cow_hudi_options).mode("append").save("s3://hidden-s3-bucket/hidden-prefix")
```
Since it's changed from the initial run, I am running xtable df515157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
