ahmedabu98 commented on code in PR #40542:
URL: https://github.com/apache/beam/pull/40542#discussion_r4265517318
##########
sdks/python/apache_beam/yaml/extended_tests/e2e/delta_lake_cdc_to_iceberg.yaml:
##########
@@ -0,0 +1,94 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
Review Comment:
We don't seem to have Dataflow YAML tests set up. I ran the following on
Dataflow though and got the expected output:
DeltaLake to Iceberg:
```yaml
pipeline:
type: chain
transforms:
- type: ReadFromDeltaCDC
config:
table: "gs://ahmedabualsaud/delta/orders"
start_version: 0
end_version: 3
include_metadata_columns: ["_change_type", "_commit_version"]
hadoop_config:
"fs.gs.impl":
"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
"fs.AbstractFileSystem.gs.impl":
"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
"fs.gs.auth.type": "APPLICATION_DEFAULT"
"fs.gs.project.id": "apache-beam-testing"
- type: WriteToIceberg
config:
table: "shop.orders_from_delta"
catalog_properties:
type: "hadoop"
warehouse: "gs://ahmedabualsaud/iceberg_cdc"
mode: "merge-on-read"
equality_columns: ["id"]
change_type_column: "_change_type"
change_type_map:
insert: "INSERT"
update_preimage: "UPDATE_BEFORE"
update_postimage: "UPDATE_AFTER"
delete: "DELETE"
sequence_number_column: "_commit_version"
```
Iceberg to Iceberg:
```yaml
pipeline:
type: chain
transforms:
- type: ReadFromIcebergCDC
config:
table: "shop.orders_from_delta"
catalog_properties:
type: "hadoop"
warehouse: "gs://ahmedabualsaud/iceberg_cdc"
config_properties:
"fs.gs.impl":
"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"
"fs.AbstractFileSystem.gs.impl":
"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS"
"fs.gs.auth.type": "APPLICATION_DEFAULT"
"fs.gs.project.id": "apache-beam-testing"
include_metadata_columns: ["_change_type",
"_commit_snapshot_sequence_number"]
- type: WriteToIceberg
config:
table: "shop.orders_from_iceberg"
catalog_properties:
type: "hadoop"
warehouse: "gs://ahmedabualsaud/iceberg_cdc"
mode: "merge-on-read"
equality_columns: ["id"]
change_type_column: "_change_type"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
