derrickaw commented on code in PR #35696:
URL: https://github.com/apache/beam/pull/35696#discussion_r2248300409
##########
sdks/python/apache_beam/yaml/tests/bigtable.yaml:
##########
@@ -85,3 +83,95 @@ pipelines:
project: 'apache-beam-testing'
instance: "{BT_INSTANCE}"
table: 'test-table'
+ - pipeline:
+ type: chain
+ transforms:
+ - type: ReadFromBigTable
+ config:
+ project: 'apache-beam-testing'
+ instance: "{BT_INSTANCE}"
+ table: 'test-table'
+ - type: MapToFields
+ config:
+ language: python
+ fields:
+ key:
+ callable: |
+ def convert_to_string(row):
+ return row.key.decode("utf-8") if "key" in row._fields
else None
+ family_name:
+ family_name
+ column_qualifier:
+ callable: |
+ def convert_to_string(row):
+ return row.column_qualifier.decode("utf-8") if
"column_qualifier" in row._fields else None
+ cells:
+ callable: |
+ def convert_to_string(row):
+ cell_bytes = []
+ for (value, timestamp) in row.cells:
+ value_bytes = value.decode("utf-8")
+ cell_bytes.append(beam.Row(value=value_bytes,
timestamp_micros=timestamp))
+ return cell_bytes
+ - type: AssertEqual
+ config:
+ elements:
+ - { key: 'row1',
+ family_name: "cf1",
+ column_qualifier: "cq1",
+ cells:[{
+ value: "value1",
+ timestamp_micros: 5000}]}
+ - { key: 'row1',
+ family_name: "cf1",
+ column_qualifier: "cq2",
+ cells: [{
+ value: "value2",
+ timestamp_micros: 1000 } ] }
+ - type: LogForTesting
+
+ - pipeline:
+ type: chain
+ transforms:
+ - type: ReadFromBigTable
+ config:
+ project: 'apache-beam-testing'
+ instance: "{BT_INSTANCE}"
+ table: 'test-table'
+ flatten: False
+ - type: MapToFields
+ config:
+ language: python
+ fields:
+ key:
+ callable: |
+ def convert_to_bytes(row):
+ return row.key.decode("utf-8") if "key" in row._fields
else None
+
+ column_families:
+ column_families
+
+ - type: AssertEqual
Review Comment:
it seems the failure is due to this assertion - two things:
1. The assertion seems to not match the expected structure. Should this
assertion structure be revised or is the actual one correct? If it just needs
revising, revise it and then comment it out based on below.
2. I know there is a BeamSchema issue that I don't think we can get around
currently that we talked about - I would recommend opening an issue about it
and assign me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
