arnavarora2004 commented on code in PR #35696:
URL: https://github.com/apache/beam/pull/35696#discussion_r2248976636
##########
sdks/python/apache_beam/yaml/tests/bigtable.yaml:
##########
@@ -85,3 +83,95 @@ pipelines:
project: 'apache-beam-testing'
instance: "{BT_INSTANCE}"
table: 'test-table'
+ - pipeline:
+ type: chain
+ transforms:
+ - type: ReadFromBigTable
+ config:
+ project: 'apache-beam-testing'
+ instance: "{BT_INSTANCE}"
+ table: 'test-table'
+ - type: MapToFields
+ config:
+ language: python
+ fields:
+ key:
+ callable: |
+ def convert_to_string(row):
+ return row.key.decode("utf-8") if "key" in row._fields
else None
+ family_name:
+ family_name
+ column_qualifier:
+ callable: |
+ def convert_to_string(row):
+ return row.column_qualifier.decode("utf-8") if
"column_qualifier" in row._fields else None
+ cells:
+ callable: |
+ def convert_to_string(row):
+ cell_bytes = []
+ for (value, timestamp) in row.cells:
+ value_bytes = value.decode("utf-8")
+ cell_bytes.append(beam.Row(value=value_bytes,
timestamp_micros=timestamp))
+ return cell_bytes
+ - type: AssertEqual
+ config:
+ elements:
+ - { key: 'row1',
+ family_name: "cf1",
+ column_qualifier: "cq1",
+ cells:[{
+ value: "value1",
+ timestamp_micros: 5000}]}
+ - { key: 'row1',
+ family_name: "cf1",
+ column_qualifier: "cq2",
+ cells: [{
+ value: "value2",
+ timestamp_micros: 1000 } ] }
+ - type: LogForTesting
+
+ - pipeline:
+ type: chain
+ transforms:
+ - type: ReadFromBigTable
+ config:
+ project: 'apache-beam-testing'
+ instance: "{BT_INSTANCE}"
+ table: 'test-table'
+ flatten: False
+ - type: MapToFields
+ config:
+ language: python
+ fields:
+ key:
+ callable: |
+ def convert_to_bytes(row):
+ return row.key.decode("utf-8") if "key" in row._fields
else None
+
+ column_families:
+ column_families
+
+ - type: AssertEqual
Review Comment:
okay sounds good,
the assertion is what is expected out of the read, structure wise per say, a
map of column families mapping to column qualifiers each with a list of cells
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
