ahmedabu98 commented on code in PR #23070:
URL: https://github.com/apache/beam/pull/23070#discussion_r986013514
##########
sdks/python/apache_beam/io/gcp/bigquery_schema_tools_test.py:
##########
@@ -61,6 +61,25 @@ def test_check_conversion_with_empty_schema(self):
the_table_schema=schema)
self.assertEqual(usertype.__annotations__, {})
+ def test_check_schema_conversions_with_timestamp(self):
+ fields = [
+ bigquery.TableFieldSchema(name='stn', type='STRING', mode="NULLABLE"),
+ bigquery.TableFieldSchema(name='temp', type='FLOAT64',
mode="REPEATED"),
+ bigquery.TableFieldSchema(
+ name='times', type='TIMESTAMP', mode="NULLABLE")
+ ]
+ schema = bigquery.TableSchema(fields=fields)
+
+ usertype = bigquery_schema_tools.generate_user_type_from_bq_schema(
+ the_table_schema=schema)
+ self.assertEqual(
+ usertype.__annotations__,
+ {
+ 'stn': typing.Optional[str],
+ 'temp': typing.Sequence[np.float64],
+ 'times': typing.Any
Review Comment:
```
'times': typing.Any
```
Should this be testing for an Optional `datetime.datetime` type?
##########
sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py:
##########
@@ -258,6 +258,123 @@ def test_table_schema_retrieve_with_direct_read(self):
utype(id=4, name='customer2', type='test')
]))
+ @pytest.mark.it_postcommit
+ def test_table_schema_retrieve_with_timestamp(self):
+ the_table = bigquery_tools.BigQueryWrapper().get_table(
+ project_id="apache-beam-testing",
+ dataset_id="beam_bigquery_io_test",
+ table_id="taxi_small")
+ table = the_table.schema
+ utype = bigquery_schema_tools. \
+ generate_user_type_from_bq_schema(table)
+ with beam.Pipeline(argv=self.args) as p:
+ result = (
+ p | apache_beam.io.gcp.bigquery.ReadFromBigQuery(
+ gcs_location="gs://bqio_schema_test",
+ dataset="beam_bigquery_io_test",
+ table="taxi_small",
+ project="apache-beam-testing",
+ output_type='BEAM_ROW'))
+ assert_that(
+ result,
+ equal_to([
+ utype(
+ event_timestamp=datetime.datetime(
+ 2021,
+ 11,
+ 5,
+ 20,
+ 57,
+ 43,
+ 612000,
+ tzinfo=datetime.timezone.utc),
+ ride_id='0004b5de-8db1-425b-8eec-7a25b04a1ee0',
+ point_idx=9,
+ latitude=40.757380000000005,
+ timestamp='2021-11-05T16:57:43.60906-04:00',
+ meter_reading=0.37846154,
+ meter_increment=0.04205128,
+ ride_status='enroute',
+ passenger_count=1,
+ longitude=-73.98969000000001),
Review Comment:
Is it possible to only test for the timestamp field since that's what this
test is focusing on? It would help make the test more concise and easy to read
in the future.
One way is to change it so that ReadFromBigQuery queries for the timestamp
field. If schema conversion isn't implemented yet for query reads, another way
may be to add a step that filters `result` so you have a PCollection of
timestamp elements.
##########
sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py:
##########
@@ -258,6 +258,123 @@ def test_table_schema_retrieve_with_direct_read(self):
utype(id=4, name='customer2', type='test')
]))
+ @pytest.mark.it_postcommit
+ def test_table_schema_retrieve_with_timestamp(self):
+ the_table = bigquery_tools.BigQueryWrapper().get_table(
+ project_id="apache-beam-testing",
+ dataset_id="beam_bigquery_io_test",
+ table_id="taxi_small")
+ table = the_table.schema
+ utype = bigquery_schema_tools. \
+ generate_user_type_from_bq_schema(table)
Review Comment:
```suggestion
schema = the_table.schema
utype = bigquery_schema_tools. \
generate_user_type_from_bq_schema(schema)
```
To avoid confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
