Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4500711630 Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) The PR bot will only process comments in the main thread (not review comments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
enzomaruffa commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4500712156 @ahmedabu98 of course! Thanks for checking! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
enzomaruffa commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4500708223 Hi hi! Anything we can do to help this move forward? cc @damccorm, @ahmedabu98 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
enzomaruffa commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4500698747 assign set of reviewers -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4280878405 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
damccorm commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4280870391 R: @ahmedabu98 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4280810297 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4260111498 Reminder, please take a look at this pr: @shunping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4214257034 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @shunping for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4198983724 Reminder, please take a look at this pr: @claudevdm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
damccorm commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4155945892 @ahmedabu98 could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4134338368 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @claudevdm for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4117872363 Reminder, please take a look at this pr: @jrmccluskey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
damccorm commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4068000624 @ahmedabu98 could you please take a look here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4054735208 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @jrmccluskey for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-4031010742 Reminder, please take a look at this pr: @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3990677006 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3926949607 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @claudevdm for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3914433940 Reminder, please take a look at this pr: @tvalentyn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
claudevdm commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3872277765 @Abacn @ahmedabu98 can you see if you approve of this approach please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
claudevdm commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3872270904 /gemini review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
claudevdm commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3872267211 Looks like some of the tests are failing def test_convert_to_usertype_with_overrides(self): """Test convert_to_usertype function with type_overrides.""" import datetime > schema = bigquery.TableSchema( fields=[ bigquery.TableFieldSchema( name='id', type='INTEGER', mode="REQUIRED"), bigquery.TableFieldSchema( name='event_date', type='DATE', mode="NULLABLE") ]) E AttributeError: module 'apache_beam.io.gcp.internal.clients.bigquery' has no attribute 'TableSchema' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3871434074 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @tvalentyn for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
enzomaruffa commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2765078302 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: @claudevdm, @ahmedabu98 - I've updated the PR with the `type_overrides` idea. Works perfectly for us. Let me know if we need any other changes on this PR other than the failing checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3847146167 Reminder, please take a look at this pr: @claudevdm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
enzomaruffa commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2733782402 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: @claudevdm sounds pretty reasonable. I have added the work here doing this + updated the PR description. We'll run some more tests with these tweaks internally to make sure they work and I can report back here later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
claudevdm commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2712828662 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: Maybe a middleground is to allow users to pass type_overrides, that override the default type mappings? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
claudevdm commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2712828662 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: Maybe a middleground is to allow users to pass type_overrides, that override the default type mappings? This way beam can add official mappings later on, and using strings now is possible without needing future breaking changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
scwe commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2710457890 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: Hi all :wave: I've been working on this stuff with @enzomaruffa for a while. I totally understand that this solution very hacky but realistically we don't have the resources to dedicate to building anything more robust. So my question essentially becomes: is there any way we can get support for all BQ types bumped in priority on your end? We've been following along/waiting for over a year at this point hoping that it would be solved and would love to be able to use python dataflow in production without maintaining our own hacky fork. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3772611666 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @claudevdm for label python. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3759778475 Reminder, please take a look at this pr: @shunping -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
ahmedabu98 commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2674621615 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: `DATE` and `DATETIME` aren't supported yet (only `TIMESTAMP`) But I'm not very keen on defaulting them to Strings, it's not very robust. We should choose a Python native type that is closer to what a DATE/DATETIME actually is. Some options are discussed in https://s.apache.org/beam-timestamp-strategy (under "Python Nanosecond Support" --> "DateTime"). It'll take more work but I think it's a better longterm solution for Beam. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
enzomaruffa commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2674606689 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: Hi @Abacn! From our experience, these are definitely missing. I am not sure if it has to do something also missing in under-the-hood xlang implementation that uses the Java SDK, but Beam would block us before even getting it deployed to Dataflow with these types being unsupported. I'd love to be wrong and have a simpler solution instead of this, though 🙏 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
Abacn commented on code in PR #37253: URL: https://github.com/apache/beam/pull/37253#discussion_r2673112820 ## sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py: ## @@ -49,6 +49,9 @@ "BYTES": bytes, "TIMESTAMP": apache_beam.utils.timestamp.Timestamp, "GEOGRAPHY": str, +"DATE": str, Review Comment: I remember DATE/DATETIME types were already supported in Python BigQuery Storage API. cc: @ahmedabu98 If there is ineed still gap, we should use Python language type rather than str. Similar treatment was done in Python JdbcIO (Jdbc Date type and time type) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3722509672 Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available commands: - `stop reviewer notifications` - opt out of the automated review tooling - `remind me after tests pass` - tag the comment author after tests pass - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers) The PR bot will only process comments in the main thread (not review comments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
github-actions[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3722190415 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] feat: Adds support for DATE, DATETIME and JSON BigQuery types in the Python SDK using Storage API [beam]
gemini-code-assist[bot] commented on PR #37253: URL: https://github.com/apache/beam/pull/37253#issuecomment-3721921414 ## Summary of Changes Hello @enzomaruffa, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Apache Beam Python SDK's BigQuery I/O capabilities by introducing support for `DATE`, `DATETIME`, and `JSON` BigQuery types. This change is crucial for users migrating Dataflow pipelines to Python, particularly those leveraging the high-performance Storage Write API with `BEAM_ROW` format, as it resolves previous `ValueError` issues encountered with these types. The implementation pragmatically maps these BigQuery types to Python strings, ensuring immediate compatibility and unblocking production pipelines. ### Highlights * **BigQuery Type Support**: Adds support for `DATE`, `DATETIME`, and `JSON` BigQuery types within the Python SDK's BigQuery I/O type mappings. * **Storage Write API Compatibility**: Enables these new types to be used seamlessly with the BigQuery Storage Write API when utilizing the `BEAM_ROW` format (i.e., `use_beam_io_types=True`). * **Python Type Mapping**: The `DATE`, `DATETIME`, and `JSON` BigQuery types are pragmatically mapped to Python's `str` type, consistent with existing `GEOGRAPHY` type handling. * **Comprehensive Testing**: Extensive unit tests have been added for each new type, covering schema conversion, type mapping verification, `bq_field_to_type` function, `convert_to_usertype`, and `BeamSchemaConversionDoFn` processing across various modes (REQUIRED, NULLABLE, REPEATED). 🧠 **New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.** Using Gemini Code Assist The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either `/gemini ` or `@gemini-code-assist `. Below is a summary of the supported commands on the current page. Feature | Command | Description --- | --- | --- Code Review | `/gemini review` | Performs a code review for the current pull request in its current state. Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state. Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments. Help | `/gemini help` | Displays a list of available commands. Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG). You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist). [^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact
