shopmatfournier commented on issue #28008:
URL: https://github.com/apache/beam/issues/28008#issuecomment-1684573680
Chiming in that we ran into this a few weeks ago. I've hacked around it,
for now, in our (java) dataflow jobs. I've been meaning to write up an issue
and go spelunking in the code to try to fix.
Our current hack:
- we create the table using the java BigQuery library directly, not via
BigQueryIO.
- we create a table with the `nullable` field with some
`.setDefaultValueExpresion()`
- as for the schema that Beam/BigQueryIO knows about, we strip this field
from it so they never validate it
- rows inserted into BQ via BigQueryIO have the defaultValueExpression
applied
Downsides: It has to be a `nullable` field for this to work. This is
actually a field we end up partitioning on, so it's not great; however, it's
working for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
