[
https://issues.apache.org/jira/browse/BEAM-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311570#comment-16311570
]
Łukasz Gajowy commented on BEAM-3406:
-------------------------------------
In short: to fix this issue right away I think we need access to
apache-beam-testing project to set schema in bigquery table.
While creating the dashboard for FileBased ioit we created the table in the
bigquery underneath but the schema was not set and it is needed there.
According to this:
[documentation|https://cloud.google.com/bigquery/docs/schemas#specify-schema-manual-cli]
we can set the schema while loading data or right after creating the table.
Perfkit does not set the schema in any way for now.
We could set the schema manually and fix the issue, but AFAIK we do not have
access to "apache-beam-testing" where bigquery seems to be located. Could you
grant it to us [~chamikara]?
I also created [an
issue|https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/1552] on
PKBs github to add an --autodetect flag while loading json files to bigquery -
as i checked locally, the schema we use can be autodetected. If this issue gets
resolved and merged in perfkit we won't have to worry about the schema anymore
as it will be set automatically.
> FIeBasedIO IT fails due to "BigQuery error"
> -------------------------------------------
>
> Key: BEAM-3406
> URL: https://issues.apache.org/jira/browse/BEAM-3406
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Chamikara Jayalath
> Assignee: Łukasz Gajowy
>
> For example,
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_FileBasedIO_IT/54/console
> beam_performance.filebasedioit_pkb_results
> /tmp/perfkitbenchmarker/runs/e2e3d02d/perfkit-bq-pub3ntgN3.json. Got return
> code (1).
> STDOUT: BigQuery error in load operation: No schema specified on job or table.
> STDERR:
> /usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
> UserWarning: You have requested explicit scopes to be used with a GCE
> service account.
> Using this argument will have no effect on the actual scopes for tokens
> requested. These scopes are set at VM instance creation time and
> can't be overridden in the request.
> warnings.warn(_SCOPES_WARNING)
> Lukasz and Darek, can you take a look ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
