[
https://issues.apache.org/jira/browse/BEAM-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667378#comment-16667378
]
Etienne Chauchot edited comment on BEAM-5906 at 10/29/18 3:59 PM:
------------------------------------------------------------------
Thanks for opening the ticket Lukasz ! I 100% agree with the target, as
discussed, we just need to be sure to reduce the code duplication to the
minimum and more important use proper mock of the BQ backend service. Right
now, the pipeline I coded uses FakeBigQueryServices to mock BQ but this class
is coupled to beam on its Options (only on this point). Challange will be to
create a BQ mock class usable to mock a pipeline or a standalone code with as
little code duplication as possible.
was (Author: echauchot):
Thanks for opening the ticket Lukasz ! As discussed I 100% agree with the
target, we just need to be sure to reduce the code duplication to the minimum
and more important use proper mock of the BQ backend service. Right now, the
pipeline I coded uses FakeBigQueryServices to mock BQ but this class is coupled
to beam on its Options (only on this point). Challange will be to create a BQ
mock class usable to mock a pipeline or a standalone code with as little code
duplication as possible.
> Remove pipeline for publishing nexmark results to bigQuery and publish using
> BigQuery API only
> ----------------------------------------------------------------------------------------------
>
> Key: BEAM-5906
> URL: https://issues.apache.org/jira/browse/BEAM-5906
> Project: Beam
> Issue Type: Improvement
> Components: examples-nexmark
> Reporter: Lukasz Gajowy
> Assignee: Lukasz Gajowy
> Priority: Minor
>
> There's no need to start a separate pipeline for uploading metrics results
> from Nexmark suites to BigQuery. We can use an API designed for that and
> place it in test-utils. Thanks to that:
> - it won't start a separate pipeline every time it publishes results
> - other suites will be able to use that code
> - We will not face problems like special long to int conversion due to
> problems in BigQueryIO (eg. BEAM-4734) because we will use a thin API instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
