[
https://issues.apache.org/jira/browse/BEAM-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000150#comment-17000150
]
Lukasz Gajowy commented on BEAM-6969:
-------------------------------------
[~pawel.pasterz] [~mwalenia] you might find this (historical) ticket
interesting... :)
> Provide way to collect start/end read/write time inside the IOs
> ---------------------------------------------------------------
>
> Key: BEAM-6969
> URL: https://issues.apache.org/jira/browse/BEAM-6969
> Project: Beam
> Issue Type: Wish
> Components: io-ideas, testing
> Reporter: Lukasz Gajowy
> Priority: Minor
>
> Currently, IO tests measure time using Metrics API but collect start/end time
> from ParDo transforms that are adjacent to the IO. It's fine for some tests
> but maybe could be done better. The drawback of the current solution is that
> we cannot collect time before PBegin and after PDone. Other than that the
> time we collect now is still not the exact time of read/write start/end but
> only the time at which first/last record appeared in the DoFn.
> See:
> [TimeMonitor.java|https://github.com/apache/beam/blob/957b7cc7746aa626d2eb4dea341f668ec19d5d39/sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/metrics/TimeMonitor.java]
> as an example of such DoFn.
> Possible solution: save metrics in startBundle / finishBundle method in IOs
> whenever a dedicated pipelineOption is set to true.
> In general, maybe it's a good idea to place some other metrics inside IOs
> too? wdyt?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
