[
https://issues.apache.org/jira/browse/BEAM-7428?focusedWorklogId=249956&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249956
]
ASF GitHub Bot logged work on BEAM-7428:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/May/19 08:29
Start Date: 29/May/19 08:29
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #8691: [BEAM-7428]
Output the timestamp on elements in ReadAllViaFileBasedSource
URL: https://github.com/apache/beam/pull/8691#discussion_r288449355
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java
##########
@@ -98,15 +107,53 @@ private ReadFileRangesFn(
public void process(ProcessContext c) throws IOException {
ReadableFile file = c.element().getKey();
OffsetRange range = c.element().getValue();
- FileBasedSource<T> source =
+ c.output(
CompressedSource.from(createSource.apply(file.getMetadata().resourceId().toString()))
- .withCompression(file.getCompression());
+ .withCompression(file.getCompression())
+ .createForSubrangeOfFile(file.getMetadata(), range.getFrom(),
range.getTo()));
+ }
+ }
+
+ /**
+ * A {@link Coder} for {@link BoundedSource}s that wraps a {@link
SerializableCoder}. We cannot
+ * safely use an unwrapped SerializableCoder because {@link
+ * SerializableCoder#structuralValue(Serializable)} assumes that coded
elements support object
+ * equality (https://issues.apache.org/jira/browse/BEAM-3807). By default,
Coders compare equality
+ * by serialized bytes, which we want in this case. It is usually safe to
depend on coded
+ * representation here because we only compare objects on bundle commit,
which compares
+ * serializations of the same object instance.
+ *
+ * <p>BoundedSources are generally not used as PCollection elements, so we
do not expose this
+ * coder for wider use.
+ */
+ public static class BoundedSourceCoder<T> extends
CustomCoder<BoundedSource<T>> {
+ private final Coder<BoundedSource<T>> coder;
+
+ public BoundedSourceCoder() {
+ coder = (Coder<BoundedSource<T>>) SerializableCoder.of((Class)
BoundedSource.class);
+ }
+
+ @Override
+ public void encode(BoundedSource<T> value, OutputStream outStream)
+ throws CoderException, IOException {
+ coder.encode(value, outStream);
+ }
+
+ @Override
+ public BoundedSource<T> decode(InputStream inStream) throws
CoderException, IOException {
+ return coder.decode(inStream);
+ }
+ }
+
+ /** Reads elements contained within an input {@link BoundedSource}. */
+ // TODO: Extend to be a Splittable DoFn.
+ public static class ReadFromBoundedSourceFn<T> extends
DoFn<BoundedSource<T>, T> {
Review comment:
Wondering the same if I should better let this as it was, because the reuse
we do here is pretty low, however is where I noticed the difference between
`JavaReadViaImpulse` and `ReadAllViaFileBasedSource`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 249956)
Time Spent: 1h 40m (was: 1.5h)
> ReadAllViaFileBasedSource does not output the timestamps of the read elements
> -----------------------------------------------------------------------------
>
> Key: BEAM-7428
> URL: https://issues.apache.org/jira/browse/BEAM-7428
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Minor
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This differs from the implementation of JavaReadViaImpulse that tackles a
> similar problem but does output the timestamps correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
