[
https://issues.apache.org/jira/browse/BEAM-7428?focusedWorklogId=249200&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249200
]
ASF GitHub Bot logged work on BEAM-7428:
----------------------------------------
Author: ASF GitHub Bot
Created on: 28/May/19 08:53
Start Date: 28/May/19 08:53
Worklog Time Spent: 10m
Work Description: iemejia commented on pull request #8691: [BEAM-7428]
Output the timestamp on elements in ReadAllViaFileBasedSource
URL: https://github.com/apache/beam/pull/8691#discussion_r287995626
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java
##########
@@ -98,15 +108,53 @@ private ReadFileRangesFn(
public void process(ProcessContext c) throws IOException {
ReadableFile file = c.element().getKey();
OffsetRange range = c.element().getValue();
- FileBasedSource<T> source =
+ c.output(
CompressedSource.from(createSource.apply(file.getMetadata().resourceId().toString()))
- .withCompression(file.getCompression());
+ .withCompression(file.getCompression())
+ .createForSubrangeOfFile(file.getMetadata(), range.getFrom(),
range.getTo()));
+ }
+ }
+
+ /**
+ * A {@link Coder} for {@link BoundedSource}s that wraps a {@link
SerializableCoder}. We cannot
+ * safely use an unwrapped SerializableCoder because {@link
+ * SerializableCoder#structuralValue(Serializable)} assumes that coded
elements support object
+ * equality (https://issues.apache.org/jira/browse/BEAM-3807). By default,
Coders compare equality
+ * by serialized bytes, which we want in this case. It is usually safe to
depend on coded
+ * representation here because we only compare objects on bundle commit,
which compares
+ * serializations of the same object instance.
+ *
+ * <p>BoundedSources are generally not used as PCollection elements, so we
do not expose this
+ * coder for wider use.
+ */
+ public static class BoundedSourceCoder<T> extends
CustomCoder<BoundedSource<T>> {
Review comment:
Wondering if I should put thisas an inner class of BoundedSource or other
place or if is ok here..
Also if we should auto register this Coder in `BoundedSource`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 249200)
Time Spent: 20m (was: 10m)
> ReadAllViaFileBasedSource does not output the timestamps of the read elements
> -----------------------------------------------------------------------------
>
> Key: BEAM-7428
> URL: https://issues.apache.org/jira/browse/BEAM-7428
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This differs from the implementation of JavaReadViaImpulse that tackles a
> similar problem but does output the timestamps correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
