[
https://issues.apache.org/jira/browse/BEAM-7428?focusedWorklogId=249874&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249874
]
ASF GitHub Bot logged work on BEAM-7428:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/May/19 06:41
Start Date: 29/May/19 06:41
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8691:
[BEAM-7428] Output the timestamp on elements in ReadAllViaFileBasedSource
URL: https://github.com/apache/beam/pull/8691#discussion_r288412035
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/io/ReadAllViaFileBasedSource.java
##########
@@ -98,15 +108,53 @@ private ReadFileRangesFn(
public void process(ProcessContext c) throws IOException {
ReadableFile file = c.element().getKey();
OffsetRange range = c.element().getValue();
- FileBasedSource<T> source =
+ c.output(
CompressedSource.from(createSource.apply(file.getMetadata().resourceId().toString()))
- .withCompression(file.getCompression());
+ .withCompression(file.getCompression())
+ .createForSubrangeOfFile(file.getMetadata(), range.getFrom(),
range.getTo()));
+ }
+ }
+
+ /**
+ * A {@link Coder} for {@link BoundedSource}s that wraps a {@link
SerializableCoder}. We cannot
+ * safely use an unwrapped SerializableCoder because {@link
+ * SerializableCoder#structuralValue(Serializable)} assumes that coded
elements support object
+ * equality (https://issues.apache.org/jira/browse/BEAM-3807). By default,
Coders compare equality
+ * by serialized bytes, which we want in this case. It is usually safe to
depend on coded
+ * representation here because we only compare objects on bundle commit,
which compares
+ * serializations of the same object instance.
+ *
+ * <p>BoundedSources are generally not used as PCollection elements, so we
do not expose this
+ * coder for wider use.
+ */
+ public static class BoundedSourceCoder<T> extends
CustomCoder<BoundedSource<T>> {
Review comment:
Probably this should be it's own file if public ?
I don't think there's a reason to register since I don't think using
BoundedSources as elements of a PCollection should be a recommended pattern for
Beam (even though we do it here as a stop gap till we get SDF).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 249874)
Time Spent: 40m (was: 0.5h)
> ReadAllViaFileBasedSource does not output the timestamps of the read elements
> -----------------------------------------------------------------------------
>
> Key: BEAM-7428
> URL: https://issues.apache.org/jira/browse/BEAM-7428
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This differs from the implementation of JavaReadViaImpulse that tackles a
> similar problem but does output the timestamps correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
