[
https://issues.apache.org/jira/browse/GOBBLIN-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Allemandou updated GOBBLIN-1558:
---------------------------------------
Description:
It's currently not possible to overwrite the behavior of the publisher when
publishing to non-existing parent directory (first run of job type for
instance). This would be needed to make TimePartitionDataPublisher save
recordPublisherOutputDirs at lowest granularity (detailed subfolders).
BaseDataPublisher: Extract new method `addWriterOutputToNewDir` that goes with
the already existing `addWriterOutputToExistingDir`. No test needed, the code
is no-op on class behavior.
TimePartitionedDataPublisher: Override the new `addWriterOutputToNewDir` method
to create the publisher parent folder and reuse the
`addWriterOutputToExistingDir` method. Rename and update
TimePartitionedStreamingDataPublisherTest class to actually test
TimePartitionedDataPublisher.
TimePartitionedStreamingDataPublisher: Remove publisher parent folder creation
as it managed in TimePartitionedDataPublisher superclass. Add test based on the
copied TimePartitionedStreamingDataPublisherTest to verify the change also
works for this child class.
was:
It's currently not possible to overwrite the behavior of the publisher when
publishing to non-existing parent directory (first run of job type for
instance). This would be needed to make TimePartitionDataPublisher save
recordPublisherOutputDirs at lowest granularity (detailed subfolders).
BaseDataPublisher: Extract new method `addWriterOutputToNewDir` that goes with
the already existing `addWriterOutputToExistingDir`. No test needed, the code
is no-op on class behavior.
TimePartitionedDataPublisher: Override the new `addWriterOutputToNewDir` method
to create the publisher parent folder and reuse the
`addWriterOutputToExistingDir` method. Rename and update
TimePartitionedStreamingDataPublisherTest class to actually test
TimePartitionedDataPublisher.
TimePartitionedStreamingDataPublisher: Remove publisher parent folder creation
as it managed in TimePartitionedDataPublisher superclass.
> Overwrite BaseDataPublisher behavior when parent-folder doesn't exist and use
> it in TimePartitionedDataPublisher
> ----------------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1558
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1558
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-core
> Reporter: Joseph Allemandou
> Assignee: Abhishek Tiwari
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It's currently not possible to overwrite the behavior of the publisher when
> publishing to non-existing parent directory (first run of job type for
> instance). This would be needed to make TimePartitionDataPublisher save
> recordPublisherOutputDirs at lowest granularity (detailed subfolders).
>
> BaseDataPublisher: Extract new method `addWriterOutputToNewDir` that goes
> with the already existing `addWriterOutputToExistingDir`. No test needed, the
> code is no-op on class behavior.
> TimePartitionedDataPublisher: Override the new `addWriterOutputToNewDir`
> method to create the publisher parent folder and reuse the
> `addWriterOutputToExistingDir` method. Rename and update
> TimePartitionedStreamingDataPublisherTest class to actually test
> TimePartitionedDataPublisher.
> TimePartitionedStreamingDataPublisher: Remove publisher parent folder
> creation as it managed in TimePartitionedDataPublisher superclass. Add test
> based on the copied TimePartitionedStreamingDataPublisherTest to verify the
> change also works for this child class.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
