autumnust commented on a change in pull request #3214:
URL: https://github.com/apache/incubator-gobblin/pull/3214#discussion_r565719362
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/StreamModelTaskRunner.java
##########
@@ -86,7 +87,10 @@ protected void run() throws Exception {
// The cancel is not propagated to the extractor's record generator when
it has been turned into a hot Flowable
// by publish, so set the shutdownRequested flag on cancel to stop the
extractor
- Flowable streamWithShutdownOnCancel = connectableStream.doOnCancel(() ->
this.shutdownRequested.set(true));
+ Flowable streamWithShutdownOnCancel = connectableStream.doOnCancel(() -> {
+ this.shutdownRequested.set(true);
+ this.extractor.shutdown();
Review comment:
I don't quite remember how was the failure propagated when the slow
write happens, but feels like the right fix here is to add finally block in the
place where `extractor.shutdown()` is called in `Task` (e.g. line 514) after
catching failure, instead of forcing extractor closure here.
##########
File path:
gobblin-core-base/src/main/java/org/apache/gobblin/source/extractor/extract/FlushingExtractor.java
##########
@@ -298,6 +298,14 @@ protected void publishTaskOutput() throws IOException {
this.flushPublisher.get().publish(Collections.singletonList(workUnitState));
}
+ @Override
+ public void shutdown() {
Review comment:
What's the reason super.shutdown is not called, is this intentional ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
