jack-moseley commented on a change in pull request #3353:
URL: https://github.com/apache/gobblin/pull/3353#discussion_r683807016
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java
##########
@@ -235,7 +231,7 @@ private void scheduleSpecsFromCatalog() {
private void clearRunningFlowState(Iterator<URI> drUris) {
while (drUris.hasNext()) {
// TODO: Instead of simply call onDeleteSpec, a callback when FlowSpec
is deleted from FlowCatalog, should also kill Azkaban Flow from
AzkabanSpecProducer.
- onDeleteSpec(drUris.next(), FlowSpec.Builder.DEFAULT_VERSION);
+ unscheduleSpec(drUris.next(), FlowSpec.Builder.DEFAULT_VERSION);
Review comment:
We're not using this DR code anyway, but based on comments it seems like
it's intended that it kills running flows that are marked as DR, so maybe let's
leave this method as is.
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java
##########
@@ -347,6 +343,30 @@ public AddSpecResponse onAddSpec(Spec addedSpec) {
return new AddSpecResponse<>(response);
}
+ /**
+ * Remove a flowSpec from schedule due to another leader being elected
+ * Unlike onDeleteSpec, we want to avoid deleting the flowSpec on the
executor
+ * and we still want to unschedule if we cannot connect to zookeeper as the
current node cannot be the master
+ * @param specURI
+ * @param specVersion
+ */
+ private void unscheduleSpec(URI specURI, String specVersion) {
Review comment:
Could we do this instead by just adding an overloaded onDeleteSpec with
a boolean option to remove from the orchestrator to avoid the code duplication?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]