Urmi Mustafi created GOBBLIN-1797:
-------------------------------------
Summary: Skip scheduling flows far into future
Key: GOBBLIN-1797
URL: https://issues.apache.org/jira/browse/GOBBLIN-1797
Project: Apache Gobblin
Issue Type: Improvement
Components: gobblin-service
Reporter: Urmi Mustafi
Assignee: Abhishek Tiwari
The unschedule feature linked below sets a schedule to run Jan 1st of 2050 so
far in advance that it will "never run"
[https://jarvis.corp.linkedin.com/codesearch/result/?name=FlowConfigResourceLocalHandler.java&path=gobblin-elr%2Fgobblin-restli%2Fgobblin-flow-config-service%2Fgobblin-flow-config-service-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fservice&reponame=linkedin%2Fgobblin-elr#62]
but potentially there are over 100k of these flows so we are loading and
scheduling many unnecessary flows. On initialization we add a check that
verifies the next run of the flow is within a certain time frame (100 days by
default) and loads it into the scheduler if it is within that time frame. We
choose that default value under the assumption that we will redeploy GaaS at
least every 100 days and then if we approach a far out scheduled flow we will
load it into the Scheduler. However, in most cases uses schedule flows for near
future or immediately and those will all be scheduled. This PR also renames
metrics and adds helpful new ones.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
