Kuai Yu created GOBBLIN-458:
-------------------------------
Summary: Refactor flowConfig resource handler to avoid single
restli request handled partially on one machine and then forward to another
machine.
Key: GOBBLIN-458
URL: https://issues.apache.org/jira/browse/GOBBLIN-458
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Kuai Yu
Assignee: Kuai Yu
This change is to consolidate the Restli request handling.
Because today what would happened after user submit a restli request is below:
# Standby node receives the request (flowConfig)
# Standby convert flowConfig to flowSpec and saved in flowCatalog
# In step 2, after persisting to flowCatalog, the scheduler was notified to
handle the real scheduling
# Because this is Standby node, we cannot schedule it, so forward the message
to the real active node.
Because the actual restli request handling part now was split into two parts
and handled by two different nodes. It implies that we have to add some
pingpong messages to ack which steps has been executed, and also it adds the
complexity for us to add rollback or partial recover mechanism in the future.
Another concern is that in the near future, we might have to refactor Helix
load balancing logic based on some partitioning logic. When this happened, we
just need to rely on Helix to do load balancing work. At that time today's load
balancing mechanism (splitting persistence and scheduling) seems not very
practical and could be error prone due to we have to concern about network
issue between two nodes while we perform the actual handling logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
