[
https://issues.apache.org/jira/browse/GOBBLIN-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hung Tran resolved GOBBLIN-458.
-------------------------------
Resolution: Fixed
Fix Version/s: 0.13.0
Issue resolved by pull request #2329
[https://github.com/apache/incubator-gobblin/pull/2329]
> Refactor flowConfig resource handler to avoid single restli request handled
> partially on one machine and then forward to another machine.
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-458
> URL: https://issues.apache.org/jira/browse/GOBBLIN-458
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Kuai Yu
> Assignee: Kuai Yu
> Priority: Major
> Fix For: 0.13.0
>
>
> This change is to consolidate the Restli request handling.
> Because today what would happened after user submit a restli request is below:
> # Standby node receives the request (flowConfig)
> # Standby convert flowConfig to flowSpec and saved in flowCatalog
> # In step 2, after persisting to flowCatalog, the scheduler was notified to
> handle the real scheduling
> # Because this is Standby node, we cannot schedule it, so forward the
> message to the real active node.
>
> Because the actual restli request handling part now was split into two parts
> and handled by two different nodes. It implies that we have to add some
> pingpong messages to ack which steps has been executed, and also it adds the
> complexity for us to add rollback or partial recover mechanism in the future.
> Another concern is that in the near future, we might have to refactor Helix
> load balancing logic based on some partitioning logic. When this happened, we
> just need to rely on Helix to do load balancing work. At that time today's
> load balancing mechanism (splitting persistence and scheduling) seems not
> very practical and could be error prone due to we have to concern about
> network issue between two nodes while we perform the actual handling logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
