[
https://issues.apache.org/jira/browse/GOBBLIN-1697?focusedWorklogId=810213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-810213
]
ASF GitHub Bot logged work on GOBBLIN-1697:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 20/Sep/22 00:43
Start Date: 20/Sep/22 00:43
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on PR #3549:
URL: https://github.com/apache/gobblin/pull/3549#issuecomment-1251711184
@arjun4084346 here is the answer
1) The normal mysqlSpecStore treat update the same as add operation, and the
add statement accepts add when key is existed. We used only have one host talk
to mysql so this is durable. But when we enable warm standby, multi host can
access mysql for the same entry at the same time. I want to fail one of the
request when such thing happens to make sure they are not over write with each
other. This require a new mysql catalog where we treat update and add
different. Add will fail when the key already exist, update will fail when it
try to update one non-existed key or the key is recently updated after we
receive the request (where concurrent write happens)
2) new resources handler is specifically for warm standby where we do not
care whether it's leader/follower, and directly compile and update the flow
catalog. Also it needs to call update method for mysql catalog when it's a
update request
3) those two are both for warmstandby, and as new resource handler needs to
handle the race condition, we introduce the new mysql catalog to handle that
Issue Time Tracking
-------------------
Worklog Id: (was: 810213)
Time Spent: 2h 10m (was: 2h)
> Have a separate resource handler to rely on CDC stream to do message
> forwarding
> -------------------------------------------------------------------------------
>
> Key: GOBBLIN-1697
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1697
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
