[
https://issues.apache.org/jira/browse/GOBBLIN-2002?focusedWorklogId=905017&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-905017
]
ASF GitHub Bot logged work on GOBBLIN-2002:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 15/Feb/24 07:21
Start Date: 15/Feb/24 07:21
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3878:
URL: https://github.com/apache/gobblin/pull/3878#discussion_r1490508747
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/MostlyInMemoryDagManagementStateStore.java:
##########
@@ -130,44 +122,42 @@ public Set<String> getFailedDagIds() throws IOException {
}
@Override
- public synchronized void deleteDagNodeState(String dagId,
Dag.DagNode<JobExecutionPlan> dagNode) {
+ public synchronized void deleteDagNodeState(DagManager.DagId dagId,
Dag.DagNode<JobExecutionPlan> dagNode) {
+ String dagIdStr = dagId.toString();
this.jobToDag.remove(dagNode);
this.dagNodes.remove(dagNode.getValue().getId());
- this.dagToDeadline.remove(dagId);
- this.dagToJobs.get(dagId).remove(dagNode);
- if (this.dagToJobs.get(dagId).isEmpty()) {
- this.dagToJobs.remove(dagId);
+ this.dagToDeadline.remove(dagIdStr);
+ this.dagToJobs.get(dagIdStr).remove(dagNode);
+ if (this.dagToJobs.get(dagIdStr).isEmpty()) {
+ this.dagToJobs.remove(dagIdStr);
}
}
@Override
- public synchronized void addDagNodeState(String dagId,
Dag.DagNode<JobExecutionPlan> dagNode) {
- Dag<JobExecutionPlan> dag = this.dags.get(dagId);
+ public synchronized void addDagNodeState(DagManager.DagId dagId,
Dag.DagNode<JobExecutionPlan> dagNode)
+ throws IOException {
+ String dagIdStr = dagId.toString();
+ Dag<JobExecutionPlan> dag = getDag(dagId);
this.jobToDag.put(dagNode, dag);
this.dagNodes.put(dagNode.getValue().getId(), dagNode);
- if (!this.dagToJobs.containsKey(dagId)) {
- this.dagToJobs.put(dagId, Lists.newLinkedList());
+ if (!this.dagToJobs.containsKey(dagIdStr)) {
+ this.dagToJobs.put(dagIdStr, Lists.newLinkedList());
}
- this.dagToJobs.get(dagId).add(dagNode);
- }
-
- @Override
- public Dag<JobExecutionPlan> getDag(DagManager.DagId dagId) {
- return this.dags.get(dagId.toString());
+ this.dagToJobs.get(dagIdStr).add(dagNode);
}
@Override
- public void addDag(Dag<JobExecutionPlan> dag) {
- this.dags.put(DagManagerUtils.generateDagId(dag).toString(), dag);
+ public Dag<JobExecutionPlan> getDag(DagManager.DagId dagId) throws
IOException {
+ return this.dagStateStore.getDag(dagId.toString());
}
@Override
- public boolean containsDag(String dagId) {
- return this.dags.containsKey(dagId);
+ public boolean containsDag(DagManager.DagId dagId) throws IOException {
+ return this.dagStateStore.existsDag(dagId.toString());
Review Comment:
`DagId` seems the better param type for the `DagStateStore` methods than
`String`. I suggest we move to that by making default methods in the DSS
interface that call `.toString()` there, within the interface itself, rather
than here
then either as part of this PR or leave a TODO to deprecate the `String
dagId` forms
Issue Time Tracking
-------------------
Worklog Id: (was: 905017)
Time Spent: 4h 10m (was: 4h)
> create MostlyInMemoryDagManagementStateStore to merge UserQuotaManager,
> DagStateStore and in-memory dag maps used in DagManager
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-2002
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2002
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> this will help in refactoring DagManager
--
This message was sent by Atlassian Jira
(v8.20.10#820010)