[
https://issues.apache.org/jira/browse/GOBBLIN-1696?focusedWorklogId=812017&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-812017
]
ASF GitHub Bot logged work on GOBBLIN-1696:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 26/Sep/22 08:11
Start Date: 26/Sep/22 08:11
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3548:
URL: https://github.com/apache/gobblin/pull/3548#discussion_r979698589
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flowgraph/BaseFlowGraphHelper.java:
##########
@@ -277,32 +282,64 @@ private List<SpecExecutor> getSpecExecutors(Config
edgeConfig) throws URISyntaxE
* @return the configuration object
* @throws IOException
*/
- protected Config loadNodeFileWithOverrides(Path filePath) throws IOException
{
+ protected Config loadNodeFileWithOverrides(Path filePath)
+ throws IOException {
Config nodeConfig = this.pullFileLoader.loadPullFile(filePath,
emptyConfig, false, false);
return getNodeConfigWithOverrides(nodeConfig, filePath);
}
-
/**
* Load the edge file.
* @param filePath path of the edge file relative to the repository root
* @return the configuration object
* @throws IOException
*/
- protected Config loadEdgeFileWithOverrides(Path filePath) throws IOException
{
+ protected Config loadEdgeFileWithOverrides(Path filePath)
+ throws IOException {
Config edgeConfig = this.pullFileLoader.loadPullFile(filePath,
emptyConfig, false, false);
return getEdgeConfigWithOverrides(edgeConfig, filePath);
}
+ /**
+ * Loads the entire flowgraph from the path configured in {@link
org.apache.gobblin.configuration.ConfigurationKeys.FLOWGRAPH_BASE_DIR }
+ * Expects nodes to be in the format of /flowGraphName/nodeA/nodeA.properties
+ * Expects edges to be in the format of
/flowGraphName/nodeA/nodeB/edgeAB.properties
+ * The current flowgraph will be swapped atomically with the new flowgraph
that is loaded
+ */
+ public void populateFlowGraphAtomically(AtomicReference<FlowGraph>
flowGraphReference) {
Review Comment:
I was able to change the GitFlowGraphMonitor such that it also follows the
same behavior as the FsFlowGraphMonitor, they will both update fully now and
not perform partial updates. Seems like a more unified solution like you
suggested
Issue Time Tracking
-------------------
Worklog Id: (was: 812017)
Time Spent: 4h 40m (was: 4.5h)
> Build a file-based flowgraph that watches for changes and updates
> -----------------------------------------------------------------
>
> Key: GOBBLIN-1696
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1696
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-service
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Gobblin-as-a-Service only has a Git based flowgraph, which is difficult to
> build CI/CD around. We can provide an alternate flowgraph that is just based
> off files. This flowgraph should update atomically.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
