[
https://issues.apache.org/jira/browse/GOBBLIN-2092?focusedWorklogId=924093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-924093
]
ASF GitHub Bot logged work on GOBBLIN-2092:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jun/24 09:02
Start Date: 24/Jun/24 09:02
Worklog Time Spent: 10m
Work Description: pratapaditya04 commented on code in PR #3978:
URL: https://github.com/apache/gobblin/pull/3978#discussion_r1650635452
##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStoreTest.java:
##########
@@ -130,6 +132,35 @@ public void setUp() throws Exception {
.withDescription("Test flow spec 4")
.withVersion("Test version 4")
.build();
+ flowSpec5=
+ FlowSpec.builder(this.uri5)
+ .withConfig(ConfigBuilder.create()
+ .addPrimitive(FLOW_SOURCE_IDENTIFIER_KEY, "source")
+ .addPrimitive(FLOW_DESTINATION_IDENTIFIER_KEY, "destination")
+ .addPrimitive(ConfigurationKeys.JOB_SCHEDULE_KEY, "0 0 8 * * ?
*")
+ .addPrimitive(ConfigurationKeys.FLOW_OWNING_GROUP_KEY,
"owningGroup")
+ .addPrimitive(ConfigurationKeys.FLOW_RUN_IMMEDIATELY, "false")
+ .addPrimitive("user.to.proxy", "userA")
+ .addPrimitive(ConfigurationKeys.FLOW_GROUP_KEY, "fg5")
+ .addPrimitive(ConfigurationKeys.FLOW_NAME_KEY, "fn5").build())
+ .withDescription("Test flow spec 5")
+ .withVersion("Test version 5")
+ .build();
+
+ flowSpec5Updated=
+ FlowSpec.builder(this.uri5)
+ .withConfig(ConfigBuilder.create()
+ .addPrimitive(FLOW_SOURCE_IDENTIFIER_KEY, "sourceUpdated")
+ .addPrimitive(FLOW_DESTINATION_IDENTIFIER_KEY,
"destinationUpdated")
+ .addPrimitive(ConfigurationKeys.JOB_SCHEDULE_KEY, "0 0 9 * * ?
*")
+ .addPrimitive(ConfigurationKeys.FLOW_OWNING_GROUP_KEY,
"owningGroupUpdated")
+ .addPrimitive(ConfigurationKeys.FLOW_RUN_IMMEDIATELY, "true")
+ .addPrimitive("user.to.proxy", "userAUpdated")
Review Comment:
Sure, have kept FLOW_RUN_IMMEDIATELY unupdated,
Issue Time Tracking
-------------------
Worklog Id: (was: 924093)
Time Spent: 40m (was: 0.5h)
> `carbon get flow-configs` search facets not consistently working
> ----------------------------------------------------------------
>
> Key: GOBBLIN-2092
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2092
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Aditya Pratap Singh
> Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The `carbon get flow-configs (search) seems to inconsistently apply search
> facets. it's possible that the facet indices are correctly set up during
> flow creation, but are not properly maintained during flow update. see
> interaction:
> {code:java}
> $ carbon get flow-configs -f prod-lva1 -s war-oh-iceberg | jq -c . | wc -l
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: None; source_identifier:
> war-oh-iceberg; destination_identifier: None; cron_schedule: None;
> run_immediately: None; owning_group: None; start: None; count: None)
> 63
> $ carbon get flow-configs -f prod-lva1 -s war-tl-iceberg | jq -c . | wc -l
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: None; source_identifier:
> war-tl-iceberg; destination_identifier: None; cron_schedule: None;
> run_immediately: None; owning_group: None; start: None; count: None)
> No flows found
> 0
> {code}
> so (at least some) results for sourceIdentifier of `war-oh-iceberg` do show,
> but none do for `war-tl-iceberg`. that's incorrect, because when I instead
> search by user, there are at least two `war-tl-iceberg` flows in `prod-lva1`:
> {code:java}
> $ carbon get flow-configs -f prod-lva1 -u lyndarel | jq -c '{flowGroup:
> .id.flowGroup, flowName: .id.flowName, user: .properties."user.to.proxy",
> between: (.properties."gobblin.flow.sourceIdentifier" + " => " +
> .properties."gobblin.flow.destinationIdentifier")}' | grep tl-iceberg
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: lyndarel; source_identifier:
> None; destination_identifier: None; cron_schedule: None; run_immediately:
> None; owning_group: None; start: None; count: None)
> {"flowGroup":"iceberg_based_openhouse_replication_u_lyndarel","flowName":"copy_to_holdem_replication_course_features","user":"lyndarel","between":"war-tl-iceberg
> => holdem-tl-iceberg"}
> {"flowGroup":"iceberg_based_openhouse_replication_u_lyndarel","flowName":"copy_to_holdem_replication_member_skill_gap","user":"lyndarel","between":"war-tl-iceberg
> => holdem-tl-iceberg"} {code}
> when the user and sourceId constraint are combined, those two no longer show
> up:
> {code:java}
> $ carbon get flow-configs -f prod-lva1 -u lyndarel -s war-tl-iceberg | jq -c
> '{flowGroup: .id.flowGroup, flowName: .id.flowName, user:
> .properties."user.to.proxy", between:
> (.properties."gobblin.flow.sourceIdentifier" + " => " +
> .properties."gobblin.flow.destinationIdentifier")}'
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: lyndarel; source_identifier:
> war-tl-iceberg; destination_identifier: None; cron_schedule: None;
> run_immediately: None; owning_group: None; start: None; count: None)
> No flows found {code}
> the reason I suspect flow update as a possible RC is that I had modified
> these two flows to use that sourceId, when they were originally created with
> another one. e.g. something like:
> {code:java}
> $ carbon update flow -f prod-lva1 -fg
> iceberg_based_openhouse_replication_u_lyndarel -fn
> copy_to_holdem_replication_member_skill_gap
> properties.gobblin.flow.sourceIdentifier=war-tl-iceberg,properties.gobblin.flow.destinationIdentifier=holdem-tl-iceberg{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
