Re: [PR] [GOBBLIN-2227] Make dag action and spec store monitor initialization config driven [gobblin]

Blazer-007 merged PR #4141: URL: https://github.com/apache/gobblin/pull/4141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] [GOBBLIN-2227] Make dag action and spec store monitor initialization config driven [gobblin]

Blazer-007 commented on code in PR #4141: URL: https://github.com/apache/gobblin/pull/4141#discussion_r2361828176 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionChangeMonitor.java: ## @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] [GOBBLIN-2227] Make dag action and spec store monitor initialization config driven [gobblin]

thisisArjit commented on code in PR #4141: URL: https://github.com/apache/gobblin/pull/4141#discussion_r2360524841 ## gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java: ## Review Comment: nit: add `@Override` to setActive m

[PR] [GOBBLIN-XXX] Make dag action and spec store monitor initialization config driven [gobblin]

Blazer-007 opened a new pull request, #4141: URL: https://github.com/apache/gobblin/pull/4141 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the follo

Re: [PR] [GOBBLIN-2227] Make dag action and spec store monitor initialization config driven [gobblin]

codecov-commenter commented on PR #4141: URL: https://github.com/apache/gobblin/pull/4141#issuecomment-3305497945 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4141?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] [GOBBLIN-2226] Construct iceberg data files during commit step [gobblin]

Blazer-007 merged PR #4140: URL: https://github.com/apache/gobblin/pull/4140 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] Revert "[GOBBLIN-2223] Optimise writing of serialised Work Unit to File system" [gobblin]

codecov-commenter commented on PR #4138: URL: https://github.com/apache/gobblin/pull/4138#issuecomment-3251499033 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4138?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] Revert "[GOBBLIN-2223] Optimise writing of serialised Work Unit to File system" [gobblin]

Blazer-007 merged PR #4138: URL: https://github.com/apache/gobblin/pull/4138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] Construct iceberg data files during commit step [gobblin]

Blazer-007 commented on code in PR #4140: URL: https://github.com/apache/gobblin/pull/4140#discussion_r2324561765 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergPartitionCopyableFile.java: ## @@ -0,0 +1,43 @@ +package org.apache.g

[PR] Read / write iceberg data files from hdfs instead of storing it in Workunit [gobblin]

thisisArjit opened a new pull request, #4139: URL: https://github.com/apache/gobblin/pull/4139 …rkunit Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR add

[PR] Construct iceberg data files during commit step [gobblin]

thisisArjit opened a new pull request, #4140: URL: https://github.com/apache/gobblin/pull/4140 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

[PR] Revert "[GOBBLIN-2223] Optimise writing of serialised Work Unit to File system" [gobblin]

thisisArjit opened a new pull request, #4138: URL: https://github.com/apache/gobblin/pull/4138 Reverts apache/gobblin#4133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Add unique id to CopyEntity [gobblin]

Blazer-007 commented on code in PR #4134: URL: https://github.com/apache/gobblin/pull/4134#discussion_r2314350636 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyEntity.java: ## @@ -66,7 +69,7 @@ public class CopyEntity implements HasGuid {

Re: [PR] Add unique id to CopyEntity [gobblin]

Copilot commented on code in PR #4134: URL: https://github.com/apache/gobblin/pull/4134#discussion_r2314368514 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyEntity.java: ## @@ -54,6 +55,8 @@ public class CopyEntity implements HasGuid {

Re: [PR] [GOBBLIN-2225] Use random id for Workunit if recovery helper is not initialised [gobblin]

Blazer-007 merged PR #4134: URL: https://github.com/apache/gobblin/pull/4134 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] Add unique id to CopyEntity [gobblin]

Blazer-007 commented on code in PR #4134: URL: https://github.com/apache/gobblin/pull/4134#discussion_r2314349733 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyEntity.java: ## @@ -66,7 +69,7 @@ public class CopyEntity implements HasGuid {

Re: [PR] [GOBBLIN-2223] Optimise writing of serialised Work Unit to File system [gobblin]

Blazer-007 merged PR #4133: URL: https://github.com/apache/gobblin/pull/4133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] [GOBBLIN-2225] Use random id for Workunit if recovery helper is not initialised [gobblin]

codecov-commenter commented on PR #4134: URL: https://github.com/apache/gobblin/pull/4134#issuecomment-3244694644 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4134?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] [GOBBLIN-2223] Optimise writing of serialised Work Unit to File system [gobblin]

codecov-commenter commented on PR #4133: URL: https://github.com/apache/gobblin/pull/4133#issuecomment-3244486354 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4133?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] [GOBBLIN-2223] Optimise writing of serialised Work Unit to File system [gobblin]

thisisArjit commented on code in PR #4133: URL: https://github.com/apache/gobblin/pull/4133#discussion_r2315043644 ## gobblin-api/src/test/java/org/apache/gobblin/compat/TextSerializerTest.java: ## @@ -49,31 +46,8 @@ public void testSerialize() ByteArrayInputStream bIn =

Re: [PR] [GOBBLIN-2223] Optimise writing of serialised Work Unit to File system [gobblin]

Blazer-007 commented on code in PR #4133: URL: https://github.com/apache/gobblin/pull/4133#discussion_r231434 ## gobblin-api/src/main/java/org/apache/gobblin/compat/hadoop/TextSerializer.java: ## @@ -31,20 +30,21 @@ public class TextSerializer { * Serialize a String usin

Re: [PR] [GOBBLIN-2223] Optimise writing of serialised Work Unit to File system [gobblin]

Copilot commented on code in PR #4133: URL: https://github.com/apache/gobblin/pull/4133#discussion_r2314368875 ## gobblin-api/src/main/java/org/apache/gobblin/compat/hadoop/TextSerializer.java: ## @@ -31,20 +30,21 @@ public class TextSerializer { * Serialize a String using t

Re: [PR] Add unique id to CopyEntity [gobblin]

Blazer-007 commented on PR #4134: URL: https://github.com/apache/gobblin/pull/4134#issuecomment-3242924780 @thisisArjit Add Jira ticket to description and title -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Add unique id to CopyEntity [gobblin]

Blazer-007 commented on code in PR #4134: URL: https://github.com/apache/gobblin/pull/4134#discussion_r2314349733 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/CopyEntity.java: ## @@ -66,7 +69,7 @@ public class CopyEntity implements HasGuid {

[PR] [GOBBLIN-2224]Replace long jobName with a unique and short jobExecutionId [gobblin]

pratapaditya04 opened a new pull request, #4137: URL: https://github.com/apache/gobblin/pull/4137 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the f

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

khandelwal-prateek commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2306072649 ## gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/modules/restli/FlowConfigsV2ResourceHandler

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

khandelwal-prateek commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2306061453 ## gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/modules/restli/FlowConfigsV2ResourceHandler

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

khandelwal-prateek commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2304557997 ## gobblin-metrics-libs/gobblin-metrics/src/main/java/org/apache/gobblin/metrics/ServiceMetricNames.java: ## @@ -96,6 +97,8 @@ public class ServiceMetricName

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

abhishekmjain commented on PR #4136: URL: https://github.com/apache/gobblin/pull/4136#issuecomment-3227315055 Please update the description of the PR with what is being added -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2303175359 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/DagProcessingEngineMetrics.java: ## @@ -163,6 +167,14 @@ public void markDagActi

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

abhishekmjain commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2303098245 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/DagProcessingEngineMetrics.java: ## @@ -163,6 +167,14 @@ public void markD

Re: [PR] [GOBBLIN-2222] Remove unnecessary CopyEntity deserialisation [gobblin]

Blazer-007 merged PR #4132: URL: https://github.com/apache/gobblin/pull/4132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] [GOBBLIN-2221] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

Blazer-007 merged PR #4131: URL: https://github.com/apache/gobblin/pull/4131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300216535 ## gobblin-metrics-libs/gobblin-metrics/src/main/java/org/apache/gobblin/metrics/ServiceMetricNames.java: ## @@ -96,6 +97,8 @@ public class ServiceMetricNames { publ

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

abhishekmjain commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300178657 ## gobblin-metrics-libs/gobblin-metrics/src/main/java/org/apache/gobblin/metrics/ServiceMetricNames.java: ## @@ -96,6 +97,8 @@ public class ServiceMetricNames {

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300213758 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/DagProcessingEngineMetrics.java: ## @@ -163,6 +167,14 @@ public void markDagActi

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300211890 ## gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/modules/restli/FlowConfigsV2ResourceHandler.java:

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300209854 ## gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/modules/restli/FlowConfigsV2ResourceHandler.java:

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300205226 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/DagProcessingEngineMetrics.java: ## @@ -89,6 +91,8 @@ public void registerAllMetr

Re: [PR] [GOBBLIN-2220]: Send Metrics When Flow Spec Already Exists For An Adhoc Flow [gobblin]

aga9900 commented on code in PR #4136: URL: https://github.com/apache/gobblin/pull/4136#discussion_r2300203662 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/task/DagProcessingEngineMetrics.java: ## @@ -163,6 +167,14 @@ public void markDagActi

[PR] [GOBBLIN-2220]: Send Metrics When an adhoc Flow Spec Already Exists For Adhoc Flow (edited) [gobblin]

aga9900 opened a new pull request, #4136: URL: https://github.com/apache/gobblin/pull/4136 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the followin

Re: [PR] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

thisisArjit commented on code in PR #4131: URL: https://github.com/apache/gobblin/pull/4131#discussion_r2299810704 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergPartitionDataset.java: ## @@ -191,8 +192,8 @@ private Map calcSrcFi

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

abhishekmjain merged PR #4135: URL: https://github.com/apache/gobblin/pull/4135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

vsinghal85 commented on code in PR #4135: URL: https://github.com/apache/gobblin/pull/4135#discussion_r229670 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/Task.java: ## @@ -329,17 +331,37 @@ private void computeAndUpdateTaskDataQuality() { try {

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

Blazer-007 commented on code in PR #4135: URL: https://github.com/apache/gobblin/pull/4135#discussion_r2296983135 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/Task.java: ## @@ -329,17 +331,37 @@ private void computeAndUpdateTaskDataQuality() { try {

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

codecov-commenter commented on PR #4135: URL: https://github.com/apache/gobblin/pull/4135#issuecomment-3218655138 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4135?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

Blazer-007 commented on code in PR #4131: URL: https://github.com/apache/gobblin/pull/4131#discussion_r2296970879 ## gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergOverwritePartitionsStepTest.java: ## @@ -53,15 +54,23 @@ public void

Re: [PR] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

Blazer-007 commented on code in PR #4131: URL: https://github.com/apache/gobblin/pull/4131#discussion_r2296970081 ## gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergOverwritePartitionsStepTest.java: ## @@ -53,15 +54,23 @@ public void

Re: [PR] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

Blazer-007 commented on code in PR #4131: URL: https://github.com/apache/gobblin/pull/4131#discussion_r2296961898 ## gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergPartitionDataset.java: ## @@ -191,8 +192,8 @@ private Map calcSrcFil

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

vsinghal85 commented on code in PR #4135: URL: https://github.com/apache/gobblin/pull/4135#discussion_r2296257754 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/Task.java: ## @@ -334,9 +334,7 @@ private void computeAndUpdateTaskDataQuality() { Log.warn(

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

vsinghal85 commented on code in PR #4135: URL: https://github.com/apache/gobblin/pull/4135#discussion_r2296257754 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/Task.java: ## @@ -334,9 +334,7 @@ private void computeAndUpdateTaskDataQuality() { Log.warn(

Re: [PR] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

codecov-commenter commented on PR #4131: URL: https://github.com/apache/gobblin/pull/4131#issuecomment-3214612551 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4131?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] DQ handling for no bytes read scenarios and metrics for bytes read/written [gobblin]

abhishekmjain commented on code in PR #4135: URL: https://github.com/apache/gobblin/pull/4135#discussion_r2292643985 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/Task.java: ## @@ -334,9 +334,7 @@ private void computeAndUpdateTaskDataQuality() { Log.wa

[PR] Change iceberg data files from byte[] to list of base64 encoded string [gobblin]

thisisArjit opened a new pull request, #4131: URL: https://github.com/apache/gobblin/pull/4131 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

[PR] Optimise writing of serialised Work Unit to File system [gobblin]

thisisArjit opened a new pull request, #4133: URL: https://github.com/apache/gobblin/pull/4133 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

[PR] Dq handling null [gobblin]

vsinghal85 opened a new pull request, #4135: URL: https://github.com/apache/gobblin/pull/4135 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the follo

[PR] Add unique id to CopyEntity [gobblin]

thisisArjit opened a new pull request, #4134: URL: https://github.com/apache/gobblin/pull/4134 …son string Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR

[PR] Remove unnecessary CopyEntity deserialisation [gobblin]

thisisArjit opened a new pull request, #4132: URL: https://github.com/apache/gobblin/pull/4132 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

Re: [PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

abhishekmjain merged PR #4130: URL: https://github.com/apache/gobblin/pull/4130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

Re: [PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

codecov-commenter commented on PR #4130: URL: https://github.com/apache/gobblin/pull/4130#issuecomment-3184098409 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4130?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

aga9900 commented on code in PR #4130: URL: https://github.com/apache/gobblin/pull/4130#discussion_r2273464978 ## gobblin-service/src/test/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobSchedulerTest.java: ## @@ -120,6 +117,70 @@ public void testIsNextRunWith

Re: [PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

abhishekmjain commented on code in PR #4130: URL: https://github.com/apache/gobblin/pull/4130#discussion_r2273040627 ## gobblin-service/src/test/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobSchedulerTest.java: ## @@ -120,6 +117,70 @@ public void testIsNextR

Re: [PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

aga9900 commented on code in PR #4130: URL: https://github.com/apache/gobblin/pull/4130#discussion_r2272989652 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java: ## @@ -698,8 +698,13 @@ public void executeImpl(JobExecut

Re: [PR] [GOBBLIN-2217] Emit metrics if dag act throws exception [gobblin]

abhishekmjain merged PR #4129: URL: https://github.com/apache/gobblin/pull/4129 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

Re: [PR] [GOBBLIN-2217] Emit metrics if dag act throws exception [gobblin]

thisisArjit commented on code in PR #4129: URL: https://github.com/apache/gobblin/pull/4129#discussion_r2272897692 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProc.java: ## @@ -79,7 +79,12 @@ public final void process(DagManagementSt

Re: [PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

abhishekmjain commented on code in PR #4130: URL: https://github.com/apache/gobblin/pull/4130#discussion_r2272690150 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java: ## @@ -698,8 +698,13 @@ public void executeImpl(Job

[PR] [GOBBLIN-2218] Fix Bug For Executions Failing With Only One Schedule [gobblin]

aga9900 opened a new pull request, #4130: URL: https://github.com/apache/gobblin/pull/4130 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the followin

Re: [PR] [GOBBLIN-XXX] Emit metrics if dag act throws exception [gobblin]

abhishekmjain commented on code in PR #4129: URL: https://github.com/apache/gobblin/pull/4129#discussion_r2269489029 ## gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProc.java: ## @@ -79,7 +79,12 @@ public final void process(DagManagement

[PR] [GOBBLIN-XXX] Emit metrics if dag act throws exception [gobblin]

thisisArjit opened a new pull request, #4129: URL: https://github.com/apache/gobblin/pull/4129 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

Re: [PR] [GOBBLIN-2216] Update job status before adding REEVALUATE dag action [gobblin]

abhishekmjain merged PR #4128: URL: https://github.com/apache/gobblin/pull/4128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

Re: [PR] add shouldEvaluate data quality check at dataset level [gobblin]

Blazer-007 merged PR #4127: URL: https://github.com/apache/gobblin/pull/4127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] add shouldEvaluate data quality check at dataset level [gobblin]

Blazer-007 commented on code in PR #4127: URL: https://github.com/apache/gobblin/pull/4127#discussion_r2262042630 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java: ## @@ -94,13 +94,17 @@ public Void call() metricContext = Instrumented.getMe

Re: [PR] add shouldEvaluate data quality check at dataset level [gobblin]

vsinghal85 commented on code in PR #4127: URL: https://github.com/apache/gobblin/pull/4127#discussion_r2262036446 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java: ## @@ -94,13 +94,17 @@ public Void call() metricContext = Instrumented.getMe

Re: [PR] add shouldEvaluate data quality check at dataset level [gobblin]

Blazer-007 commented on code in PR #4127: URL: https://github.com/apache/gobblin/pull/4127#discussion_r2262027352 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java: ## @@ -94,13 +94,17 @@ public Void call() metricContext = Instrumented.getMe

Re: [PR] add shouldEvaluate data quality check at dataset level [gobblin]

Blazer-007 commented on code in PR #4127: URL: https://github.com/apache/gobblin/pull/4127#discussion_r2262024460 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java: ## @@ -94,13 +94,17 @@ public Void call() metricContext = Instrumented.getMe

Re: [PR] [GOBBLIN-2215] Send flow event if flow deadline is crossed [gobblin]

abhishekmjain merged PR #4126: URL: https://github.com/apache/gobblin/pull/4126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

[PR] Update job status before adding REEVALUATE dag action [gobblin]

thisisArjit opened a new pull request, #4128: URL: https://github.com/apache/gobblin/pull/4128 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

[PR] add shouldEvaluate data quality check at dataset level [gobblin]

vsinghal85 opened a new pull request, #4127: URL: https://github.com/apache/gobblin/pull/4127 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the follo

[PR] [GOBBLIN-XXX] Send flow event if flow deadline is crossed [gobblin]

thisisArjit opened a new pull request, #4126: URL: https://github.com/apache/gobblin/pull/4126 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the foll

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

Blazer-007 merged PR #4113: URL: https://github.com/apache/gobblin/pull/4113 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.apa

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

khandelwal-prateek commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2253112881 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/LocalTaskStateTracker.java: ## @@ -89,8 +90,11 @@ public void onTaskRunCompletion(Task task

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

khandelwal-prateek commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2252021542 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/Task.java: ## @@ -942,7 +970,14 @@ public void commit() { if (failedForkIds.size() ==

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

khandelwal-prateek commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2251957581 ## gobblin-core/src/main/java/org/apache/gobblin/qualitychecker/task/TaskLevelPolicyChecker.java: ## @@ -28,20 +29,13 @@ * executes each one, and then stor

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

vsinghal85 commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2244838065 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/SafeDatasetCommit.java: ## @@ -90,6 +94,14 @@ public Void call() metricContext = Instrumented.getMet

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

vsinghal85 commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2244656168 ## gobblin-core/src/main/java/org/apache/gobblin/qualitychecker/task/TaskLevelPolicyChecker.java: ## @@ -28,20 +29,13 @@ * executes each one, and then stores the o

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

vsinghal85 commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2244505067 ## gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java: ## @@ -613,6 +629,11 @@ private boolean checkDataQuality(Optional schema) TaskLeve

Re: [PR] [GOBBLIN-2204] Data quality change [gobblin]

khandelwal-prateek commented on code in PR #4113: URL: https://github.com/apache/gobblin/pull/4113#discussion_r2243350309 ## gobblin-core/src/main/java/org/apache/gobblin/policies/size/FileSizePolicy.java: ## @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] [GOBBLIN-2211] Insert Final Error Classification entry to issues table in DB [gobblin]

abhishekmjain merged PR #4125: URL: https://github.com/apache/gobblin/pull/4125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

[PR] [GOBBLIN-2211] Insert Final Error Classification entry to issues table in DB [gobblin]

NamsB7 opened a new pull request, #4125: URL: https://github.com/apache/gobblin/pull/4125 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the following

Re: [PR] [GOBBLIN-2210] Wrap temporal heartbeat runnable [gobblin]

codecov-commenter commented on PR #4119: URL: https://github.com/apache/gobblin/pull/4119#issuecomment-3126319073 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/4119?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&

Re: [PR] [GOBBLIN-2210] Wrap temporal heartbeat runnable [gobblin]

Blazer-007 commented on code in PR #4119: URL: https://github.com/apache/gobblin/pull/4119#discussion_r2235460111 ## gobblin-utility/src/main/java/org/apache/gobblin/util/ExecutorsUtils.java: ## @@ -162,6 +165,27 @@ public static Runnable loggingDecorator(Runnable runnable) {

Re: [PR] [GOBBLIN-2210] Wrap temporal heartbeat runnable [gobblin]

Blazer-007 commented on code in PR #4119: URL: https://github.com/apache/gobblin/pull/4119#discussion_r2235436944 ## gobblin-utility/src/main/java/org/apache/gobblin/util/ExecutorsUtils.java: ## @@ -162,6 +165,27 @@ public static Runnable loggingDecorator(Runnable runnable) {

Re: [PR] [GOBBLIN-2209] Emit GaaS Executor Otel Metrics [gobblin]

Blazer-007 commented on code in PR #4118: URL: https://github.com/apache/gobblin/pull/4118#discussion_r2234996765 ## gobblin-metrics-libs/gobblin-metrics/src/main/java/org/apache/gobblin/metrics/opentelemetry/GaaSOpenTelemetryMetricsConstants.java: ## @@ -0,0 +1,40 @@ +/* + * Li

Re: [PR] [GOBBLIN-2209] Emit GaaS Executor Otel Metrics [gobblin]

Blazer-007 commented on code in PR #4118: URL: https://github.com/apache/gobblin/pull/4118#discussion_r223474 ## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/launcher/ExecuteGobblinJobLauncher.java: ## @@ -96,8 +103,21 @@ public void submitJob(List workunit

Re: [PR] [GOBBLIN-2209] Emit GaaS Executor Otel Metrics [gobblin]

Blazer-007 commented on code in PR #4118: URL: https://github.com/apache/gobblin/pull/4118#discussion_r2234786983 ## gobblin-metrics-libs/gobblin-metrics/src/main/java/org/apache/gobblin/metrics/opentelemetry/OpenTelemetryInstrumentation.java: ## @@ -0,0 +1,167 @@ +/* + * Licens

Re: [PR] [GOBBLIN-2209] Emit GaaS Executor Otel Metrics [gobblin]

Blazer-007 commented on code in PR #4118: URL: https://github.com/apache/gobblin/pull/4118#discussion_r2234744355 ## gobblin-metrics-libs/gobblin-metrics/src/main/java/org/apache/gobblin/metrics/opentelemetry/GaaSOpenTelemetryMetrics.java: ## @@ -0,0 +1,40 @@ +/* + * Licensed to

Re: [PR] [GOBBLIN-2210] Wrap temporal heartbeat runnable [gobblin]

thisisArjit commented on code in PR #4119: URL: https://github.com/apache/gobblin/pull/4119#discussion_r2234728408 ## gobblin-utility/src/main/java/org/apache/gobblin/util/ExecutorsUtils.java: ## @@ -162,6 +165,27 @@ public static Runnable loggingDecorator(Runnable runnable) {

Re: [PR] [GOBBLIN-2213] Propagate file system job properties to Work unit metadata [gobblin]

abhishekmjain merged PR #4124: URL: https://github.com/apache/gobblin/pull/4124 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@gobblin.

Re: [PR] [GOBBLIN-2213] Propagate file system job properties to Work unit metadata [gobblin]

thisisArjit commented on code in PR #4124: URL: https://github.com/apache/gobblin/pull/4124#discussion_r2234629453 ## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/ddm/workflow/impl/ExecuteGobblinWorkflowImpl.java: ## @@ -85,6 +85,9 @@ public ExecGobblinStats execut

Re: [PR] [GOBBLIN-2210] Wrap temporal heartbeat runnable [gobblin]

abhishekmjain commented on code in PR #4119: URL: https://github.com/apache/gobblin/pull/4119#discussion_r2234616990 ## gobblin-utility/src/main/java/org/apache/gobblin/util/ExecutorsUtils.java: ## @@ -162,6 +165,27 @@ public static Runnable loggingDecorator(Runnable runnable)

  1   2   3   4   5   6   7   8   9   10   >