phet commented on code in PR #3843: URL: https://github.com/apache/gobblin/pull/3843#discussion_r1416850865
########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/timing/EventTimer.java: ########## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.gobblin.temporal.workflows.timing; + +import java.io.Closeable; + +import org.apache.gobblin.metrics.event.TimingEvent; + + +/** + * A timer that can be used to track the duration of an event. This event differs from the {@link TimingEvent} in that + * this class is not meant to be used outside of {@link io.temporal.workflow.Workflow} code. Review Comment: the Q this doesn't answer is why not leverage `TimingEvent`, rather than inventing something new? ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/timing/TemporalEventTimer.java: ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.temporal.workflows.timing; + +import java.time.Duration; +import java.time.Instant; + +import io.temporal.workflow.Workflow; + +import org.apache.gobblin.metrics.event.EventSubmitter; +import org.apache.gobblin.metrics.event.GobblinEventBuilder; +import org.apache.gobblin.metrics.event.TimingEvent; +import org.apache.gobblin.temporal.workflows.trackingevent.activity.GobblinTrackingEventActivity; + + +/** + * Boiler plate for tracking elapsed time of events that is compatible with {@link Workflow} + * by using activities to record time + */ +public class TemporalEventTimer implements EventTimer { + private final GobblinTrackingEventActivity trackingEventActivity; + private final EventSubmitter eventSubmitter; + private final GobblinEventBuilder eventBuilder; + + private Instant startTime; Review Comment: could this be `final`? also if you put the `getCurrentTime()` call into `Factory`, couldn't we then use `@RequiredArgsConstructor` to eliminate boilerplate? ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/timing/TemporalEventTimer.java: ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.temporal.workflows.timing; + +import java.time.Duration; +import java.time.Instant; + +import io.temporal.workflow.Workflow; + +import org.apache.gobblin.metrics.event.EventSubmitter; +import org.apache.gobblin.metrics.event.GobblinEventBuilder; +import org.apache.gobblin.metrics.event.TimingEvent; +import org.apache.gobblin.temporal.workflows.trackingevent.activity.GobblinTrackingEventActivity; + + +/** + * Boiler plate for tracking elapsed time of events that is compatible with {@link Workflow} + * by using activities to record time + */ +public class TemporalEventTimer implements EventTimer { + private final GobblinTrackingEventActivity trackingEventActivity; + private final EventSubmitter eventSubmitter; + private final GobblinEventBuilder eventBuilder; + + private Instant startTime; + + private TemporalEventTimer(GobblinTrackingEventActivity trackingEventActivity, EventSubmitter eventSubmitter, GobblinEventBuilder eventBuilder) { + this.trackingEventActivity = trackingEventActivity; + this.eventBuilder = eventBuilder; + this.eventSubmitter = eventSubmitter; + this.startTime = getCurrentTime(); + } + + @Override + public void close() { + close(getCurrentTime()); + } + + private void close(Instant endTime) { + this.eventBuilder.addMetadata(EventSubmitter.EVENT_TYPE, TimingEvent.METADATA_TIMING_EVENT); + this.eventBuilder.addMetadata(TimingEvent.METADATA_START_TIME, Long.toString(this.startTime.toEpochMilli())); + this.eventBuilder.addMetadata(TimingEvent.METADATA_END_TIME, Long.toString(endTime.toEpochMilli())); + Duration duration = Duration.between(this.startTime, endTime); + this.eventBuilder.addMetadata(TimingEvent.METADATA_DURATION, Long.toString(duration.toMillis())); + + trackingEventActivity.submitGTE(this.eventSubmitter, this.eventBuilder); + } + + private Instant getCurrentTime() { + return Instant.ofEpochMilli(Workflow.currentTimeMillis()); + } + + public static class Factory { + private final GobblinTrackingEventActivity trackingEventActivity; + private final EventSubmitter eventSubmitter; + + public Factory(GobblinTrackingEventActivity trackingEventActivity, EventSubmitter eventSubmitter) { + this.trackingEventActivity = trackingEventActivity; + this.eventSubmitter = eventSubmitter; + } + + public TemporalEventTimer get(String eventName) { + GobblinEventBuilder eventBuilder = new GobblinEventBuilder(eventName, eventSubmitter.getNamespace()); + return new TemporalEventTimer(this.trackingEventActivity, eventSubmitter, eventBuilder); + } + + /** + * Creates a timer that emits separate events at the start and end of a job + * @return a timer that emits an event at the beginning of the job and a completion event ends at the end of the job + */ + public EventTimer getJobTimer() { + TemporalEventTimer startTimer = get(TimingEvent.LauncherTimings.JOB_START); + Instant jobStartTime = startTimer.startTime; + startTimer.close(Instant.EPOCH); // Job start event contains a stub end time + + return () -> { + TemporalEventTimer endTimer = get(TimingEvent.LauncherTimings.JOB_COMPLETE); + endTimer.startTime = jobStartTime; Review Comment: as suggested, move choice of value to factory method, to permit immutability ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/GobblinTemporalConfigurationKeys.java: ########## @@ -43,6 +43,13 @@ public interface GobblinTemporalConfigurationKeys { String GOBBLIN_TEMPORAL_JOB_LAUNCHER_ARG_PREFIX = GOBBLIN_TEMPORAL_JOB_LAUNCHER_PREFIX + "arg."; + /** + * Suffix for metrics emitted by GobblinTemporalJobLauncher for preventing collisions with prod jobs + * during testing + */ + String GOBBLIN_TEMPORAL_JOB_METRICS_SUFFIX = PREFIX + "job.metrics.suffix"; + String DEFAULT_GOBBLIN_TEMPORAL_JOB_METRICS_SUFFIX = ""; Review Comment: any consequences from having the default not avoid collisions? ########## gobblin-temporal/src/test/java/org/apache/gobblin/temporal/workflows/trackingevent/activity/EventSubmitterTemporalSerializationTest.java: ########## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.gobblin.temporal.workflows.trackingevent.activity; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; + +import org.testng.Assert; +import org.testng.annotations.Test; + +import com.fasterxml.jackson.databind.ObjectMapper; + +import lombok.Builder; +import lombok.extern.slf4j.Slf4j; + +import org.apache.gobblin.configuration.State; +import org.apache.gobblin.instrumented.Instrumented; +import org.apache.gobblin.metrics.MetricContext; +import org.apache.gobblin.metrics.Tag; +import org.apache.gobblin.metrics.event.EventSubmitter; + +import static org.apache.gobblin.instrumented.GobblinMetricsKeys.CLASS_META; + + +/** + * Explicit test for testing deser of EventSubmitter that checks for edge cases where the tests don't exist. This test + * Cannot exist in {@link org.apache.gobblin.metrics} because it tests integration with the gobblin-core-base module. + * + * The reason to place this in the temporal module is because Jackson serialization is only truly critical for + * {@link io.temporal.workflow.Workflow} and {@link io.temporal.activity.Activity} + */ +@Slf4j +public class EventSubmitterTemporalSerializationTest { + private final ObjectMapper objectMapper = new ObjectMapper(); + @Test + public void testGet() throws Exception { + List<Tag<?>> tags = Arrays.asList(new Tag<String>("jobId", "stub")); + State state = new State(); + state.setProp("someState", "stub"); + + EventSubmitter eventSubmitter = TestEventSubmitter.builder() + .callerClass(EventSubmitterTemporalSerializationTest.class) + .state(state) + .namespace("test") + .tags(tags) + .build().get(); + + String bytes = objectMapper.writeValueAsString(eventSubmitter); + Assert.assertTrue(bytes != null && !bytes.isEmpty()); + eventSubmitter = objectMapper.readValue(bytes, EventSubmitter.class); Review Comment: nit: don't overwrite the original ES... maybe call `deserEventSubmitter`? ########## gobblin-metrics-libs/gobblin-metrics-base/src/test/java/org/apache/gobblin/metrics/TagTest.java: ########## @@ -44,4 +53,16 @@ public void testTags() { Assert.assertEquals(projectVersionTag.getKey(), PROJECT_VERSION_KEY); Assert.assertEquals(projectVersionTag.getValue().intValue(), PROJECT_VERSION); } + + @Test + public void testSerde() throws IOException { Review Comment: nice test! ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/timing/TemporalEventTimer.java: ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.temporal.workflows.timing; + +import java.time.Duration; +import java.time.Instant; + +import io.temporal.workflow.Workflow; + +import org.apache.gobblin.metrics.event.EventSubmitter; +import org.apache.gobblin.metrics.event.GobblinEventBuilder; +import org.apache.gobblin.metrics.event.TimingEvent; +import org.apache.gobblin.temporal.workflows.trackingevent.activity.GobblinTrackingEventActivity; + + +/** + * Boiler plate for tracking elapsed time of events that is compatible with {@link Workflow} + * by using activities to record time + */ +public class TemporalEventTimer implements EventTimer { + private final GobblinTrackingEventActivity trackingEventActivity; + private final EventSubmitter eventSubmitter; + private final GobblinEventBuilder eventBuilder; + + private Instant startTime; + + private TemporalEventTimer(GobblinTrackingEventActivity trackingEventActivity, EventSubmitter eventSubmitter, GobblinEventBuilder eventBuilder) { + this.trackingEventActivity = trackingEventActivity; + this.eventBuilder = eventBuilder; + this.eventSubmitter = eventSubmitter; + this.startTime = getCurrentTime(); + } + + @Override + public void close() { + close(getCurrentTime()); + } + + private void close(Instant endTime) { + this.eventBuilder.addMetadata(EventSubmitter.EVENT_TYPE, TimingEvent.METADATA_TIMING_EVENT); + this.eventBuilder.addMetadata(TimingEvent.METADATA_START_TIME, Long.toString(this.startTime.toEpochMilli())); + this.eventBuilder.addMetadata(TimingEvent.METADATA_END_TIME, Long.toString(endTime.toEpochMilli())); + Duration duration = Duration.between(this.startTime, endTime); + this.eventBuilder.addMetadata(TimingEvent.METADATA_DURATION, Long.toString(duration.toMillis())); + + trackingEventActivity.submitGTE(this.eventSubmitter, this.eventBuilder); + } + + private Instant getCurrentTime() { Review Comment: could this be `static`? ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/trackingevent/activity/GobblinTrackingEventActivity.java: ########## @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.temporal.workflows.trackingevent.activity; + +import io.temporal.activity.ActivityInterface; +import io.temporal.activity.ActivityMethod; + +import org.apache.gobblin.metrics.event.EventSubmitter; +import org.apache.gobblin.metrics.event.GobblinEventBuilder; + + +@ActivityInterface +public interface GobblinTrackingEventActivity { Review Comment: nit: some of our other activities are named, with a verb, more like functions. here that might be `SendGobblinTrackingEventActivity`, which informs on what it does w/ GTEs ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/timing/TemporalEventTimer.java: ########## @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.gobblin.temporal.workflows.timing; + +import java.time.Duration; +import java.time.Instant; + +import io.temporal.workflow.Workflow; + +import org.apache.gobblin.metrics.event.EventSubmitter; +import org.apache.gobblin.metrics.event.GobblinEventBuilder; +import org.apache.gobblin.metrics.event.TimingEvent; +import org.apache.gobblin.temporal.workflows.trackingevent.activity.GobblinTrackingEventActivity; + + +/** + * Boiler plate for tracking elapsed time of events that is compatible with {@link Workflow} + * by using activities to record time + */ +public class TemporalEventTimer implements EventTimer { + private final GobblinTrackingEventActivity trackingEventActivity; + private final EventSubmitter eventSubmitter; + private final GobblinEventBuilder eventBuilder; + + private Instant startTime; + + private TemporalEventTimer(GobblinTrackingEventActivity trackingEventActivity, EventSubmitter eventSubmitter, GobblinEventBuilder eventBuilder) { + this.trackingEventActivity = trackingEventActivity; + this.eventBuilder = eventBuilder; + this.eventSubmitter = eventSubmitter; + this.startTime = getCurrentTime(); + } + + @Override + public void close() { + close(getCurrentTime()); + } + + private void close(Instant endTime) { + this.eventBuilder.addMetadata(EventSubmitter.EVENT_TYPE, TimingEvent.METADATA_TIMING_EVENT); + this.eventBuilder.addMetadata(TimingEvent.METADATA_START_TIME, Long.toString(this.startTime.toEpochMilli())); + this.eventBuilder.addMetadata(TimingEvent.METADATA_END_TIME, Long.toString(endTime.toEpochMilli())); + Duration duration = Duration.between(this.startTime, endTime); + this.eventBuilder.addMetadata(TimingEvent.METADATA_DURATION, Long.toString(duration.toMillis())); + + trackingEventActivity.submitGTE(this.eventSubmitter, this.eventBuilder); + } + + private Instant getCurrentTime() { + return Instant.ofEpochMilli(Workflow.currentTimeMillis()); + } + + public static class Factory { + private final GobblinTrackingEventActivity trackingEventActivity; + private final EventSubmitter eventSubmitter; + + public Factory(GobblinTrackingEventActivity trackingEventActivity, EventSubmitter eventSubmitter) { + this.trackingEventActivity = trackingEventActivity; + this.eventSubmitter = eventSubmitter; + } + + public TemporalEventTimer get(String eventName) { Review Comment: nit: `create` for a factory, since you're not accessing a stable instance, but rather generating a new one each time ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/util/nesting/workflow/AbstractNestingExecWorkflowImpl.java: ########## @@ -109,7 +109,7 @@ protected NestingExecWorkflow<WORK_ITEM> createChildWorkflow(final WorkflowAddr String thisWorkflowId = Workflow.getInfo().getWorkflowId(); String childWorkflowId = thisWorkflowId.replaceAll("-[^-]+$", "") + "-" + childAddr; ChildWorkflowOptions childOpts = ChildWorkflowOptions.newBuilder() - .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON) + .setParentClosePolicy(ParentClosePolicy.PARENT_CLOSE_POLICY_TERMINATE) Review Comment: good call! ########## gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/Tag.java: ########## @@ -21,14 +21,15 @@ import java.util.List; import java.util.Map; -import javax.annotation.Nullable; - +import com.fasterxml.jackson.annotation.JsonCreator; import com.google.common.base.Function; import com.google.common.base.Splitter; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; +import javax.annotation.Nullable; Review Comment: isn't this supposed to live right after `java.` imports? ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/helloworld/HelloWorldJobLauncher.java: ########## @@ -44,8 +46,9 @@ * </p> */ @Alpha -@Slf4j public class HelloWorldJobLauncher extends GobblinTemporalJobLauncher { + private static Logger log = Workflow.getLogger(HelloWorldJobLauncher.class); Review Comment: curious: what's the benefit of this over `@Slf4j`, esp. since this is the job launcher, which "invokes" temporal, but is not itself "executed by" temporal? ########## gobblin-temporal/src/main/java/org/apache/gobblin/temporal/workflows/timing/EventTimer.java: ########## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.gobblin.temporal.workflows.timing; + +import java.io.Closeable; + +import org.apache.gobblin.metrics.event.TimingEvent; + + +/** + * A timer that can be used to track the duration of an event. This event differs from the {@link TimingEvent} in that + * this class is not meant to be used outside of {@link io.temporal.workflow.Workflow} code. + * + * It also differs from {@link Closeable} because the close method does not throw any checked exceptions + */ +public interface EventTimer extends Closeable { + void close(); Review Comment: being the same signature of `Closeable.close()` this doesn't really overload it, does it? semantically, what's the relationship you intend here? also, why does `EventTimer` have only `close`, but no method related to it being an event timer? use of the try-with-resources hack for invocation might be clearer as: ``` void submitTiming(); // can this be used in an interface??? @Override default void close() { submitTiming(); } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
