ZihanLi58 opened a new pull request, #3693: URL: https://github.com/apache/gobblin/pull/3693
Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR" - https://issues.apache.org/jira/browse/GOBBLIN-1830 ### Description - [ ] Here are some details about my PR, including screenshots (if applicable): Currently, we rely on the flushing event to indicate when a container transition occurs. However, if the ingestion process fails to successfully flush the data, we won't know which container is responsible. This becomes problematic when the pipeline restarts, as we won't be able to identify the root cause without manually reviewing the logs of thousands of containers. To avoid this issue, we need to develop a more reliable way of tracking container transitions that doesn't rely solely on the flushing event. The change for this PR is to emit the event when the extractor is firstly initialized ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Unit test, and print out the event information: 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: jobName 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: KafkaHdfsStreamingTrackingOrcTest 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: helixInstance 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: GobblinYarnTaskRunner_1 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: taskAttemptId 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: GobblinYarnTaskRunner_1 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: kafkaTopic 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: PremiumInsightsNotableAlumniImpressionEvent 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: dataset.urn 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: clusterIdentifier 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: holdem 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: construct 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: Extractor 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: metricContextName 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: org.apache.gobblin.prototype.kafka.KafkaAvroBinaryStreamingExtractor.510352353 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: jobId 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: job_KafkaHdfsStreamingTrackingOrcTest_1683594035964 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: helixTaskId 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: b5ef4c8b-7b3d-47fb-9380-d92c11509050 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: partition 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: 0 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: metricContextID 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: 7355f230-dad9-450f-b8e4-4e4c6a0e0b87 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: helixJobId 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: job_KafkaHdfsStreamingTrackingOrcTest_1683594035964_job_KafkaHdfsStreamingTrackingOrcTest_1683594035964 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: topic 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: PremiumInsightsNotableAlumniImpressionEvent 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: containerNode 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: xxxx.xxx.xx.xx (remove this info as it's internal specific) 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: containerId 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: container_e32_1683322902150_274389_01_000003 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: class 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: org.apache.gobblin.prototype.kafka.KafkaAvroBinaryStreamingExtractor 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$ key: taskId 2023-05-08 18:03:11 PDT INFO [TaskStateModelFactory-task_thread-0] org.apache.gobblin.metrics.MetricContext - $$$value: task_KafkaHdfsStreamingTrackingOrcTest_1683594035964_0 ### Commits - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 2. Subject is limited to 50 characters 3. Subject does not end with a period 4. Subject uses the imperative mood ("add", not "adding") 5. Body wraps at 72 characters 6. Body explains "what" and "why", not "how" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
