ibuenros commented on a change in pull request #2636: [GOBBLIN-766] Emit
Workunits Created event
URL: https://github.com/apache/incubator-gobblin/pull/2636#discussion_r288294736
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java
##########
@@ -377,6 +378,13 @@ public void apply(JobListener jobListener, JobContext
jobContext)
jobState.setState(JobState.RunningState.COMMITTED);
isWorkUnitsEmpty = true;
return;
+ } else {
+ if (workUnitStream.isSafeToMaterialize()) {
+ this.eventSubmitter.submit(JobEvent.WORK_UNITS_CREATED,
"workUnitsCreated",
+
Long.toString(workUnitStream.getMaterializedWorkUnitCollection().size()));
Review comment:
This call will force work units to be materialized and prevent all
optimizations from streaming them (in terms of memory and speed). Ideally, the
correct way of doing this is to count the work units as we work on them and
emit the event at the end of the run.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
