hanghangliu commented on a change in pull request #3252:
URL: https://github.com/apache/gobblin/pull/3252#discussion_r603718830
##########
File path:
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java
##########
@@ -132,6 +136,36 @@ public void publishData(Collection<? extends
WorkUnitState> states) throws IOExc
return newFiles;
}
+ /**
+ * Choose one file from the work unit state. There will be no modification
to the file.
+ * It's used in GMCE writer {@link GobblinMCEWriter} merely for getting the
DB and table name.
+ * @throws IOException
+ */
+ private Map<Path, Metrics> computeDummyFile (State state) throws IOException
{
Review comment:
Used a priority queue to chose file based on max modification_time to
ensure the dummy file is the latest available file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
