This is an automated email from the ASF dual-hosted git repository.
zihanli58 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/gobblin.git
The following commit(s) were added to refs/heads/master by this push:
new ae257acae [HOTFIX] catch exception when fail to compute the iceberg
file metrics (#3909)
ae257acae is described below
commit ae257acae0b321946c8173f0cdf71be7d9fed753
Author: Zihan Li <[email protected]>
AuthorDate: Fri Mar 29 16:33:08 2024 -0700
[HOTFIX] catch exception when fail to compute the iceberg file metrics
(#3909)
* address comments
* use connectionmanager when httpclient is not cloesable
* [GOBBLIN-2024] Improve the performance of GobblinMCEWriter
* not catch exception but just throwing them when failing to get file
metrics
* [hotFix] catch exception when fail to compute the metrics
---------
Co-authored-by: Zihan Li <[email protected]>
---
.../org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java
b/gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java
index 9298b9b49..49979b4ca 100644
---
a/gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java
+++
b/gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/publisher/GobblinMCEPublisher.java
@@ -210,7 +210,13 @@ public class GobblinMCEPublisher extends DataPublisher {
//This means the table is not compatible with iceberg, so return a
dummy metric
return DUMMY_METRICS;
}
- return OrcMetrics.fromInputFile(HadoopInputFile.fromPath(path, conf),
MetricsConfig.getDefault(), mapping);
+ try {
+ return OrcMetrics.fromInputFile(HadoopInputFile.fromPath(path,
conf), MetricsConfig.getDefault(), mapping);
+ } catch (Exception e) {
+ //This means either file is corrupted or the table is not compatible
with iceberg (contains complex union), so return a dummy metric
+ return DUMMY_METRICS;
+ }
+
}
case AVRO: {
try {
