Pavel Stepanov created HDDS-12116:
-------------------------------------
Summary: Customizable Protobuf shaded prefix in
ozonefs-hadoop3-client
Key: HDDS-12116
URL: https://issues.apache.org/jira/browse/HDDS-12116
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Affects Versions: 2.0.0
Reporter: Pavel Stepanov
Assignee: Pavel Stepanov
I tried running Trino together with Ozone over OFS but encountered a Protobuf
shading incompatibility between the [Trino Hadoop
library|https://github.com/trinodb/trino-hadoop-apache/tree/master] and the
Ozone Hadoop client:
{code:java}
2024-12-27T09:09:30.178Z DEBUG hive-hive-0
org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem Ozone URI for OFS
initialization is ofs://ozone-om-1
2024-12-27T09:09:30.183Z WARN hive-hive-0
org.apache.hadoop.ozone.om.helpers.OzoneFSUtils Ignoring ozone.fs.hsync.enabled
= true because HBase enhancements are disallowed. To enable it, set
ozone.client.hbase.enhancements.allowed = true as well.
2024-12-27T09:09:30.184Z DEBUG hive-hive-0
org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem hsyncEnabled = false
2024-12-27T09:09:30.473Z INFO dispatcher-query-3
io.trino.event.QueryMonitor TIMELINE: Query 20241227_090929_00003_x4ukx ::
FINISHED :: elapsed 480ms :: planning 134ms :: waiting 23ms :: scheduling 202ms
:: running 128ms :: finishing 16ms :: begin 2024-12-27T09:09:29.959Z :: end
2024-12-27T09:09:30.641Z INFO hive-hive-0
org.apache.hadoop.hdds.utils.LeakDetector Starting leak detector thread
OzoneClientObject0.
2024-12-27T09:09:30.816Z DEBUG hive-hive-0
org.apache.hadoop.ozone.om.protocolPB.OmTransportFactory Loading OM
transport implementation
org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransportFactory as specified by
configuration.
2024-12-27T09:09:31.241Z DEBUG hive-hive-0
org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM
om1: class
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest
cannot be cast to class io.trino.hadoop.$internal.com.google.protobuf.Message
(org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OMRequest
and io.trino.hadoop.$internal.com.google.protobuf.Message are in unnamed module
of loader io.trino.filesystem.manager.HdfsClassLoader @43fffec2)
2024-12-27T09:09:31.243Z DEBUG hive-hive-0
org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Incrementing OM
proxy index to 0, nodeId: om1
2024-12-27T09:09:33.246Z DEBUG hive-hive-0
org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase Failing over OM
from om1:0 to om1:0
2024-12-27T09:09:33.246Z DEBUG hive-hive-0
org.apache.hadoop.ozone.om.ha.OMFailoverProxyProviderBase RetryProxy: OM
om1: null{code}
The simplest way to resolve this issue is to adjust the ozonefs-hadoop3-client
module, which already shades Protobuf, but currently uses a fixed prefix. I
suggest making the prefix configurable as a separate property to allow
customization if needed.
With the proposed changes, running
{code:java}
mvn clean package -Dproto.shaded.prefix='io.trino.hadoop.\$internal' {code}
inside the ozonefs-hadoop3-client module will properly shade Protobuf and allow
the output JAR to be used with Trino.
Related trino issue: https://github.com/trinodb/trino/discussions/18026
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
