Most of the problems are downstream applications using Hadoop's private
APIs.
Tez:
17:08:38 2020/04/16 00:08:38 INFO : [ERROR] COMPILATION ERROR :
17:08:38 2020/04/16 00:08:38 INFO : [INFO]
-------------------------------------------------------------
17:08:38 2020/04/16 00:08:38 INFO : [ERROR]
/grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[757,45]
incompatible types: com.google.protobuf.ByteString cannot be converted
to org.apache.hadoop.thirdparty.protobuf.ByteString
17:08:38 2020/04/16 00:08:38 INFO : [INFO] 1 error
Tez keeps track of job tokens internally.
The change would look like this:
private void recordJobShuffleInfo(JobID jobId, String user,
Token<JobTokenIdentifier> jobToken) throws IOException {
if (stateDb != null) {
TokenProto tokenProto = ProtobufHelper.protoFromToken(jobToken);
/*TokenProto tokenProto = TokenProto.newBuilder()
.setIdentifier(ByteString.copyFrom(jobToken.getIdentifier()))
.setPassword(ByteString.copyFrom(jobToken.getPassword()))
.setKind(jobToken.getKind().toString())
.setService(jobToken.getService().toString())
.build();*/
JobShuffleInfoProto proto = JobShuffleInfoProto.newBuilder()
.setUser(user).setJobToken(tokenProto).build();
try {
stateDb.put(bytes(jobId.toString()), proto.toByteArray());
} catch (DBException e) {
throw new IOException("Error storing " + jobId, e);
}
}
addJobToken(jobId, user, jobToken);
}
HBase:
1. HBASE-23833 <https://issues.apache.org/jira/browse/HBASE-23833> (this
is recently fixed in the master branch)
2.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hbase-server: Compilation failure:
Compilation failure:
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[361,44]
cannot access org.apache.hadoop.thirdparty.protobuf.MessageOrBuilder
[ERROR] class file for
org.apache.hadoop.thirdparty.protobuf.MessageOrBuilder not found
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[362,14]
cannot access org.apache.hadoop.thirdparty.protobuf.GeneratedMessageV3
[ERROR] class file for
org.apache.hadoop.thirdparty.protobuf.GeneratedMessageV3 not found
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[366,16]
cannot access org.apache.hadoop.thirdparty.protobuf.ByteString
[ERROR] class file for
org.apache.hadoop.thirdparty.protobuf.ByteString not found
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[375,12]
cannot find symbol
[ERROR] symbol: method
writeDelimitedTo(org.apache.hbase.thirdparty.io.netty.buffer.ByteBufOutputStream)
[ERROR] location: variable proto of type
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.DataTransferEncryptorMessageProto
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[702,81]
incompatible types:
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.DataTransferEncryptorMessageProto
cannot be converted to com.google.protobuf.MessageLite
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java:[314,66]
incompatible types:
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.BlockOpResponseProto
cannot be converted to com.google.protobuf.MessageLite
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java:[330,81]
cannot access org.apache.hadoop.thirdparty.protobuf.ProtocolMessageEnum
[ERROR] class file for
org.apache.hadoop.thirdparty.protobuf.ProtocolMessageEnum not found
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java:[380,10]
cannot find symbol
[ERROR] symbol: method
writeDelimitedTo(org.apache.hbase.thirdparty.io.netty.buffer.ByteBufOutputStream)
[ERROR] location: variable proto of type
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java:[422,77]
cannot access org.apache.hadoop.thirdparty.protobuf.Descriptors
[ERROR] class file for
org.apache.hadoop.thirdparty.protobuf.Descriptors not found
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutput.java:[323,64]
incompatible types:
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.PipelineAckProto
cannot be converted to com.google.protobuf.MessageLite
[ERROR]
/Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/SyncReplicationWALProvider.java:[209,68]
invalid method reference
[ERROR] non-static method get() cannot be referenced from a
static context
Ozone:
17:01:19 2020/04/16 00:01:19 INFO : [ERROR] COMPILATION ERROR :
17:01:19 2020/04/16 00:01:19 INFO : [INFO]
-------------------------------------------------------------
17:01:19 2020/04/16 00:01:19 INFO : [ERROR]
/grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java:[110,47]
incompatible types: com.google.protobuf.ServiceException cannot be
converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
17:01:19 2020/04/16 00:01:19 INFO : [ERROR]
/grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java:[116,47]
incompatible types: com.google.protobuf.ServiceException cannot be
converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
17:01:19 2020/04/16 00:01:19 INFO : [INFO] 2 errors
There's another error where Ozone uses the Hadoop RPC framework which uses
the hadoop.thirdparty protobuf.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-hdds-container-service: Compilation
failure
[ERROR]
/Users/weichiu/sandbox/ozone/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java:[103,41]
incompatible types: com.google.protobuf.BlockingService cannot be converted
to org.apache.hadoop.thirdparty.protobuf.BlockingService
BlockingService scmDatanodeService =
StorageContainerDatanodeProtocolService.
newReflectiveBlockingService(
new StorageContainerDatanodeProtocolServerSideTranslatorPB(
server, Mockito.mock(ProtocolMessageMetrics.class)));
Ratis probably breaks as well since it depends on the Hadoop RPC framework
too.
On Tue, Apr 28, 2020 at 10:58 PM Vinayakumar B <vinayakum...@apache.org>
wrote:
> hi Wei-Chiu,
>
> Can you elaborate on what failures you are facing related to relocated
> protobuf classes.. ?
>
> IFAIK, if the issue with location of protobuf classes, still old jar
> protobuf-2.5.0.jar will be available in classpath. So downstream depending
> on 2.5.0 version of protobuf still be able to access them.
>
> -vinay
>
> On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, <weic...@cloudera.com>
> wrote:
>
>> I'm sorry for coming to this late. I missed this message. It should have
>> been a DISCUSS thread rather than NOTICE.
>>
>> Looks like this is inevitable. But we should make the downstream
>> developers aware & make the update easier. As long as it is stated clearly
>> how to update the code to support Hadoop 3.3, I am okay with that.
>>
>> Here's what I suggest:
>> (1) label the jira incompatible (just updated the jira) and updated the
>> release note to tell app developer how to update.
>> (2) declare ProtobufHelper a public API HADOOP-17019
>> <https://issues.apache.org/jira/browse/HADOOP-17019>
>>
>> Tez doesn't use the removed Token API, but there's code that breaks with
>> the relocated protobuf class. The ProtobufHelper API will make this
>> transition much easier.
>>
>> Other downstreamers that break with the relocated protobuf include: Ozone
>> and HBase. but neither of them use the removed Token API.
>>
>>
>> On Wed, Jan 8, 2020 at 4:40 AM Vinayakumar B <vinayakum...@apache.org>
>> wrote:
>>
>>> Hi All,
>>>
>>> This mail is to notify about the Removal of following public APIs from
>>> Hadoop Common.
>>>
>>> ClassName: org.apache.hadoop.security.token.Token
>>> APIs:
>>> public Token(TokenProto tokenPB);
>>> public TokenProto toTokenProto();
>>>
>>> Reason: These APIs are having Generated protobuf classes in the
>>> signature. Right now due to protobuf upgrade in trunk (soon to be 3.3.0
>>> release) these APIs are breaking the downstream builds, even though
>>> downstreams dont use these APIs (just Loading Token class). Downstreams
>>> are
>>> still referencing having older version (2.5.0) of protobuf, hence build
>>> is
>>> being broken.
>>>
>>> These APIs were added for the internal purpose(HADOOP-12563), to
>>> support serializing tokens using protobuf in UGI Credentials.
>>> Same purpose can be achieved using the Helper classes without introducing
>>> protobuf classes in API signatures.
>>>
>>> Token.java is marked as Evolving, so I believe APIs can be changed
>>> whenever
>>> absolute necessary.
>>>
>>> Jira https://issues.apache.org/jira/browse/HADOOP-16621 has been
>>> reported to solve downstream build failure.
>>>
>>> So since this API was added for internal purpose easy approach to solve
>>> this is to remove APIs and use helper classes. Otherwise, as mentioned in
>>> HADOOP-16621, workaround will add unnecessary codes to be maintained.
>>>
>>> If anyone using these APIs outside hadoop project accidentally, please
>>> reply to this mail immediately.
>>>
>>> If no objection by next week, will go ahead with removal of above said
>>> APIs
>>> in HADOOP-16621.
>>>
>>> -Vinay
>>>
>>
