Hi, > However, up to now, we could use these artifacts with Hadoop 3.0 and they > worked. (Or we just didn't hit the incompatibilities in our test suite.)
I have a question reg Phoenix 5.x supporting Hadoop 3 only. I believe this means Phoenix 5.x cannot work with HBase 2.x that is compiled with Hadoop 2.x (HBase 2.x will have Hadoop 2 jars in classpath whereas Phoenix 5.x requires Hadoop 3 jars). Is this correct? If this is correct, shall we not support Hadoop 2 because HBase 2 still has default Hadoop profile: 2.0? Moreover, what is the root cause behind Phoenix 5.x only supporting Hadoop 3.x? As per the compatibility guidelines [1], HBase 2.3 is fully compatible with Hadoop 2.10.x. Apologies for these questions might be repeated ones but somehow I still could not trace back the root cause. 1. https://hbase.apache.org/book.html#basic.prerequisites On 2020/07/07 08:39:23, Istvan Toth <[email protected]> wrote: > Hi! > > (https://issues.apache.org/jira/browse/PHOENIX-5993 is open for this, but I > am repeating it here for greater visibility) > > The HBase artifacts downloadable from maven central are built with Hadoop > 2.x > > However, up to now, we could use these artifacts with Hadoop 3.0 and they > worked. (Or we just didn't hit the incompatibilities in our test suite.) > > This seems to have changed with 2.2.5, as the public maven artifact doesn't > work with Hadoop 3.0.3 or 3.1.2 . > > This is a known issue in HBase, and documented, but this means that > > Any client JAR we'd build with Hbase 2.2.5 would have the same problem. > We cannot run our tests with Hbase 2.2.5 > HBase's suggested solution is to rebuild HBase with the Hadoop version used > in the cluster, and use that. This, however, doesn't fit into our test or > distribution process. > > I'm looking for your input on how to fix this. > > The options I could think of are: > > - Ask HBase to fix it for us (it doesn't seem to be a priority) > - HBase could publish artifacts for Hadoop3 > - Or could try to stick the common subset of Hadoop APIs > - Re-publish the official HBase releases built with Hadoop 3 under the > org.apache.phoenix groupId > - Hack up our build system to rebuild HBase from source, and use that for > Phoenix > > None of these are particularly compelling, but I expect that this will have > to be solved somehow if we want to keep up with future Hadoop releases. > > Looking forward to hearing your ideas and opinion on this > > Istvan >
