[
https://issues.apache.org/jira/browse/YETUS-222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15246284#comment-15246284
]
Colin Patrick McCabe commented on YETUS-222:
--------------------------------------------
A lot of Hadoop's native dependencies are not "generally available... in OS
package managers." For example, if you do {{apt-get install protobuf-dev}},
you are likely to get a version of protocol buffers that doesn't work with
Hadoop. (Hadoop currently needs 2.5.0, not one of the earlier or later
versions which are incompatible.) Similarly, we added snappy support well
before most distributions. If we wait for all Linux distributions to have
something before using it, we're always at least 5 years out of date.
(Side note: Probably, we should have implemented a dependency management system
in the build system, similar to how Maven works, to avoid this kind of issue.)
bq. I'll be honest: I'm debating closing this as won't fix. ISAL should be
getting autodetected during precommit runs now that it is in the Dockerfile
(although it's there in a less than ideal way), but I haven't verified that.
(and if that's not working, that's a bug on the Hadoop side)
The reason why we have the {{-Drequire.foo}} flags is because in the past we've
had a lot of silent breakages where various "foos" have not been found. And so
we've gone for weeks or months without any unit testing on these components.
It's a really bad situation to be in, and a serious shortcoming for Yetus if
it's not fixed.
bq. Adding it here only makes it a requirement which, given the above, is going
to be painful for anyone not using the Dockerfile. Non-Linux but Intel based
platforms (since ISAL is only supported on those) are in an even bigger world
of hurt. There's also the cross platform issues: I wish Hadoop hadn't directly
used ISAL and instead used liberasurecode as a wrapper, but corporate
priorities tend to rule the roost over there. Linux but not Intel based? Yeah,
good luck with that.
User priorities "rule the roost" at Hadoop, and the vast majority of Hadoop's
users are on x86_64 and will benefit greatly from the use of ISAL. Of course,
we would love to make erasure encoding acceleration support cross platform. If
you are interested in that, please open a JIRA on the HADOOP project. In the
meantime, why not simply have Yetus check the build architecture, and add
{{-Drequire.isal}} when appropriate?
> hadoop: add ISA-L support
> -------------------------
>
> Key: YETUS-222
> URL: https://issues.apache.org/jira/browse/YETUS-222
> Project: Yetus
> Issue Type: Improvement
> Components: Test Patch
> Reporter: Allen Wittenauer
> Assignee: Kai Zheng
> Attachments: YETUS-222-v1.patch, YETUS-222-v2.patch
>
>
> Hadoop personality should support building the native code under Jenkins when
> Docker is invoked.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
