[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255983#comment-16255983
]
Hudson commented on AMBARI-22444:
-
SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #8404 (See
[https://builds.apache.org/job/Ambari-trunk-Commit/8404/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (part4) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=365c91e92698f6a566b8503486f80c736af79e70])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255852#comment-16255852
]
Hudson commented on AMBARI-22444:
-
FAILURE: Integrated in Jenkins build Ambari-branch-2.6 #464 (See
[https://builds.apache.org/job/Ambari-branch-2.6/464/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (part4) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=72785da88a1c93a12d45f4fdad815e144a1beb5f])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255655#comment-16255655
]
Hudson commented on AMBARI-22444:
-
SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #8402 (See
[https://builds.apache.org/job/Ambari-trunk-Commit/8402/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (part3) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=a393926c5cdae85b567349c89098712d21a5e1ec])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255614#comment-16255614
]
Hudson commented on AMBARI-22444:
-
FAILURE: Integrated in Jenkins build Ambari-branch-2.6 #462 (See
[https://builds.apache.org/job/Ambari-branch-2.6/462/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (part3) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=3279bef5288cc84f89cbd1a170b78f5af276eae3])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255433#comment-16255433
]
Nate Cole commented on AMBARI-22444:
+1 on the new change
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255431#comment-16255431
]
Jonathan Hurley commented on AMBARI-22444:
--
Seems like this patch has issues with non-root and {{tarfile}}:
{code}
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/copy_tarball.py",
line 92, in _prepare_tez_tarball
with closing(tarfile.open(tez_tarball_with_native_lib, "w:gz")) as
new_tez_tarball:
File "/usr/lib64/python2.6/tarfile.py", line 1670, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/lib64/python2.6/tarfile.py", line 1714, in gzopen
fileobj = bltn_open(name, mode + "b")
IOError: [Errno 13] Permission denied:
'/var/lib/ambari-agent/tmp/tez-native-tarball-staging/tez-native.tar.gz'
{code}
Attached [^AMBARI-22444-2.patch] to correct it
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444-2.patch, AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254475#comment-16254475
]
Hudson commented on AMBARI-22444:
-
SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #8399 (See
[https://builds.apache.org/job/Ambari-trunk-Commit/8399/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (part2) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=0936f463042cf5ba195b0461479d8c29c88330dd])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254441#comment-16254441
]
Hudson commented on AMBARI-22444:
-
SUCCESS: Integrated in Jenkins build Ambari-branch-2.6 #459 (See
[https://builds.apache.org/job/Ambari-branch-2.6/459/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (part2) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=a8496eedbf55310bc37e44d0012520ed0e014737])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254238#comment-16254238
]
Hudson commented on AMBARI-22444:
-
FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #8397 (See
[https://builds.apache.org/job/Ambari-trunk-Commit/8397/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (jonathanhurley) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=c67a3246e2121c6977094e10cdb1414f45cf1482])
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
* (edit) ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254084#comment-16254084
]
Hudson commented on AMBARI-22444:
-
FAILURE: Integrated in Jenkins build Ambari-branch-2.6 #457 (See
[https://builds.apache.org/job/Ambari-branch-2.6/457/])
AMBARI-22444 - Add Native Libraries To Tez Tarball (jonathanhurley) (jhurley:
[http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=6e247b3f5c06bb108687e8278f474ed3d6330ade])
* (edit) ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
* (edit)
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-22444) Add Native Libraries To Tez Tarball
[
https://issues.apache.org/jira/browse/AMBARI-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253653#comment-16253653
]
Hadoop QA commented on AMBARI-22444:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12897772/AMBARI-22444.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/Ambari-trunk-test-patch/12674//console
This message is automatically generated.
> Add Native Libraries To Tez Tarball
> ---
>
> Key: AMBARI-22444
> URL: https://issues.apache.org/jira/browse/AMBARI-22444
> Project: Ambari
> Issue Type: Bug
>Affects Versions: 2.6.1
>Reporter: Jonathan Hurley
>Assignee: Jonathan Hurley
>Priority: Blocker
> Fix For: 2.6.1
>
> Attachments: AMBARI-22444.patch
>
>
> As part of the investigation for getting patch upgrades working in Ambari
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop
> native libraries added to it so that they can be detected from the tarball.
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> When Hive commands run, they will attempt to load the native snappy libraries
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to
> missing native libraries for the codec.
> at
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> In order to fix this, the Tez tarball should include the native hadoop
> libraries as well:
> {code}
> ├── tez
> │ ├── lib
> │ │ ├── native
> │ │ │ ├── libhadoop.a
> │ │ │ ├── libhadoop.so -> libhadoop.so.1.0.0
> │ │ │ ├── libhadoop.so.1.0.0
> │ │ │ ├── libhadooppipes.a
> │ │ │ ├── libhadooputils.a
> │ │ │ ├── libhdfs.a
> │ │ │ ├── libhdfs.so -> libhdfs.so.0.0.0
> │ │ │ ├── libhdfs.so.0.0.0
> │ │ │ ├── libsnappy.so -> libsnappy.so.1.1.4
> │ │ │ ├── libsnappy.so.1 -> libsnappy.so.1.1.4
> │ │ │ └── libsnappy.so.1.1.4
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
