Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-15 Thread Jonathan Hurley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/
---

(Updated Nov. 15, 2017, 9:07 a.m.)


Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.


Bugs: AMBARI-22444
https://issues.apache.org/jira/browse/AMBARI-22444


Repository: ambari


Description
---

As part of the investigation for getting patch upgrades working in Ambari 
2.6.1, it was determined that the Tez tarball will need to have the hadoop 
native libraries added to it so that they can be detected from the tarball.

STR:
- Install ZK, MapR, Tez, Yarn, Hive
- Enable a non-LZO codec, like Snappy
- Patch Hive to a new version
- Change the following properties in {{tez-site}}:
-- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
-- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native

When Hive commands run, they will attempt to load the native snappy libraries 
from the Tez tarball and will fail with:
{code}
Caused by: java.io.IOException: Unable to get CompressorType for codec 
(org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing 
native libraries for the codec.
at 
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
{code}

In order to fix this, the Tez tarball should include the native hadoop 
libraries as well:
{code}
??? tez
?   ??? lib
?   ?   ??? native
?   ?   ?   ??? libhadoop.a
?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
?   ?   ?   ??? libhadoop.so.1.0.0
?   ?   ?   ??? libhadooppipes.a
?   ?   ?   ??? libhadooputils.a
?   ?   ?   ??? libhdfs.a
?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
?   ?   ?   ??? libhdfs.so.0.0.0
?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
?   ?   ?   ??? libsnappy.so.1.1.4
{code}


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 03b6213ac4 
  
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
 c682c3e24a 


Diff: https://reviews.apache.org/r/63807/diff/2/

Changes: https://reviews.apache.org/r/63807/diff/1-2/


Testing
---

Manual testing to ensure the file is created and uploaded.

UNIT TESTS PENDING...


Thanks,

Jonathan Hurley



Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-15 Thread Dmitro Lisnichenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191046
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On Nov. 14, 2017, 11:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 11:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Nate Cole

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191022
---


Ship it!




Ship It!

- Nate Cole


On Nov. 14, 2017, 4:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 4:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Dmytro Grinenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)


this is a bit dangerous, as would return the same function and calling it 
would make infinite recursion.

better replace to something like that, by skipping values which r not 
needed:

  _, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
  _, tez_source_file, _, _ = get_tarball_paths("tez")
  
Round brackets do not needed here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)


why we trying to copy directory, if it could be even not exists? 

SHoudk we fail at line 70 then?



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant


- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Dmytro Grinenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191012
---



- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Re: Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Dmytro Grinenko

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/#review191009
---


Fix it, then Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 54 (patched)


this is a bit dangerous, as would return the same function and calling it 
would make infinite recursion.

better replace to something like that, by skipping values which r not 
needed:

  _, mapreduce_source_file, _, _ = get_tarball_paths("mapreduce")
  _, tez_source_file, _, _ = get_tarball_paths("tez")
  
Round brackets do not needed here



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 73 (patched)


why we trying to copy directory, if it could be even not exists? 

SHoudk we fail at line 70 then?



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Lines 219 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant



ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
Line 261 (original), 316 (patched)


I know, they were here, but that is not pythonic. Please remove round 
brackets, they r reducant


- Dmytro Grinenko


On Nov. 14, 2017, 9:21 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63807/
> ---
> 
> (Updated Nov. 14, 2017, 9:21 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22444
> https://issues.apache.org/jira/browse/AMBARI-22444
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> As part of the investigation for getting patch upgrades working in Ambari 
> 2.6.1, it was determined that the Tez tarball will need to have the hadoop 
> native libraries added to it so that they can be detected from the tarball.
> 
> STR:
> - Install ZK, MapR, Tez, Yarn, Hive
> - Enable a non-LZO codec, like Snappy
> - Patch Hive to a new version
> - Change the following properties in {{tez-site}}:
> -- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
> -- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native
> 
> When Hive commands run, they will attempt to load the native snappy libraries 
> from the Tez tarball and will fail with:
> {code}
> Caused by: java.io.IOException: Unable to get CompressorType for codec 
> (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to 
> missing native libraries for the codec.
>   at 
> org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
> {code}
> 
> In order to fix this, the Tez tarball should include the native hadoop 
> libraries as well:
> {code}
> ??? tez
> ?   ??? lib
> ?   ?   ??? native
> ?   ?   ?   ??? libhadoop.a
> ?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadoop.so.1.0.0
> ?   ?   ?   ??? libhadooppipes.a
> ?   ?   ?   ??? libhadooputils.a
> ?   ?   ?   ??? libhdfs.a
> ?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
> ?   ?   ?   ??? libhdfs.so.0.0.0
> ?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
> ?   ?   ?   ??? libsnappy.so.1.1.4
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  03b6213ac4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
>  c682c3e24a 
> 
> 
> Diff: https://reviews.apache.org/r/63807/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to ensure the file is created and uploaded.
> 
> UNIT TESTS PENDING...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>



Review Request 63807: Add Native Libraries To Tez Tarball

2017-11-14 Thread Jonathan Hurley

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63807/
---

Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.


Bugs: AMBARI-22444
https://issues.apache.org/jira/browse/AMBARI-22444


Repository: ambari


Description
---

As part of the investigation for getting patch upgrades working in Ambari 
2.6.1, it was determined that the Tez tarball will need to have the hadoop 
native libraries added to it so that they can be detected from the tarball.

STR:
- Install ZK, MapR, Tez, Yarn, Hive
- Enable a non-LZO codec, like Snappy
- Patch Hive to a new version
- Change the following properties in {{tez-site}}:
-- tez.am.launch.env = LD_LIBRARY=./tezlib/lib/native
-- tez.task.launch.env = LD_LIBRARY=./tezlib/lib/native

When Hive commands run, they will attempt to load the native snappy libraries 
from the Tez tarball and will fail with:
{code}
Caused by: java.io.IOException: Unable to get CompressorType for codec 
(org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing 
native libraries for the codec.
at 
org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.(ExternalSorter.java:217)
{code}

In order to fix this, the Tez tarball should include the native hadoop 
libraries as well:
{code}
??? tez
?   ??? lib
?   ?   ??? native
?   ?   ?   ??? libhadoop.a
?   ?   ?   ??? libhadoop.so -> libhadoop.so.1.0.0
?   ?   ?   ??? libhadoop.so.1.0.0
?   ?   ?   ??? libhadooppipes.a
?   ?   ?   ??? libhadooputils.a
?   ?   ?   ??? libhdfs.a
?   ?   ?   ??? libhdfs.so -> libhdfs.so.0.0.0
?   ?   ?   ??? libhdfs.so.0.0.0
?   ?   ?   ??? libsnappy.so -> libsnappy.so.1.1.4
?   ?   ?   ??? libsnappy.so.1 -> libsnappy.so.1.1.4
?   ?   ?   ??? libsnappy.so.1.1.4
{code}


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 03b6213ac4 
  
ambari-common/src/main/python/resource_management/libraries/functions/tar_archive.py
 c682c3e24a 


Diff: https://reviews.apache.org/r/63807/diff/1/


Testing
---

Manual testing to ensure the file is created and uploaded.

UNIT TESTS PENDING...


Thanks,

Jonathan Hurley