-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64348/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-22598
https://issues.apache.org/jira/browse/AMBARI-22598
Repository: ambari
Description
-------
When Pig (and other hadoop clients) are upgraded by themselves, they need to
create jobs with versions of the LZO libraries which match _their_ versions.
*STR*
# Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version:
2.6.1.0-129 (HDFS has lzo enabled)
# Upgrade Ambari to Target Version: 2.6.1.0-84
# Perform full stack upgrade to 2.6.4.0-55
# Perform 4th digit PU of some services in batches (see attached for list)
# Run Pig service check
*Result*
```
Caused by: java.lang.IllegalArgumentException: Compression codec
com.hadoop.compression.lzo.LzoCodec not found.
at
org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
at
org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:180)
at
org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java:59)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:399)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:265)
... 28 more
Caused by: java.lang.ClassNotFoundException: Class
com.hadoop.compression.lzo.LzoCodec not found
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2147)
at
org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132)
... 32 more
2017-12-04 12:07:30,322 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2118: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
Details at logfile: /home/ambari-qa/pig_1512389247042.log
2017-12-04 12:07:30,338 [main] INFO org.apache.pig.Main - Pig script completed
in 3 seconds and 457 milliseconds (3457 ms)
2017-12-04 12:07:30,338 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down
thread pool
```
Diffs
-----
ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout.py
f2c3c18077
ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig.py
83f70488a3
ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/setup_spark.py
426509cb0e
ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/setup_spark.py
82b6c63d6d
Diff: https://reviews.apache.org/r/64348/diff/1/
Testing
-------
OK
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:29 min
[INFO] Finished at: 2017-12-05T13:27:57-05:00
[INFO] Final Memory: 21M/619M
[INFO] ------------------------------------------------------------------------
Thanks,
Jonathan Hurley
