-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51863/
-----------------------------------------------------------

(Updated Sept. 13, 2016, 9:36 p.m.)


Review request for Ambari, Andrew Onischuk, Laszlo Puskas, Sumit Mohanty, and 
Venkat Ranganathan.


Summary (updated)
-----------------

AMBARI-18383 : Oozie spark failed due to incomplete spark share-lib directory


Bugs: AMBARI-18383
    https://issues.apache.org/jira/browse/AMBARI-18383


Repository: ambari


Description
-------

Failed exception:

"Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], 
main() 
threw exception, Missing py4j and/or pyspark zip files. Please add them to the 
lib folder 
or to the Spark sharelib.

org.apache.oozie.action.hadoop.OozieActionConfiguratorException: Missing py4j 
and/or 
pyspark zip files. Please add them to the lib folder or to the Spark sharelib."


Those two files were not present in the share lib
hrt_qa@hn0-3ad6fe:~$ hdfs dfs -ls /user/oozie/share/lib/spark/*.zip
ls: `/user/oozie/share/lib/spark/*.zip': No such file or directory


Diffs
-----

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_service.py
 8d80638 
  ambari-server/src/main/resources/scripts/Ambaripreupload.py 7c8c601 

Diff: https://reviews.apache.org/r/51863/diff/


Testing
-------

Manually tested on sys-prepped cluster. 

Execution log

2016-09-13 21:17:32,590 - Execute[('mv', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark', 
'/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig')] {'sudo': True}
2016-09-13 21:17:32,613 - 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'owner': 'oozie', 
'create_parents': True}
2016-09-13 21:17:32,635 - Creating directory 
Directory['/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] since it doesn't exist.
2016-09-13 21:17:32,746 - Changing owner for 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark from 0 to oozie
2016-09-13 21:17:32,771 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark-orig/oozie-sharelib-spark*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:32,804 - Execute['cp -P /usr/hdp/2.5.1.0-16/spark/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,125 - Execute['find 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/ -type l -delete'] {'user': 'oozie'}
2016-09-13 21:17:33,163 - Execute['rm -f 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark/spark-examples*.jar'] {'user': 
'oozie'}
2016-09-13 21:17:33,220 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.zip 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
2016-09-13 21:17:33,258 - Execute['cp -f 
/usr/hdp/2.5.1.0-16/spark/python/lib/*.jar 
/usr/hdp/2.5.1.0-16/oozie/share/lib/spark'] {'user': 'oozie'}
No jar files found in Spark client python lib.
2016-09-13 21:17:33,293 - HdfsResource['/tmp/oozie/user/oozie/share'] 
{'security_enabled': False, 'hadoop_conf_dir': '/etc/hadoop/conf', 'keytab': 
None, 'source': '/usr/hdp/2.5.1.0-16/oozie/share', 'default_fs': 
'adl://oozie3hadl25.azuredatalakestore.net:443', 'user': 'hdfs', 
'hdfs_resource_ignore_file': 
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': 
{'dfs.webhdfs.enabled': False}, 'kinit_path_local': None, 'principal_name': 
None, 'recursive_chmod': True, 'owner': 'oozie', 'hadoop_bin_dir': 
'/usr/hdp/2.5.1.0-16/hadoop/bin', 'type': 'directory', 'action': 
['create_on_execute'], 'mode': 0755}


Thanks,

Aravindan Vijayan

Reply via email to