timyuer commented on PR #3365:
URL: https://github.com/apache/ambari/pull/3365#issuecomment-1272229048
Success run MR check after update /etc/passwd.
```bash
stderr:
None
stdout:
2022-10-08 04:44:09,626 - Stack Feature Version Info: Cluster Stack=3.2.0,
Command Stack=None, Command Version=None -> 3.2.0
2022-10-08 04:44:09,627 - Skipping get_stack_version since
/usr/bin/distro-select is not yet available
2022-10-08 04:44:09,632 - HdfsResource['/user/ambari-qa']
{'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': [EMPTY],
'dfs_type': 'HDFS', 'default_fs': 'hdfs://ambari-agent-01:8020',
'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user':
'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/etc/hadoop/conf', 'type':
'directory', 'action': ['create_on_execute'], 'immutable_paths':
[u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0770}
2022-10-08 04:44:09,634 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmpAV7s9y 2>/tmp/tmpszfLHv''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:09,687 - call returned (0, '')
2022-10-08 04:44:09,688 - get_user_call_output returned (0,
u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16388,"group":"hdfs","length":0,"modificationTime":1665204196800,"owner":"ambari-qa","pathSuffix":"","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200',
u'')
2022-10-08 04:44:09,689 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmpRO4Fqu 2>/tmp/tmpS00zD1''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:09,807 - call returned (0, '')
2022-10-08 04:44:09,807 - get_user_call_output returned (0,
u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16388,"group":"hdfs","length":0,"modificationTime":1665204196800,"owner":"ambari-qa","pathSuffix":"","permission":"770","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200',
u'')
2022-10-08 04:44:09,808 - HdfsResource['/user/ambari-qa/mapredsmokeinput']
{'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': [EMPTY],
'source': '/etc/passwd', 'dfs_type': 'HDFS', 'default_fs':
'hdfs://ambari-agent-01:8020', 'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user':
'hdfs', 'owner': 'ambari-qa', 'hadoop_conf_dir': '/etc/hadoop/conf', 'type':
'file', 'action': ['create_on_execute'], 'immutable_paths':
[u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0770}
2022-10-08 04:44:09,810 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa/mapredsmokeinput?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmpsMzXqS 2>/tmp/tmpB57GCf''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:10,053 - call returned (0, '')
2022-10-08 04:44:10,053 - get_user_call_output returned (0,
u'{"FileStatus":{"accessTime":1665204118565,"blockSize":134217728,"childrenNum":0,"fileId":16397,"group":"hdfs","length":1123,"modificationTime":1665204119318,"owner":"ambari-qa","pathSuffix":"","permission":"770","replication":3,"storagePolicy":0,"type":"FILE"}}200',
u'')
2022-10-08 04:44:10,054 - Creating new file /user/ambari-qa/mapredsmokeinput
in DFS
2022-10-08 04:44:10,055 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT --data-binary @/etc/passwd -H
'"'"'Content-Type: application/octet-stream'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa/mapredsmokeinput?op=CREATE&user.name=hdfs&overwrite=True&permission=770'"'"'
1>/tmp/tmpFbeZFF 2>/tmp/tmpnererF''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:10,554 - call returned (0, '')
2022-10-08 04:44:10,554 - get_user_call_output returned (0, u'201', u'')
2022-10-08 04:44:10,555 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa/mapredsmokeinput?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmpQmU81q 2>/tmp/tmpkYAwuF''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:10,598 - call returned (0, '')
2022-10-08 04:44:10,598 - get_user_call_output returned (0,
u'{"FileStatus":{"accessTime":1665204250115,"blockSize":134217728,"childrenNum":0,"fileId":16452,"group":"hdfs","length":1160,"modificationTime":1665204250549,"owner":"hdfs","pathSuffix":"","permission":"770","replication":3,"storagePolicy":0,"type":"FILE"}}200',
u'')
2022-10-08 04:44:10,599 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa/mapredsmokeinput?op=SETOWNER&owner=ambari-qa&group=&user.name=hdfs'"'"'
1>/tmp/tmpCMvTJ_ 2>/tmp/tmpPLp6sI''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:10,637 - call returned (0, '')
2022-10-08 04:44:10,637 - get_user_call_output returned (0, u'200', u'')
2022-10-08 04:44:10,638 - HdfsResource['/user/ambari-qa/mapredsmokeoutput']
{'security_enabled': False, 'hadoop_bin_dir': '/usr/bin', 'keytab': [EMPTY],
'dfs_type': 'HDFS', 'default_fs': 'hdfs://ambari-agent-01:8020',
'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user':
'hdfs', 'action': ['delete_on_execute'], 'hadoop_conf_dir': '/etc/hadoop/conf',
'type': 'directory', 'immutable_paths': [u'/mr-history/done', u'/app-logs',
u'/tmp']}
2022-10-08 04:44:10,639 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa/mapredsmokeoutput?op=GETFILESTATUS&user.name=hdfs'"'"'
1>/tmp/tmptWqM2d 2>/tmp/tmp8437WK''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:10,673 - call returned (0, '')
2022-10-08 04:44:10,674 - get_user_call_output returned (0,
u'{"FileStatus":{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16437,"group":"hdfs","length":0,"modificationTime":1665204208837,"owner":"ambari-qa","pathSuffix":"","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}}200',
u'')
2022-10-08 04:44:10,675 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c
'curl -sS -L -w '"'"'%{http_code}'"'"' -X DELETE -d '"'"''"'"' -H
'"'"'Content-Length: 0'"'"'
'"'"'http://ambari-agent-01:50070/webhdfs/v1/user/ambari-qa/mapredsmokeoutput?op=DELETE&recursive=True&user.name=hdfs'"'"'
1>/tmp/tmpiE9byr 2>/tmp/tmpXtIqou''] {'logoutput': None, 'quiet': False}
2022-10-08 04:44:10,715 - call returned (0, '')
2022-10-08 04:44:10,715 - get_user_call_output returned (0,
u'{"boolean":true}200', u'')
2022-10-08 04:44:10,716 - HdfsResource[None] {'security_enabled': False,
'hadoop_bin_dir': '/usr/bin', 'keytab': [EMPTY], 'dfs_type': 'HDFS',
'default_fs': 'hdfs://ambari-agent-01:8020', 'hdfs_resource_ignore_file':
'/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ...,
'kinit_path_local': '/usr/bin/kinit', 'principal_name': [EMPTY], 'user':
'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/etc/hadoop/conf',
'immutable_paths': [u'/mr-history/done', u'/app-logs', u'/tmp']}
2022-10-08 04:44:10,717 - ExecuteHadoop['jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-3.*.jar wordcount
/user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput']
{'bin_dir':
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/bin:/usr/lib/hadoop-yarn/bin',
'conf_dir': '/etc/hadoop/conf', 'logoutput': True, 'try_sleep': 5, 'tries': 1,
'user': 'ambari-qa'}
2022-10-08 04:44:10,717 - Execute['hadoop --config /etc/hadoop/conf jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-3.*.jar wordcount
/user/ambari-qa/mapredsmokeinput /user/ambari-qa/mapredsmokeoutput']
{'logoutput': True, 'try_sleep': 5, 'environment': {}, 'tries': 1, 'user':
'ambari-qa', 'path':
['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/bin:/usr/lib/hadoop-yarn/bin']}
22/10/08 04:44:13 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting
to ResourceManager at ambari-agent-02/172.19.0.4:8050
22/10/08 04:44:13 INFO client.AHSProxy: Connecting to Application History
server at localhost/127.0.0.1:10200
22/10/08 04:44:13 INFO mapreduce.JobResourceUploader: Disabling Erasure
Coding for path: /user/ambari-qa/.staging/job_1665204113174_0003
22/10/08 04:44:14 INFO input.FileInputFormat: Total input files to process :
1
22/10/08 04:44:15 INFO mapreduce.JobSubmitter: number of splits:1
22/10/08 04:44:15 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1665204113174_0003
22/10/08 04:44:15 INFO mapreduce.JobSubmitter: Executing with tokens: []
22/10/08 04:44:15 INFO conf.Configuration: resource-types.xml not found
22/10/08 04:44:15 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
22/10/08 04:44:15 INFO impl.YarnClientImpl: Submitted application
application_1665204113174_0003
22/10/08 04:44:15 INFO mapreduce.Job: The url to track the job:
http://ambari-agent-02:8088/proxy/application_1665204113174_0003/
22/10/08 04:44:15 INFO mapreduce.Job: Running job: job_1665204113174_0003
22/10/08 04:44:23 INFO mapreduce.Job: Job job_1665204113174_0003 running in
uber mode : false
22/10/08 04:44:23 INFO mapreduce.Job: map 0% reduce 0%
22/10/08 04:44:29 INFO mapreduce.Job: map 100% reduce 0%
22/10/08 04:44:35 INFO mapreduce.Job: map 100% reduce 100%
22/10/08 04:44:36 INFO mapreduce.Job: Job job_1665204113174_0003 completed
successfully
22/10/08 04:44:36 INFO mapreduce.Job: Counters: 54
File System Counters
FILE: Number of bytes read=1376
FILE: Number of bytes written=565787
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1284
HDFS: Number of bytes written=1230
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3068
Total time spent by all reduces in occupied slots (ms)=2784
Total time spent by all map tasks (ms)=3068
Total time spent by all reduce tasks (ms)=2784
Total vcore-milliseconds taken by all map tasks=3068
Total vcore-milliseconds taken by all reduce tasks=2784
Total megabyte-milliseconds taken by all map tasks=4712448
Total megabyte-milliseconds taken by all reduce tasks=4276224
Map-Reduce Framework
Map input records=26
Map output records=35
Map output bytes=1300
Map output materialized bytes=1376
Input split bytes=124
Combine input records=35
Combine output records=35
Reduce input groups=35
Reduce shuffle bytes=1376
Reduce input records=35
Reduce output records=35
Spilled Records=70
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=129
CPU time spent (ms)=1470
Physical memory (bytes) snapshot=1034907648
Virtual memory (bytes) snapshot=6090969088
Total committed heap usage (bytes)=883949568
Peak Map Physical memory (bytes)=837943296
Peak Map Virtual memory (bytes)=3028852736
Peak Reduce Physical memory (bytes)=196964352
Peak Reduce Virtual memory (bytes)=3062116352
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1160
File Output Format Counters
Bytes Written=1230
2022-10-08 04:44:36,594 - ExecuteHadoop['fs -test -e
/user/ambari-qa/mapredsmokeoutput'] {'bin_dir':
'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/bin:/usr/lib/hadoop-yarn/bin',
'user': 'ambari-qa', 'conf_dir': '/etc/hadoop/conf'}
2022-10-08 04:44:36,595 - Execute['hadoop --config /etc/hadoop/conf fs -test
-e /user/ambari-qa/mapredsmokeoutput'] {'logoutput': None, 'try_sleep': 0,
'environment': {}, 'tries': 1, 'user': 'ambari-qa', 'path':
['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/bin:/usr/lib/hadoop-yarn/bin']}
Command completed successfully!
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
