Github user Sherry302 commented on the issue:
https://github.com/apache/spark/pull/14659
Hi, @tgravescs Thank you so much for the review. I have updated the PR
based on your every comment.
The only one question left is this one (in `Task`) "are these params all
optional just to make it easier for different task types?" I have replied this.
Could you check it again and give your opinion?
To make the caller context more readable, at commit
[10dbc6f](https://github.com/apache/spark/commit/10dbc6f26ac7d224803b721f32a9a0b4306e1f47),
I added the static strings `AttemptId` back ( for stage, task and app) which
had been deleted at commit
[748e7a9](https://github.com/apache/spark/commit/748e7a9b6f6fe928df9e49f8e020d02126123be8).
Yes, this PR will set up the caller context for both HDFS and YARN. At very
beginning, to make the review easier, I created two different jiras to set up
caller contexts for HDFS(SPARK-16757) and YARN (SPARK-16758) although the code
is the same. I have updated the jiras, the title of this PR, and the
description of this PR. In the âHow was this patch testedâ of the PRâs
description, you can see what are showing in HDFS hdfs-audit.log and Yarn RM
audit log.
When invoking Hadoop CallerContext API in Yarn Client, the caller context
(including `SPARK_CLIENT` with AppId only) will be written to both HDFS audit
log and Yarn RM audit log.
In hdfs-audit.log:
```
2016-09-20 11:54:24,116 INFO FSNamesystem.audit: allowed=true ugi=wyang
(auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/lr_big.txt dst=null perm=null
proto=rpc callerContext=SPARK_CLIENT_AppId_application_1474394339641_0005
```
In Yarn RM log:
```
2016-09-20 11:59:24,050 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=wyang
IP=127.0.0.1 OPERATION=Submit Application Request TARGET=ClientRMService
RESULT=SUCCESS APPID=application_1474394339641_0006
CALLERCONTEXT=SPARK_CLIENT_AppId_application_1474394339641_0006
```
Also, I have tested this with multiple tasks running in the same executor.
Take `application_1474394339641_0006` as example.
My command line to run tests as below:
```
./bin/spark-submit --verbose --executor-cores 3 --num-executors 1 --master
yarn --deploy-mode client --class org.apache.spark.examples.SparkKMeans
examples/target/original-spark-examples_2.11-2.1.0-SNAPSHOT.jar
hdfs://localhost:9000/lr_big.txt 2 5
```
In Spark History Application page, you can see there are two executors (one
is driver), in the executor, there are 46 tasks:
<img width="1411" alt="screen shot 2016-09-20 at 12 53 52 pm"
src="https://cloud.githubusercontent.com/assets/8546874/18686920/a2617e70-7f32-11e6-947e-dfe83c4185e3.png";>
In HDFS audit log, there are 46 task records.:
```
2016-09-20 11:59:33,868 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs
src=/private/tmp/hadoop-wyang/nm-local-dir/usercache/wyang/appcache/application_1474394339641_0006/container_1474394339641_0006_01_000001/spark-warehouse
dst=null perm=wyang:supergroup:rwxr-xr-x proto=rpc
callerContext=SPARK_APPLICATION_MASTER_AppId_application_1474394339641_0006_AttemptId_1
2016-09-20 11:59:37,214 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_1_AttemptNum_0
2016-09-20 11:59:37,215 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_2_AttemptNum_0
2016-09-20 11:59:37,215 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_0_AttemptNum_0
2016-09-20 11:59:42,391 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_3_AttemptNum_0
2016-09-20 11:59:42,432 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_4_AttemptNum_0
2016-09-20 11:59:42,445 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_5_AttemptNum_0
2016-09-20 11:59:46,720 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_6_AttemptNum_0
2016-09-20 11:59:46,735 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_7_AttemptNum_0
2016-09-20 11:59:46,739 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_8_AttemptNum_0
2016-09-20 11:59:50,915 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_9_AttemptNum_0
2016-09-20 11:59:50,932 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_10_AttemptNum_0
2016-09-20 11:59:50,934 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_10_AttemptNum_0
2016-09-20 11:59:50,941 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_11_AttemptNum_0
2016-09-20 11:59:50,943 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_11_AttemptNum_0
2016-09-20 11:59:54,757 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_9_AttemptNum_0
2016-09-20 11:59:54,774 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_12_AttemptNum_0
2016-09-20 11:59:54,776 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_12_AttemptNum_0
2016-09-20 11:59:54,799 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_13_AttemptNum_0
2016-09-20 11:59:54,803 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_13_AttemptNum_0
2016-09-20 11:59:54,812 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_14_AttemptNum_0
2016-09-20 11:59:54,814 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_14_AttemptNum_0
2016-09-20 11:59:59,024 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_15_AttemptNum_0
2016-09-20 11:59:59,025 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_15_AttemptNum_0
2016-09-20 11:59:59,044 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_16_AttemptNum_0
2016-09-20 11:59:59,046 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_16_AttemptNum_0
2016-09-20 11:59:59,059 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_17_AttemptNum_0
2016-09-20 11:59:59,060 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_17_AttemptNum_0
2016-09-20 12:00:03,065 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_18_AttemptNum_0
2016-09-20 12:00:03,066 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_18_AttemptNum_0
2016-09-20 12:00:03,107 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_19_AttemptNum_0
2016-09-20 12:00:03,107 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_20_AttemptNum_0
2016-09-20 12:00:03,108 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_20_AttemptNum_0
2016-09-20 12:00:03,109 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_19_AttemptNum_0
2016-09-20 12:00:07,359 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_22_AttemptNum_0
2016-09-20 12:00:07,360 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_22_AttemptNum_0
2016-09-20 12:00:07,366 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_21_AttemptNum_0
2016-09-20 12:00:07,367 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_0_StageId_0_AttemptId_0_TaskId_21_AttemptNum_0
2016-09-20 12:00:11,076 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_27_AttemptNum_0
2016-09-20 12:00:11,094 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_28_AttemptNum_0
2016-09-20 12:00:11,094 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_29_AttemptNum_0
2016-09-20 12:00:15,082 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_30_AttemptNum_0
2016-09-20 12:00:15,094 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_31_AttemptNum_0
2016-09-20 12:00:15,189 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_32_AttemptNum_0
2016-09-20 12:00:19,481 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_33_AttemptNum_0
2016-09-20 12:00:19,530 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_34_AttemptNum_0
2016-09-20 12:00:19,532 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_34_AttemptNum_0
2016-09-20 12:00:19,620 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_35_AttemptNum_0
2016-09-20 12:00:19,621 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_35_AttemptNum_0
2016-09-20 12:00:24,193 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_33_AttemptNum_0
2016-09-20 12:00:24,210 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_36_AttemptNum_0
2016-09-20 12:00:24,211 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_36_AttemptNum_0
2016-09-20 12:00:24,279 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_37_AttemptNum_0
2016-09-20 12:00:24,281 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_37_AttemptNum_0
2016-09-20 12:00:24,303 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_38_AttemptNum_0
2016-09-20 12:00:24,305 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_38_AttemptNum_0
2016-09-20 12:00:29,118 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_39_AttemptNum_0
2016-09-20 12:00:29,120 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_39_AttemptNum_0
2016-09-20 12:00:29,149 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_40_AttemptNum_0
2016-09-20 12:00:29,151 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_40_AttemptNum_0
2016-09-20 12:00:29,161 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_41_AttemptNum_0
2016-09-20 12:00:29,163 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_41_AttemptNum_0
2016-09-20 12:00:33,143 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_42_AttemptNum_0
2016-09-20 12:00:33,144 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_43_AttemptNum_0
2016-09-20 12:00:33,144 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_42_AttemptNum_0
2016-09-20 12:00:33,145 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_43_AttemptNum_0
2016-09-20 12:00:33,184 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_44_AttemptNum_0
2016-09-20 12:00:33,185 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_44_AttemptNum_0
2016-09-20 12:00:37,142 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_45_AttemptNum_0
2016-09-20 12:00:37,144 INFO FSNamesystem.audit: allowed=true
ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=open
src=/lr_big.txt dst=null perm=null proto=rpc
callerContext=SPARK_TASK_AppId_application_1474394339641_0006_AttemptId_1_JobId_1_StageId_1_AttemptId_0_TaskId_45_AttemptNum_0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://cloud.githubusercontent.com/assets/8546874/18686920/a2617e70-7f32-11e6-947e-dfe83c4185e3.png"](https://cloud.githubusercontent.com/assets/8546874/18686920/a2617e70-7f32-11e6-947e-dfe83c4185e3.png"){kind=link}