Dear all: ????flink??????????????6???????????????????????????????????????? ??????????kafka??hdfs??????????????100G ??????kafka????kerberos??????????????kerberos???? ?????? 1.??????checkpoint??????
2.??????????flink??????????????????kafka??????????????????/etc/krb5.conf??????????????hosts??????KDC????????????????????????????????????6???????????????????????????????????????? 3.?????????????????????????????????????????? 2020-03-06 01:11:27,027 INFO org.apache.hadoop.hdfs.DFSClient - Could not complete /user/shtermuser/holmes_hdfs/checkpoint/ab806ce6e5b0dda5eae3139eb784ddb8/chk-12673/b1992c81-7821-4d68-9c0b-1302b24c42d0 retrying... 2020-03-06 01:11:33,457 INFO org.apache.hadoop.hdfs.DFSClient - Could not complete /user/shtermuser/holmes_hdfs/checkpoint/ab806ce6e5b0dda5eae3139eb784ddb8/chk-12673/b1992c81-7821-4d68-9c0b-1302b24c42d0 retrying... 2020-03-06 01:11:56,869 INFO org.apache.hadoop.hdfs.DFSClient - Could not complete /user/shtermuser/holmes_hdfs/checkpoint/ab806ce6e5b0dda5eae3139eb784ddb8/chk-12674/58c158bf-0583-4c14-a6d4-2873d1b0d9ac retrying... 2020-03-06 01:12:03,270 INFO org.apache.hadoop.hdfs.DFSClient - Could not complete /user/shtermuser/holmes_hdfs/checkpoint/ab806ce6e5b0dda5eae3139eb784ddb8/chk-12674/58c158bf-0583-4c14-a6d4-2873d1b0d9ac retrying... 2020-03-06 01:12:18,283 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to cancel task Source: Custom Source -> Sink: Unnamed (2/6) (90484955ec6ef7ff8d705142bbf6b5e0). 2020-03-06 01:12:18,283 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> Sink: Unnamed (2/6) (90484955ec6ef7ff8d705142bbf6b5e0) switched from RUNNING to CANCELING. 2020-03-06 01:12:18,283 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: Custom Source -> Sink: Unnamed (2/6) (90484955ec6ef7ff8d705142bbf6b5e0). 2020-03-06 01:12:18,442 WARN org.apache.kafka.common.security.kerberos.KerberosLogin - [[email protected]]: TGT renewal thread has been interrupted and will exit. 2020-03-06 01:12:18,443 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> Sink: Unnamed (2/6) (90484955ec6ef7ff8d705142bbf6b5e0) switched from CANCELING to CANCELED. 2020-03-06 01:12:18,443 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source -> Sink: Unnamed (2/6) (90484955ec6ef7ff8d705142bbf6b5e0). 2020-03-06 01:12:18,445 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Sink: Unnamed (2/6) (90484955ec6ef7ff8d705142bbf6b5e0) [CANCELED] 2020-03-06 01:12:18,447 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state CANCELED to JobManager for task Source: Custom Source -> Sink: Unnamed 90484955ec6ef7ff8d705142bbf6b5e0. 2020-03-06 01:12:30,966 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Free slot TaskSlot(index:0, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.7976931348623157E308, heapMemoryInMB=2147483647, directMemoryInMB=2147483647, nativeMemoryInMB=2147483647, networkMemoryInMB=2147483647, managedMemoryInMB=1935}, allocationId: 13e1e5aaa46415eb80b67fda822519f5, jobId: ab806ce6e5b0dda5eae3139eb784ddb8). 2020-03-06 01:12:30,967 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Remove job ab806ce6e5b0dda5eae3139eb784ddb8 from job leader monitoring. 2020-03-06 01:12:30,967 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close JobManager connection for job ab806ce6e5b0dda5eae3139eb784ddb8. 2020-03-06 01:12:31,931 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close JobManager connection for job ab806ce6e5b0dda5eae3139eb784ddb8. 2020-03-06 01:12:31,931 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Cannot reconnect to job ab806ce6e5b0dda5eae3139eb784ddb8 because it is not registered. 2020-03-06 01:13:07,706 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager connection 1aae239700043eb7d1bf819836cea65a. 2020-03-06 01:13:07,706 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@bitpt417r01:36666/user/resourcemanager(00000000000000000000000000000000). 2020-03-06 01:13:07,722 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration 2020-03-06 01:13:07,722 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms) 2020-03-06 01:13:07,752 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager was declined: unrecognized TaskExecutor 2020-03-06 01:13:07,753 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Pausing and re-attempting registration in 30000 ms 2020-03-06 01:13:07,993 INFO org.apache.flink.yarn.YarnTaskExecutorRunner - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested. 2020-03-06 01:13:07,994 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache ????
