hi all, 最近在1.11上使用io.extra-file加载外部http文件出现一点小问题 由于http的文件在FileSystem.getFileStatus去拿状态时会走HttpFileSystem的类的getFileStatus方法,该方法返回的FileStatus中length为-1。 在client端校验通过后,在decodeYarnLocalResourceDescriptor的时候会碰到问题。 异常如下: 2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager - Could not start TaskManager in container containerXXXXXX. org.apache.flink.util.FlinkException: Error to parse YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=XXXXX.jar, path=https://XXXXXXX.jar, size=-1, modificationTime=0, visibility=APPLICATION} at org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99) at org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721) at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626) at org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746) at org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726) at org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500) at org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455) at org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415) 。。。。。 因为正则匹配的size只能匹配数字
private static final Pattern LOCAL_RESOURCE_DESC_FORMAT = Pattern.compile("YarnLocalResourceDescriptor\\{" + "key=(\\S+), path=(\\S+), size=([\\d]+), modificationTime=([\\d]+), visibility=(\\S+), type=(\\S+)}"); 负号不能被匹配上,想要实现加载http的文件需要mock一个length大于-1的FileStatus。 想问下各位大佬,看到有加载remote文件的功能,这块功能是不是给远程hdfs上文件用的,并没有考虑到http文件呢?