Thank you Kostas for spending time on my case. Relating to the issue I mentioned, I have another issue caused by having a lot of files to list. From the error msg, I understand that the listing was taking more than 30s, and the JM thought that it hung and killed it. Is that possible to increase this 30s timer?
Thanks and regards, Averell 2018-09-25 12:01:13.222 [Canceler/Interrupts for Source: Custom File Source (1/1) (a5f5434070044510eafc9103bc24af43).] WARN org.apache.flink.runtime.taskmanager.Task - Task 'Source: Custom File Source (1/1)' did not react to cancelling signal for 30 seconds, but is stuck in method: java.net.URI$Parser.scan(URI.java:2998) java.net.URI$Parser.parseAuthority(URI.java:3138) java.net.URI$Parser.parseHierarchical(URI.java:3097) java.net.URI$Parser.parse(URI.java:3053) java.net.URI.<init>(URI.java:746) org.apache.hadoop.fs.Path.makeQualified(Path.java:467) org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:464) com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem$$Lambda$63/515305348.apply(Unknown Source) com.amazon.ws.emr.hadoop.fs.s3n.BasicFileStatusFactory.newFile(BasicFileStatusFactory.java:69) com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.newFile(S3NativeFileSystem.java:1154) com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.listStatus(S3NativeFileSystem.java:962) com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.listStatus(S3NativeFileSystem.java:914) com.amazon.ws.emr.hadoop.fs.EmrFileSystem.listStatus(EmrFileSystem.java:364) org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:157) org.apache.flink.core.fs.SafetyNetWrapperFileSystem.listStatus(SafetyNetWrapperFileSystem.java:97) org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.listEligibleFiles(ContinuousFileMonitoringFunction.java:395) org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.listEligibleFiles(ContinuousFileMonitoringFunction.java:416) org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.listEligibleFiles(ContinuousFileMonitoringFunction.java:416) org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.monitorDirAndForwardSplits(ContinuousFileMonitoringFunction.java:327) org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction.run(ContinuousFileMonitoringFunction.java:292) org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:56) org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99) org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300) org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) java.lang.Thread.run(Thread.java:748) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
