Hi,
I am running issues when trying to move from HDFS to S3 using Flink 1.6.
I am getting an exception from Hadoop code:
IOException("Resource " + sCopy +
" changed on src filesystem (expected " + resource.getTimestamp() +
", was " + sStat.getModificationTime());
Digging into this, I found there was one commit
<https://github.com/apache/flink/commit/c90a757b29f168144b1bae99df532911ae682e63>
made
by Nico trying to fix this issue in 2018. However, the fix did not work for
my case, as the fs.setTimes() method was not implemented in the hadoop-aws
S3AFilesystem I am using. And it seems S3 does not allow you to override
the last modified time for an object.
I am able to make an workaround the other way round: reading the timestamp
from S3 and override the local resource. Just wonder if any one has seen
similar issues, or he/she can actually make it work by using different
implementation of S3AFilesystem? Thanks!
--
Best,
Yan
