Hi,
> Would then the assumption that this possibility ( part reported length >
> part file size ( reported by FileStatus on NN) ) is only attributable to
> this edge case be correct ?
Yes, I think so.
> Or do you see a case where in though the above is true, the part file would
> need
Thank you for your email.
Would then the assumption that this possibility ( part reported length >
part file size ( reported by FileStatus on NN) ) is only attributable to
this edge case be correct ?
Or do you see a case where in though the above is true, the part file would
need truncation as
Hi Vishal,
I’ve come across the same problem. The problem is that by default the file
length is not updated when the output stream is not closed properly.
I modified the writer to update file lengths on each flush, but it comes with
some overhead, so this approach should be used when strong
This seems strange. When I pull the ( copyToLocal ) the part file to local
FS, it has the same length as reported by the length file. The fileStatus
from hadoop seems to have a wrong length.
This seems to be true for all these type of discrepancies. It might be that
the block information did not
Hello folks,
I have flink 1.7.2 working with hadoop 2.6 and b'coz there
is no in build truncate ( in hadoop 2.6 ) I am writing a method to cleanup
( truncate ) part files based on the length in the valid-length files
dropped by flink during restore. I see some thing very strange
