n3nash commented on a change in pull request #2440:
URL: https://github.com/apache/hudi/pull/2440#discussion_r557104633
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##########
@@ -274,19 +275,27 @@ private boolean isBlockCorrupt(int blocksize) throws
IOException {
}
private long scanForNextAvailableBlockOffset() throws IOException {
+ // Make buffer large enough to scan through the file as quick as possible
especially if it is on S3/GCS.
+ // Using lower buffer is incurring a lot of API calls thus drastically
increasing the cost of the storage
+ // and also may take days to complete scanning trough the large files.
Review comment:
nit: s/trough/through
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFileReader.java
##########
@@ -274,19 +275,27 @@ private boolean isBlockCorrupt(int blocksize) throws
IOException {
}
private long scanForNextAvailableBlockOffset() throws IOException {
+ // Make buffer large enough to scan through the file as quick as possible
especially if it is on S3/GCS.
+ // Using lower buffer is incurring a lot of API calls thus drastically
increasing the cost of the storage
Review comment:
Let's remove these 2 line comments and keep only the first line since
that is self explanatory
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
