Github user ppadma commented on a diff in the pull request:
https://github.com/apache/drill/pull/826#discussion_r117112392
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetRecordWriter.java
---
@@ -380,14 +384,21 @@ public void endRecord() throws IOException {
// since ParquetFileWriter will overwrite empty output file (append
is not supported)
// we need to re-apply file permission
- parquetFileWriter = new ParquetFileWriter(conf, schema, path,
ParquetFileWriter.Mode.OVERWRITE);
+ if (useConfiguredBlockSize) {
+ // Round up blockSize to multiple of 64K.
+ long writeBlockSize = ((long)
ceil((double)blockSize/BLOCKSIZE_MULTIPLE)) * BLOCKSIZE_MULTIPLE;
--- End diff --
Underlying FS API expects blockSize specified during create to be multiple
of 64K. So, I rounded it up. This will be greater than (to nearest 64K) or
equal to what is used in checkBlockSizeReached. Should I change
checkBlockSizeReached to do the same ? I was trying to avoid any changes to
existing behavior/code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
