Github user kkhatua commented on the issue:

    https://github.com/apache/drill/pull/826
  
    @ppadma , Khurram [~khfaraaz] and I were looking at the details in the PR 
and it's not very clear what new behavior does the PR allow. If you need to 
specify the block-size as described in the [comment 
](https://issues.apache.org/jira/browse/DRILL-5379?focusedCommentId=15981366&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15981366)by
 @fmethot , doesn't Drill already do that? I thought Drill implicitly creates 
files with a single row-group anyway. 
    
    My understanding of the JIRA's problem statement was that if the Parquet 
block-size (i.e. the rowgroup size) is set to a large value that exceeds the 
HDFS block size, using the flag would allow Drill to ignore the larger value in 
the options and write with a parquet-blocksize that matches the target HDFS 
location. So, I could have {{store.parquet.block-size=1073741824}} (i.e. 1GB), 
but when writing an output worth 512MB, instead of 1 file... Drill would read 
the HDFS block-size (say 128GB) and apply that as the parquet-block-size and 
write 4 files. 
    
    @fmethot is that what you were looking for? An **automatic scaling down** 
of the parquet file's size to match (and be contained within) the HDFS block 
size?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Reply via email to