[ https://issues.apache.org/jira/browse/HDDS-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967931#comment-16967931 ]
Anu Engineer commented on HDDS-2384: ------------------------------------ Nope, we can encode the data in a set of small packets or read them as a sequence of small packets. Let us say, 8KB/64KB buffer and we read and write the data continually to the underlying disk. > Large chunks during write can have memory pressure on DN with multiple clients > ------------------------------------------------------------------------------ > > Key: HDDS-2384 > URL: https://issues.apache.org/jira/browse/HDDS-2384 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Reporter: Rajesh Balamohan > Assignee: Anu Engineer > Priority: Major > Labels: performance > > During large file writes, it ends up writing {{16 MB}} chunks. > https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java#L691 > In large clusters, 100s of clients may connect to DN. In such cases, > depending on the incoming write workload mem load on DN can increase > significantly. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org