[
https://issues.apache.org/jira/browse/HADOOP-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12534012
]
Amareshwari Sri Ramadasu commented on HADOOP-1788:
--------------------------------------------------
I also observed that there is no change in writes and reads with the patch.
It seems like socket buffer size tuning is done by operating system
automatically, atleast on linux.
/proc/sys/net/core/rmem_default, /proc/sys/net/core/wmem_default,
/proc/sys/net/core/rmem_max and /proc/sys/net/core/wmem_max values influence
the buffer size.
With machine /proc/sys/net/core/rmem_max and /proc/sys/net/core/rmem_max
values 128KB, I did writes of 100KB keys. The behavior is same with or without
patch. It does a 1024byte write followed by 100K write.
write(3, "oooooooooooooooooooooooooooooooo"..., 1024) = 1024
write(3, "oooooooooooooooooooooooooooooooo"..., 99328) = 99328
When I did 500KB key writes without the patch, the kernel did memory mapped io.
mmap2(NULL, 507904, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xb7f61000
write(3, "oooooooooooooooooooooooooooooooo"..., 1024) = 1024
write(3, "oooooooooooooooooooooooooooooooo"..., 498688) = 498688
munmap(0xb7f61000, 507904)
The effect of setting buffer size might differ from one OS to the other. So at
this point I am not too sure whether we should control this from our end
especially because the processes are in the same machine and the kernel does
optmizations like using the mem mapping.
> Increase the buffer size of pipes from 1k to 128k
> -------------------------------------------------
>
> Key: HADOOP-1788
> URL: https://issues.apache.org/jira/browse/HADOOP-1788
> Project: Hadoop
> Issue Type: Bug
> Components: pipes
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1788.txt
>
>
> Currently pipes applications use 1k writes to the socket and it should be
> larger to increase throughput.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
