Doug Cutting wrote:
Raghu Angadi wrote:
I don't think there is an explanation of why increasing the handlers
proportionally helps (I does help, but it might be a big hammer
approach). I think ipc Q-length and q management also matters a lot.
I will open a Jira with couple thoughts/explanation/improvements regd
Q mangagement in our IPC.
Regardless, we should try to make such proportional parameters
automatically adjust to cluster size.
Agreed. I was implying that we should have at least some
back-of-the-envolope explantions for these formulas. Yes, such
proportional config would be very useful.
Right now Namenode does not know about the cluster size before starting
IPC server.
Raghu.
So instead of having parameters
whose values must be proportional to cluster size, we should have
parameters that are constants of proportionality (e.g., multiplied by
the cluster size to determine the runtime value). We have a list here
of three such parameters and we should thus work to fix each of these to
be proportional to cluster size, no? If today it's handlers and
tomorrow it's queues we'll be better served if folks don't have to edit
their config files when we make such changes.
Doug
- Re: [Lucene-hadoop Wiki] Update of "FAQ" by ... Raghu Angadi
-
