Allen Wittenauer commented on HADOOP-13717:

Ah, OK. 

bq. Is it reasonable to also skip these checks in this situation?

In the general daemon case we definitely need to leave the hadoop_verify_logdir 
in place.  This will get followed up with verifying the pid dir as well.  This 
is needed because many folks run the daemon commands "interactively" from a 
shell code perspective but do their own daemonization from init.  There's also 
the docker case where individual daemons are in their own containers.   etc, 
etc. We're almost always better checking for the existence of this dirs rather 
than letting Java go haywire.

But balancer (and mover and probably a handful of others) are, in many ways, 

We effectively have a problem of history. sbin/start-balancer.sh was added way 
back when as a convenience  to background the balancer.  When the rewrite 
happened, balancer was added to the commands that could be daemonized as a 

There might also be a privilege problem here too.  I don't think balancer can 
be run as by a non-privileged user, usually making the log and pid dir 
verification (usually) safe and somewhat required if we want all of our daemons 
to function in the same way (consistency!).  The pid check at the UNIX level 
definitely helps prevent multiple instances running on the same host way faster 
(and somewhat more reliably!) than the Java checks. 

To me, the questions become:

a) should balancer stay as a daemon? what about the other 
not-really-daemons-but-can-run-in-daemon-mode subcommands?
b) should instead there be logic to say that failing logdir and piddir are 
optionally fatal?
c) should a local override be used instead? (e.g., define a shell profile that 
replaces the hdfs balancer command)

> Shell scripts call hadoop_verify_logdir even when command is not started as 
> daemon
> ----------------------------------------------------------------------------------
>                 Key: HADOOP-13717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13717
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Andrew Wang
> Issue found when working with the HDFS balancer.
> In {{hadoop_daemon_handler}}, it calls {{hadoop_verify_logdir}} even for the 
> "default" case which calls {{hadoop_start_daemon}}. {{daemon_outfile}} which 
> specifies the log location isn't even used here, since the command is being 
> started in the foreground.
> I think we can push the {{hadoop_verify_logdir}} call down into 
> {{hadoop_start_daemon_wrapper}} instead, which does use the outfile.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

Reply via email to