Just as an FYI and hopefully leading to a bug-fix: the command to start Datanode daemons in .../sbin/start-dfs.sh and .../sbin/stop-dfs.sh from the prebuilt Hadoop distribution (both 3.0.1 and 3.1.0) won't run as written.

Here's the command that errors out:

   hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
        --workers \
        --config "${HADOOP_CONF_DIR}" \
        --daemon start \
        datanode ${dataStartOpt}

What happens is that that it thinks the path to the workers file is the network name of a datanode.

To get this to work properly, I use the --hostnames option and supply as a value a space-delimited version of the one-name-per-line workers file like so:

hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --hostnames `sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/,/g' ${HADOOP_CONF_DIR}/workers` \
    --config "${HADOOP_CONF_DIR}" \
    --daemon start \
    datanode ${dataStartOpt}


Reply via email to