Hi,
While i am executing flume-ng, i am getting following out put on my
terminal, it seems got struck.
root@md-trngpoc1:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT#
bin/flume-ng agent -n agent1 -c conf -f conf/agent1.conf
Info: Sourcing environment configuration script
/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf/flume-env.sh
Info: Including Hadoop libraries found via
(/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from
classpath
Info: Excluding
/usr/local/hadoop_dir/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from
classpath
+ exec /usr/lib/jvm/java-6-sun-1.6.0.26/jre/bin/java -Xms100m -Xmx200m -cp
'/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib/*:/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf:/usr/local/hadoop_dir/hadoop/libexec/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/local/hadoop_dir/hadoop/libexec/..:/usr/local/hadoop_dir/hadoop/libexec/../hadoop-core-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/asm-3.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjrt-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/aspectjtools-1.6.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/commons-net-1.4.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-fairscheduler-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hadoop-thriftfs-1.0.3.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/junit-4.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop_dir/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar'
-Djava.library.path=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib:/usr/local/hadoop_dir/hadoop/libexec/../lib/native/Linux-i386-32
org.apache.flume.node.Application -n agent1 -f conf/agent1.conf
in execution summary:
----------------------------------------
Info: Including Hadoop libraries found via
(/usr/local/hadoop_dir/hadoop/bin/hadoop) for HDFS access
hadoop lib path pointing to /usr/local/hadoop_dir/hadoop/bin/hadoop
directory insted of /usr/local/hadoop_dir/hadoop/lib path.
Here my questions are;
1. in flume-ng script anything changes need in add_hadoop_paths function.
2. anything wrong in flume-env.sh script
3. anything wrong in agent1.conf file.
Please find the attached configuaration details and let me know any more
details.
Could you pleae suggest me on this issue.
Thanks,
Prabhu
Version details:
===============
Hadoop 1.0.3
flume 1.2.0
Java 1.6
flume.log
==================
2012-07-10 17:49:05,225 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
2012-07-10 17:49:05,226 INFO node.FlumeNode: Flume node starting - agent1
2012-07-10 17:49:05,228 INFO nodemanager.DefaultLogicalNodeManager: Node
manager starting
2012-07-10 17:49:05,228 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
2012-07-10 17:49:05,229 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
2012-07-10 17:49:05,230 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:conf/agent1.conf
2012-07-10 17:49:05,237 INFO conf.FlumeConfiguration: Processing:HDFS
2012-07-10 17:49:05,238 INFO conf.FlumeConfiguration: Processing:HDFS
2012-07-10 17:49:05,238 INFO conf.FlumeConfiguration: Processing:HDFS
2012-07-10 17:49:05,238 INFO conf.FlumeConfiguration: Processing:HDFS
2012-07-10 17:49:05,238 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent:
agent1
2012-07-10 17:49:05,254 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration for agents: [agent1]
2012-07-10 17:49:05,255 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
2012-07-10 17:49:05,258 INFO properties.PropertiesFileConfigurationProvider:
created channel MemoryChannel-2
2012-07-10 17:49:05,267 INFO sink.DefaultSinkFactory: Creating instance of sink
HDFS typehdfs
agent1.conf
================
agent1.sources = tail
agent1.channels = MemoryChannel-2
agent1.sinks = HDFS
agent1.sources.tail.type = exec
agent1.sources.tail.command = tail -F /var/log/syslog.1
agent1.sources.tail.channels = MemoryChannel-2
agent1.sinks.HDFS.channel = MemoryChannel-2
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://10.5.114.110:9000/flume
agent1.sinks.HDFS.hdfs.file.Type = DataStream
agent1.channels.MemoryChannel-2.type = memory
flume-env.sh
======================
# Enviroment variables can be set here.
#JAVA_HOME=/usr/lib/jvm/java-6-sun
JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.26/jre
# Give Flume more memory and pre-allocate
JAVA_OPTS="-Xms100m -Xmx200m"
# Note that the Flume conf directory is always included in the classpath.
FLUME_CLASSPATH=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf
HADOOP_HOME=/usr/local/hadoop_dir/hadoop
flume-ng
==================
################################
# constants
################################
FLUME_AGENT_CLASS="org.apache.flume.node.Application"
FLUME_AVRO_CLIENT_CLASS="org.apache.flume.client.avro.AvroCLIClient"
################################
# functions
################################
info() {
local msg=$1
echo "Info: $msg" >&2
}
warn() {
local msg=$1
echo "Warning: $msg" >&2
}
error() {
local msg=$1
local exit_code=$2
echo "Error: $msg" >&2
if [ -n "$exit_code" ] ; then
exit $exit_code
fi
}
# If avail, add Hadoop paths to the FLUME_CLASSPATH and to the
# FLUME_JAVA_LIBRARY_PATH env vars.
# Requires Flume jars to already be on FLUME_CLASSPATH.
add_hadoop_paths() {
local HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" \
which hadoop 2>/dev/null)
if [ -f "${HADOOP_IN_PATH}" ]; then
info "Including Hadoop libraries found via ($HADOOP_IN_PATH) for HDFS
access"
# determine hadoop java.library.path and use that for flume
local HADOOP_CLASSPATH=/usr/local/hadoop_dir/hadoop/lib/
local HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$FLUME_CLASSPATH" \
${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProperty \
java.library.path 2>/dev/null)
# look for the line that has the desired property value
# (considering extraneous output from some GC options that write to stdout)
# IFS = InternalFieldSeparator (set to recognize only newline char as
delimiter)
IFS=$'\n'
for line in $HADOOP_JAVA_LIBRARY_PATH; do
if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then
HADOOP_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]}
break
fi
done
unset IFS
if [ -n "${HADOOP_JAVA_LIBRARY_PATH}" ]; then
FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HADOOP_JAVA_LIBRARY_PATH"
fi
# determine hadoop classpath
HADOOP_CLASSPATH=$($HADOOP_IN_PATH classpath 2>/dev/null)
# hack up and filter hadoop classpath
local ELEMENTS=$(sed -e 's/:/ /g' <<<${HADOOP_CLASSPATH})
local ELEMENT
for ELEMENT in $ELEMENTS; do
local PIECE
for PIECE in $(echo $ELEMENT); do
if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then
info "Excluding $PIECE from classpath"
continue
else
FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE"
fi
done
done
fi
}
add_HBASE_paths() {
local HBASE_IN_PATH=$(PATH="${HBASE_HOME}/bin:$PATH" \
which hbase 2>/dev/null)
if [ -f "${HBASE_IN_PATH}" ]; then
info "Including HBASE libraries found via ($HBASE_IN_PATH) for HBASE access"
# determine HBASE java.library.path and use that for flume
local HBASE_CLASSPATH=""
local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" \
${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \
java.library.path 2>/dev/null)
# look for the line that has the desired property value
# (considering extraneous output from some GC options that write to stdout)
# IFS = InternalFieldSeparator (set to recognize only newline char as
delimiter)
IFS=$'\n'
for line in $HBASE_JAVA_LIBRARY_PATH; do
if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then
HBASE_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]}
break
fi
done
unset IFS
if [ -n "${HBASE_JAVA_LIBRARY_PATH}" ]; then
FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HBASE_JAVA_LIBRARY_PATH"
fi
# determine HBASE classpath
HBASE_CLASSPATH=$($HBASE_IN_PATH classpath 2>/dev/null)
# hack up and filter HBASE classpath
local ELEMENTS=$(sed -e 's/:/ /g' <<<${HBASE_CLASSPATH})
local ELEMENT
for ELEMENT in $ELEMENTS; do
local PIECE
for PIECE in $(echo $ELEMENT); do
if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then
info "Excluding $PIECE from classpath"
continue
else
FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE"
fi
done
done
FLUME_CLASSPATH="$FLUME_CLASSPATH:$HBASE_HOME/conf"
fi
}
display_help() {
cat <<EOF
Usage: $0 <command> [options]...
commands:
help display this help text
agent run a Flume agent
avro-client run an avro Flume client
global options:
--conf,-c <conf> use configs in <conf> directory
--classpath,-C <cp> append to the classpath
--dryrun,-d do not actually start Flume, just print the command
-Dproperty=value sets a JDK system property value
agent options:
--conf-file,-f <file> specify a config file (required)
--name,-n <name> the name of this agent (required)
--help,-h display help text
avro-client options:
--host,-H <host> hostname to which events will be sent (required)
--port,-p <port> port of the avro source (required)
--filename,-F <file> text file to stream to avro source [default: std input]
--headerFile,-R <file> headerFile containing headers as key/value pairs on
each new line
--help,-h display help text
Note that if <conf> directory is specified, then it is always included first
in the classpath.
EOF
}
run_flume() {
local FLUME_APPLICATION_CLASS
if [ "$#" -gt 0 ]; then
FLUME_APPLICATION_CLASS=$1
shift
else
error "Must specify flume application class" 1
fi
set -x
$EXEC $JAVA_HOME/bin/java $JAVA_OPTS -cp "$FLUME_CLASSPATH" \
-Djava.library.path=$FLUME_JAVA_LIBRARY_PATH "$FLUME_APPLICATION_CLASS" $*
}
################################
# main
################################
# set default params
HADOOP_HOME=/usr/local/hadoop_dir/hadoop
FLUME_CLASSPATH=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/conf
FLUME_JAVA_LIBRARY_PATH=/usr/local/flume_dir/flume/flume-1.2.0-incubating-SNAPSHOT/lib
JAVA_OPTS="-Xmx20m"
opt_conf=""
opt_classpath=""
opt_java_props=""
opt_dryrun=""
mode=$1
shift
case "$mode" in
help)
display_help
exit 0
;;
agent)
opt_agent=1
;;
node)
opt_agent=1
warn "The \"node\" command is deprecated. Please use \"agent\" instead."
;;
avro-client)
opt_avro_client=1
;;
*)
error "Unknown or unspecified command '$mode'"
echo
display_help
exit 1
;;
esac
while [ -n "$*" ] ; do
arg=$1
shift
case "$arg" in
--conf|-c)
[ -n "$1" ] || error "Option --conf requires an argument" 1
opt_conf=$1
shift
;;
--classpath|-C)
[ -n "$1" ] || error "Option --classpath requires an argument" 1
opt_classpath=$1
shift
;;
--dryrun|-d)
opt_dryrun="1"
;;
-D*)
opt_java_props="$opt_java_props $arg"
;;
*)
args="$args $arg"
;;
esac
done
# make opt_conf absolute
if [[ -n "$opt_conf" && -d "$opt_conf" ]]; then
opt_conf=$(cd $opt_conf; pwd)
fi
# allow users to override the default env vars via conf/flume-env.sh
if [ -z "$opt_conf" ]; then
warn "No configuration directory set! Use --conf <dir> to override."
elif [ -f "$opt_conf/flume-env.sh" ]; then
info "Sourcing environment configuration script $opt_conf/flume-env.sh"
source "$opt_conf/flume-env.sh"
fi
# append command-line java options to stock or env script JAVA_OPTS
if [ -n "${opt_java_props}" ]; then
JAVA_OPTS="${JAVA_OPTS} ${opt_java_props}"
fi
# prepend command-line classpath to env script classpath
if [ -n "${opt_classpath}" ]; then
if [ -n "${FLUME_CLASSPATH}" ]; then
FLUME_CLASSPATH="${opt_classpath}:${FLUME_CLASSPATH}"
else
FLUME_CLASSPATH="${opt_classpath}"
fi
fi
if [ -z "${FLUME_HOME}" ]; then
FLUME_HOME=$(cd $(dirname $0)/..; pwd)
fi
# prepend $FLUME_HOME/lib jars to the specified classpath (if any)
if [ -n "${FLUME_CLASSPATH}" ] ; then
FLUME_CLASSPATH="${FLUME_HOME}/lib/*:$FLUME_CLASSPATH"
else
FLUME_CLASSPATH="${FLUME_HOME}/lib/*"
fi
# find java
if [ -z "${JAVA_HOME}" ] ; then
warn "JAVA_HOME is not set!"
# Try to use Bigtop to autodetect JAVA_HOME if it's available
if [ -e /usr/libexec/bigtop-detect-javahome ] ; then
. /usr/libexec/bigtop-detect-javahome
elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ] ; then
. /usr/lib/bigtop-utils/bigtop-detect-javahome
fi
# Using java from path if bigtop is not installed or couldn't find it
if [ -z "${JAVA_HOME}" ] ; then
JAVA_DEFAULT=$(type -p java)
[ -n "$JAVA_DEFAULT" ] || error "Unable to find java executable. Is it in
your PATH?" 1
JAVA_HOME=$(cd $(dirname $JAVA_DEFAULT)/..; pwd)
fi
fi
# look for hadoop libs
add_hadoop_paths
add_HBASE_paths
# prepend conf dir to classpath
if [ -n "$opt_conf" ]; then
FLUME_CLASSPATH="$opt_conf:$FLUME_CLASSPATH"
fi
# allow dryrun
EXEC="exec"
if [ -n "${opt_dryrun}" ]; then
warn "Dryrun mode enabled (will not actually initiate startup)"
EXEC="echo"
fi
# finally, invoke the appropriate command
if [ -n "$opt_agent" ] ; then
run_flume $FLUME_AGENT_CLASS $args
elif [ -n "$opt_avro_client" ] ; then
run_flume $FLUME_AVRO_CLIENT_CLASS $args
else
error "This message should never appear" 1
fi
exit 0
