HI, What will be best hadoop version for the flume 1.2.0 version. As I have read that if there is difference between hadoop version and flume version then will not possible to upload the file into hadoop. As I have got experience on Linux system, every time I was getting the Broken Pipe error message.
Could you please let me know, so that accordingly, i configure the hadoop. Many thanks On Mon, Jul 30, 2012 at 5:53 PM, mardan Khan <[email protected]> wrote: > Thanks Gumnaan, > > Now the 2nd version is working. The first one was not killable for the > command prompt. Many thanks for your help. > > As i have just working on flume and have no experience. I have data file > at location c:\user\test.txt and i want to upload it to hadoop. My hadoop > is already working on my machine. What will be the possible configuration > file for this scenario. > > > Many thanks > > > > On Mon, Jul 30, 2012 at 5:27 PM, Gumnaam Sur <[email protected]>wrote: > >> Here's the second ver. >> The earlier ver. was not killable from command-line via <Ctrl-C>, >> This one is :). >> >> >> On Mon, Jul 30, 2012 at 12:19 PM, Gumnaam Sur <[email protected]>wrote: >> >>> Hi Mardan, >>> Please try with the attached, flume-ng script, >>> I've modified it to be able to run under cygwin. >>> >>> Put the file under where-ever you've extracted flume dist, under bin >>> directory. >>> >>> I've tested it , and it works, if you can confirm, then I can submit a >>> patch to flume devs. >>> >>> cheers >>> >>> On Mon, Jul 30, 2012 at 11:48 AM, mardan Khan <[email protected]>wrote: >>> >>>> yeah, you right. >>>> >>>> After successfully build the project through mvn package -DskipTest. >>>> >>>> I have got two directories in flume source: flume-ng-dist/target/ >>>> apache-flume-1.2.0-dist.tar and apache-flume-1.2.0-src.tar. >>>> >>>> I have copied the apache-flume-1.2.0-dist.tar and unpacked into >>>> d:\apache-flume. >>>> >>>> Thanks >>>> >>>> >>>> >>>> On Mon, Jul 30, 2012 at 3:58 PM, alo alt <[email protected]> wrote: >>>> >>>>> Ah, okay. >>>>> >>>>> For non-cygwin: >>>>> >>>>> You have done git-checkout and build flume successfully (mvn >>>>> -DSkipTests)? >>>>> >>>>> After you've build you've copied the binaries, located in flume's >>>>> source directory (flume-ng-dist\target\ >>>>> flume-ng-dist-1.3.0-SNAPSHOT-dist.tar.gz) onto another directory and >>>>> unpacked them? And that directory is apache-flume? >>>>> >>>>> cheers, >>>>> Alex >>>>> >>>>> >>>>> >>>>> mardan Khan <[email protected]> >>>>> July 30, 2012 4:44 PM >>>>> HI, >>>>> >>>>> Actually i am doing conversation with Inder regarding the Cygwin. As I >>>>> am trying to run on cygwin and with you on MS-DOS command prompt. >>>>> >>>>> I am just telling you important location >>>>> >>>>> >>>>> 1). My flume installation directory: D:\apache-flume >>>>> 2). Lib path: D:\apache-flume\lib >>>>> 3). Configuration File: >>>>> D:\apache-flume\conf\flume-conf.properties.template (I have copied the >>>>> configuration file from your previous post). >>>>> 4). My Java path: D:\program >>>>> files\java\jdk1.7.0_03 >>>>> >>>>> My Command: >>>>> >>>>> D:\apache-flume>"c:\Program Files\Java\jdk1.7.0_03\bin\java.exe" >>>>> -Xmx20m -Dlog4j >>>>> >>>>> .configuration=file:///%CD%\conf\log4j.properties -cp >>>>> "d:\apache-flume\lib\*" org.apache.flume.node.Application >>>>> >>>>> -f d:\apache-flume\conf\flume-conf.properties.template -n syslog-agent >>>>> >>>>> >>>>> >>>>> Error: Could not find or load main class >>>>> d:\apache-flume\lib\asynchbase-1.2.0.jar >>>>> >>>>> >>>>> Now please let me know where I am doing mistake. >>>>> >>>>> >>>>> >>>>> >>>>> alo alt <[email protected]> >>>>> July 30, 2012 4:20 PM >>>>> I think we have two different conversations about the same topic. I >>>>> reply here. >>>>> >>>>> Thats not an issue of a env script in windows, because we haven't one >>>>> so far. You may need to cp (Classpath) the flume - libs as Inder says. >>>>> As I wrote in my howto: >>>>> >>>>> As example, starting from inside of your flume-installation (the >>>>> directory where you unzip'ed the build), lets assume D:\apache-flume: >>>>> d:\apache flume > "PATH\to\java.exe" -Xmx20m >>>>> -Dlog4j.configuration=file:///%CD%\conf\log4j.properties -cp >>>>> "d:\apache-flume\flume-1.3.0-SNAPSHOT\lib\*" >>>>> org.apache.flume.node.Application -f >>>>> d:\flume\flume-1.3.0-SNAPSHOT\conf\YOUR_CONFIG.conf >>>>> -n agent >>>>> >>>>> Please be sure, that the lib-directory from flume is in place as well >>>>> in the CLASSPATH (cp). >>>>> >>>>> - Alex >>>>> Inder Pall <[email protected]> >>>>> July 30, 2012 4:02 PM >>>>> That's the problem. I am not well versed with this script to fix on >>>>> WIN but i guess as a workaround you might want to add >>>>> $FLUME_INSTALL_DIR/lib/. this lib has all the required jars and should get >>>>> you moving forward. >>>>> >>>>> Mind, filing a JIRA for the same. >>>>> >>>>> - inder >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> - Inder >>>>> Tech Platforms @Inmobi >>>>> Linkedin - http://goo.gl/eR4Ub >>>>> mardan Khan <[email protected]> >>>>> July 30, 2012 3:53 PM >>>>> Thanks Inder, >>>>> >>>>> when I am running the $echo $CLASSPATH, I am just getting . (dot) and >>>>> nothing more. >>>>> >>>>> Please see below the flume-ng files. Can you just pointout where I >>>>> should put echo$CLASSPATH. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> CLEAN_FLAG=1 >>>>> ################################ >>>>> # functions >>>>> ################################ >>>>> >>>>> info() { >>>>> if [ ${CLEAN_FLAG} -ne 0 ]; then >>>>> local msg=$1 >>>>> echo "Info: $msg" >&2 >>>>> fi >>>>> } >>>>> >>>>> warn() { >>>>> if [ ${CLEAN_FLAG} -ne 0 ]; then >>>>> local msg=$1 >>>>> echo "Warning: $msg" >&2 >>>>> fi >>>>> } >>>>> >>>>> error() { >>>>> local msg=$1 >>>>> local exit_code=$2 >>>>> >>>>> echo "Error: $msg" >&2 >>>>> >>>>> if [ -n "$exit_code" ] ; then >>>>> exit $exit_code >>>>> fi >>>>> } >>>>> >>>>> # If avail, add Hadoop paths to the FLUME_CLASSPATH and to the >>>>> # FLUME_JAVA_LIBRARY_PATH env vars. >>>>> # Requires Flume jars to already be on FLUME_CLASSPATH. >>>>> add_hadoop_paths() { >>>>> local >>>>> HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" \ >>>>> which hadoop 2>/dev/null) >>>>> >>>>> if [ -f "${HADOOP_IN_PATH}" ]; then >>>>> info "Including Hadoop libraries found via ($HADOOP_IN_PATH) for >>>>> HDFS access" >>>>> >>>>> # determine hadoop java.library.path and use that for flume >>>>> local HADOOP_CLASSPATH="" >>>>> local >>>>> HADOOP_JAVA_LIBRARY_PATH=$(HADOOP_CLASSPATH="$FLUME_CLASSPATH" \ >>>>> ${HADOOP_IN_PATH} org.apache.flume.tools.GetJavaProperty \ >>>>> java.library.path 2>/dev/null) >>>>> >>>>> # look for the line that has the desired property value >>>>> # (considering extraneous output from some GC options that write >>>>> to stdout) >>>>> # IFS = InternalFieldSeparator (set to recognize only newline char >>>>> as delimiter) >>>>> IFS=$'\n' >>>>> for line in $HADOOP_JAVA_LIBRARY_PATH; do >>>>> if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then >>>>> HADOOP_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]} >>>>> break >>>>> fi >>>>> done >>>>> unset IFS >>>>> >>>>> if [ -n "${HADOOP_JAVA_LIBRARY_PATH}" ]; then >>>>> >>>>> FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HADOOP_JAVA_LIBRARY_PATH" >>>>> fi >>>>> >>>>> # determine hadoop classpath >>>>> HADOOP_CLASSPATH=$($HADOOP_IN_PATH classpath 2>/dev/null) >>>>> >>>>> # hack up and filter hadoop classpath >>>>> local ELEMENTS=$(sed -e 's/:/ /g' <<<${HADOOP_CLASSPATH}) >>>>> local ELEMENT >>>>> for ELEMENT in $ELEMENTS; do >>>>> local PIECE >>>>> for PIECE in $(echo $ELEMENT); do >>>>> if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then >>>>> info "Excluding $PIECE from classpath" >>>>> continue >>>>> else >>>>> FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE" >>>>> fi >>>>> done >>>>> done >>>>> >>>>> fi >>>>> } >>>>> add_HBASE_paths() { >>>>> local HBASE_IN_PATH=$(PATH="${HBASE_HOME}/bin:$PATH" \ >>>>> which hbase 2>/dev/null) >>>>> >>>>> if [ -f "${HBASE_IN_PATH}" ]; then >>>>> info "Including HBASE libraries found via ($HBASE_IN_PATH) for >>>>> HBASE access" >>>>> >>>>> # determine HBASE java.library.path and use that for flume >>>>> local HBASE_CLASSPATH="" >>>>> local HBASE_JAVA_LIBRARY_PATH=$(HBASE_CLASSPATH="$FLUME_CLASSPATH" >>>>> \ >>>>> ${HBASE_IN_PATH} org.apache.flume.tools.GetJavaProperty \ >>>>> java.library.path 2>/dev/null) >>>>> >>>>> # look for the line that has the desired property value >>>>> # (considering extraneous output from some GC options that write >>>>> to stdout) >>>>> # IFS = InternalFieldSeparator (set to recognize only newline char >>>>> as delimiter) >>>>> IFS=$'\n' >>>>> for line in $HBASE_JAVA_LIBRARY_PATH; do >>>>> if [[ $line =~ ^java\.library\.path=(.*)$ ]]; then >>>>> HBASE_JAVA_LIBRARY_PATH=${BASH_REMATCH[1]} >>>>> break >>>>> fi >>>>> done >>>>> unset IFS >>>>> >>>>> if [ -n "${HBASE_JAVA_LIBRARY_PATH}" ]; then >>>>> >>>>> FLUME_JAVA_LIBRARY_PATH="$FLUME_JAVA_LIBRARY_PATH:$HBASE_JAVA_LIBRARY_PATH" >>>>> fi >>>>> >>>>> # determine HBASE classpath >>>>> HBASE_CLASSPATH=$($HBASE_IN_PATH classpath 2>/dev/null) >>>>> >>>>> # hack up and filter HBASE classpath >>>>> local ELEMENTS=$(sed -e 's/:/ /g' <<<${HBASE_CLASSPATH}) >>>>> local ELEMENT >>>>> for ELEMENT in $ELEMENTS; do >>>>> local PIECE >>>>> for PIECE in $(echo $ELEMENT); do >>>>> if [[ $PIECE =~ slf4j-(api|log4j12).*\.jar ]]; then >>>>> info "Excluding $PIECE from classpath" >>>>> continue >>>>> else >>>>> FLUME_CLASSPATH="$FLUME_CLASSPATH:$PIECE" >>>>> fi >>>>> done >>>>> done >>>>> FLUME_CLASSPATH="$FLUME_CLASSPATH:$HBASE_HOME/conf" >>>>> >>>>> fi >>>>> } >>>>> >>>>> display_help() { >>>>> cat <<EOF >>>>> Usage: $0 <command> [options]... >>>>> >>>>> commands: >>>>> help display this help text >>>>> agent run a Flume agent >>>>> avro-client run an avro Flume client >>>>> version show Flume version info >>>>> >>>>> global options: >>>>> --conf,-c <conf> use configs in <conf> directory >>>>> --classpath,-C <cp> append to the classpath >>>>> --dryrun,-d do not actually start Flume, just print the >>>>> command >>>>> -Dproperty=value sets a JDK system property value >>>>> >>>>> agent options: >>>>> --conf-file,-f <file> specify a config file (required) >>>>> --name,-n <name> the name of this agent (required) >>>>> --help,-h display help text >>>>> >>>>> avro-client options: >>>>> --host,-H <host> hostname to which events will be sent >>>>> (required) >>>>> --port,-p <port> port of the avro source (required) >>>>> --filename,-F <file> text file to stream to avro source [default: >>>>> std input] >>>>> --headerFile,-R <file> headerFile containing headers as key/value >>>>> pairs on each new line >>>>> --help,-h display help text >>>>> >>>>> Note that if <conf> directory is specified, then it is always included >>>>> first >>>>> in the classpath. >>>>> >>>>> EOF >>>>> } >>>>> >>>>> run_flume() { >>>>> local FLUME_APPLICATION_CLASS >>>>> >>>>> if [ "$#" -gt 0 ]; then >>>>> FLUME_APPLICATION_CLASS=$1 >>>>> shift >>>>> else >>>>> error "Must specify flume application class" 1 >>>>> fi >>>>> >>>>> if [ ${CLEAN_FLAG} -ne 0 ]; then >>>>> set -x >>>>> fi >>>>> $EXEC $JAVA_HOME/bin/java $JAVA_OPTS -cp "$FLUME_CLASSPATH" \ >>>>> -Djava.library.path=$FLUME_JAVA_LIBRARY_PATH >>>>> "$FLUME_APPLICATION_CLASS" $* >>>>> } >>>>> >>>>> ################################ >>>>> # main >>>>> ################################ >>>>> >>>>> # set default params >>>>> FLUME_CLASSPATH="" >>>>> FLUME_JAVA_LIBRARY_PATH="" >>>>> JAVA_OPTS="-Xmx20m" >>>>> >>>>> opt_conf="" >>>>> opt_classpath="" >>>>> opt_java_props="" >>>>> opt_dryrun="" >>>>> >>>>> mode=$1 >>>>> shift >>>>> >>>>> case "$mode" in >>>>> help) >>>>> display_help >>>>> exit 0 >>>>> ;; >>>>> agent) >>>>> opt_agent=1 >>>>> ;; >>>>> node) >>>>> opt_agent=1 >>>>> warn "The \"node\" command is deprecated. Please use \"agent\" >>>>> instead." >>>>> ;; >>>>> avro-client) >>>>> opt_avro_client=1 >>>>> ;; >>>>> version) >>>>> opt_version=1 >>>>> CLEAN_FLAG=0 >>>>> ;; >>>>> *) >>>>> error "Unknown or unspecified command '$mode'" >>>>> echo >>>>> display_help >>>>> exit 1 >>>>> ;; >>>>> esac >>>>> >>>>> while [ -n "$*" ] ; do >>>>> arg=$1 >>>>> shift >>>>> >>>>> case "$arg" in >>>>> --conf|-c) >>>>> [ -n "$1" ] || error "Option --conf requires an argument" 1 >>>>> opt_conf=$1 >>>>> shift >>>>> ;; >>>>> --classpath|-C) >>>>> [ -n "$1" ] || error "Option --classpath requires an argument" 1 >>>>> opt_classpath=$1 >>>>> shift >>>>> ;; >>>>> --dryrun|-d) >>>>> opt_dryrun="1" >>>>> ;; >>>>> -D*) >>>>> opt_java_props="$opt_java_props $arg" >>>>> ;; >>>>> *) >>>>> args="$args $arg" >>>>> ;; >>>>> esac >>>>> done >>>>> >>>>> # make opt_conf absolute >>>>> if [[ -n "$opt_conf" && -d "$opt_conf" ]]; then >>>>> opt_conf=$(cd $opt_conf; pwd) >>>>> fi >>>>> >>>>> # allow users to override the default env vars via conf/flume-env.sh >>>>> if [ -z "$opt_conf" ]; then >>>>> warn "No configuration directory set! Use --conf <dir> to override." >>>>> elif [ -f "$opt_conf/flume-env.sh" ]; then >>>>> info "Sourcing environment configuration script >>>>> $opt_conf/flume-env.sh" >>>>> source "$opt_conf/flume-env.sh" >>>>> fi >>>>> >>>>> # append command-line java options to stock or env script JAVA_OPTS >>>>> if [ -n "${opt_java_props}" ]; then >>>>> JAVA_OPTS="${JAVA_OPTS} ${opt_java_props}" >>>>> fi >>>>> >>>>> # prepend command-line classpath to env script classpath >>>>> if [ -n "${opt_classpath}" ]; then >>>>> if [ -n "${FLUME_CLASSPATH}" ]; then >>>>> FLUME_CLASSPATH="${opt_classpath}:${FLUME_CLASSPATH}" >>>>> else >>>>> FLUME_CLASSPATH="${opt_classpath}" >>>>> fi >>>>> fi >>>>> >>>>> if [ -z "${FLUME_HOME}" ]; then >>>>> FLUME_HOME=$(cd $(dirname $0)/..; pwd) >>>>> fi >>>>> >>>>> # prepend $FLUME_HOME/lib jars to the specified classpath (if any) >>>>> if [ -n "${FLUME_CLASSPATH}" ] ; then >>>>> FLUME_CLASSPATH="${FLUME_HOME}/lib/*:$FLUME_CLASSPATH" >>>>> else >>>>> FLUME_CLASSPATH="${FLUME_HOME}/lib/*" >>>>> fi >>>>> >>>>> # find java >>>>> if [ -z "${JAVA_HOME}" ] ; then >>>>> warn "JAVA_HOME is not set!" >>>>> # Try to use Bigtop to autodetect JAVA_HOME if it's available >>>>> if [ -e /usr/libexec/bigtop-detect-javahome ] ; then >>>>> . /usr/libexec/bigtop-detect-javahome >>>>> elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ] ; then >>>>> . /usr/lib/bigtop-utils/bigtop-detect-javahome >>>>> fi >>>>> >>>>> # Using java from path if bigtop is not installed or couldn't find it >>>>> if [ -z "${JAVA_HOME}" ] ; then >>>>> JAVA_DEFAULT=$(type -p java) >>>>> [ -n "$JAVA_DEFAULT" ] || error "Unable to find java executable. >>>>> Is it in your PATH?" 1 >>>>> JAVA_HOME=$(cd $(dirname $JAVA_DEFAULT)/..; pwd) >>>>> fi >>>>> fi >>>>> >>>>> # look for hadoop libs >>>>> add_hadoop_paths >>>>> add_HBASE_paths >>>>> >>>>> # prepend conf dir to classpath >>>>> if [ -n "$opt_conf" ]; then >>>>> FLUME_CLASSPATH="$opt_conf:$FLUME_CLASSPATH" >>>>> fi >>>>> >>>>> # allow dryrun >>>>> EXEC="exec" >>>>> if [ -n "${opt_dryrun}" ]; then >>>>> warn "Dryrun mode enabled (will not actually initiate startup)" >>>>> EXEC="echo" >>>>> fi >>>>> >>>>> # finally, invoke the appropriate command >>>>> if [ -n "$opt_agent" ] ; then >>>>> run_flume $FLUME_AGENT_CLASS $args >>>>> elif [ -n "$opt_avro_client" ] ; then >>>>> run_flume $FLUME_AVRO_CLIENT_CLASS $args >>>>> elif [ -n "${opt_version}" ] ; then >>>>> run_flume $FLUME_VERSION_CLASS $args >>>>> else >>>>> error "This message should never appear" 1 >>>>> fi >>>>> >>>>> exit 0 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Inder Pall <[email protected]> >>>>> July 30, 2012 3:40 PM >>>>> Hey if you edit flume-ng script and just before java is invoked put an >>>>> >>>>> echo $CLASSPATH that should help in figuring what all dir's are there >>>>> in CLASSPATH. >>>>> I am sorry but i am a big fan of windows hence don't run that O.S. to >>>>> try it out. >>>>> >>>>> Thanks, >>>>> - inder >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> - Inder >>>>> Tech Platforms @Inmobi >>>>> Linkedin - http://goo.gl/eR4Ub >>>>> >>>>> >>>> >>> >> >
<<compose-unknown-contact.jpg>>
<<postbox-contact.jpg>>
<<postbox-contact.jpg>>
