Download DataTorrent VM and executed same apa file there...
every thing working... seems like setup issue in my earlier VM (Cloudera
5.6 and apache kafka 0.9)
Thank you all for quick responses
On Fri, May 5, 2017 at 5:42 PM, Sushil Apex <sushil.aa...@gmail.com> wrote:
> thanks @vikram, now this is running in actual yarn
>
> container log has nothing
>
> greping the application id in node manager logs is below
>
> 2017-05-05 17:26:49,822 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.ContainerManagerImpl: Creating a new
> application reference for app application_1493983487923_0001
> 2017-05-05 17:26:49,840 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.application.Application: Application
> application_1493983487923_0001 transitioned from NEW to INITING
> 2017-05-05 17:26:49,895 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
> USER=cloudera IP=127.0.0.1 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1493983487923_0001
> CONTAINERID=container_1493983487923_0001_01_000001
> 2017-05-05 17:26:52,333 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.application.Application: Adding
> container_1493983487923_0001_01_000001 to application
> application_1493983487923_0001
> 2017-05-05 17:26:52,339 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.application.Application: Application
> application_1493983487923_0001 transitioned from INITING to RUNNING
> 2017-05-05 17:26:52,360 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for
> appId application_1493983487923_0001
> 2017-05-05 17:26:52,471 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Copying from /yarn/nm/nmPrivate/container_1493983487923_0001_01_000001.tokens
> to /yarn/nm/usercache/cloudera/appcache/application_
> 1493983487923_0001/container_1493983487923_0001_01_000001.tokens
> 2017-05-05 17:26:52,471 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Localizer CWD set to
> /yarn/nm/usercache/cloudera/appcache/application_1493983487923_0001
> = file:/yarn/nm/usercache/cloudera/appcache/application_1493983487923_0001
> 2017-05-05 17:26:57,697 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> launchContainer: [bash, /yarn/nm/usercache/cloudera/appcache/application_
> 1493983487923_0001/container_1493983487923_0001_01_000001/
> default_container_executor.sh]
> 2017-05-05 17:27:22,592 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.application.Application: Adding
> container_1493983487923_0001_01_000002 to application
> application_1493983487923_0001
> 2017-05-05 17:27:22,592 INFO
> org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
> USER=cloudera IP=127.0.0.1 OPERATION=Start Container Request
> TARGET=ContainerManageImpl RESULT=SUCCESS
> APPID=application_1493983487923_0001
> CONTAINERID=container_1493983487923_0001_01_000002
> 2017-05-05 17:27:22,597 INFO org.apache.hadoop.yarn.server.
> nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for
> appId application_1493983487923_0001
> 2017-05-05 17:27:22,743 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> launchContainer: [bash, /yarn/nm/usercache/cloudera/appcache/application_
> 1493983487923_0001/container_1493983487923_0001_01_000002/
> default_container_executor.sh]
>
>
>
> resourceMnaager logs
>
> 2017-05-05 17:26:45,749 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
> USER=cloudera IP=127.0.0.1 OPERATION=Submit Application Request
> TARGET=ClientRMService RESULT=SUCCESS APPID=application_
> 1493983487923_0001
> 2017-05-05 17:26:45,749 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.RMAppImpl: Storing application with id
> application_1493983487923_0001
> 2017-05-05 17:26:46,290 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.RMAppImpl: application_1493983487923_0001 State
> change from NEW to NEW_SAVING
> 2017-05-05 17:26:46,290 INFO org.apache.hadoop.yarn.server.
> resourcemanager.recovery.RMStateStore: Storing info for app:
> application_1493983487923_0001
> 2017-05-05 17:26:46,565 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.RMAppImpl: application_1493983487923_0001 State
> change from NEW_SAVING to SUBMITTED
> 2017-05-05 17:26:46,603 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.fair.FairScheduler: Accepted application
> application_1493983487923_0001 from user: cloudera, in queue: default,
> currently num of applications: 1
> 2017-05-05 17:26:46,609 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.RMAppImpl: application_1493983487923_0001 State
> change from SUBMITTED to ACCEPTED
> 2017-05-05 17:26:47,569 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
> USER=cloudera OPERATION=AM Allocated Container TARGET=SchedulerApp
> RESULT=SUCCESS APPID=application_1493983487923_0001
> CONTAINERID=container_1493983487923_0001_01_000001
> 2017-05-05 17:26:47,789 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId:
> application_1493983487923_0001 AttemptId: appattempt_1493983487923_0001_000001
> MasterContainer: Container: [ContainerId:
> container_1493983487923_0001_01_000001,
> NodeId: quickstart.cloudera:8041, NodeHttpAddress:
> quickstart.cloudera:8042, Resource: <memory:1024, vCores:1>, Priority: 0,
> Token: Token { kind: ContainerToken, service: 127.0.0.1:8041 }, ]
> 2017-05-05 17:26:48,253 INFO org.apache.hadoop.yarn.server.
> resourcemanager.amlauncher.AMLauncher: Command to launch container
> container_1493983487923_0001_01_000001 : ${JAVA_HOME}/bin/java
> -Djava.io.tmpdir=$PWD/tmp -Xmx768m -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/dt-heap-1.bin -Dhadoop.root.logger=INFO,RFA
> -Dhadoop.log.dir=<LOG_DIR> -Ddt.attr.APPLICATION_PATH=
> hdfs://quickstart.cloudera:8020/user/cloudera/
> datatorrent/apps/application_1493983487923_0001
> com.datatorrent.stram.StreamingAppMaster
> 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr
> 2017-05-05 17:27:18,526 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.RMAppImpl: application_1493983487923_0001 State
> change from ACCEPTED to RUNNING
> 2017-05-05 17:27:18,526 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
> USER=cloudera IP=127.0.0.1 OPERATION=Register App Master
> TARGET=ApplicationMasterService
> RESULT=SUCCESS APPID=application_1493983487923_0001
> APPATTEMPTID=appattempt_1493983487923_0001_000001
> 2017-05-05 17:27:20,197 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger:
> USER=cloudera OPERATION=AM Allocated Container TARGET=SchedulerApp
> RESULT=SUCCESS APPID=application_1493983487923_0001
> CONTAINERID=container_1493983487923_0001_01_000002
> 2017-05-05 17:27:20,203 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.fair.FSAppAttempt: Making reservation:
> node=quickstart.cloudera app_id=application_1493983487923_0001
> 2017-05-05 17:27:20,205 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.fair.FSSchedulerNode: Reserved container
> container_1493983487923_0001_01_000003 on node host:
> quickstart.cloudera:8041 #containers=2 available=1024 used=2048 for
> application application_1493983487923_0001
> 2017-05-05 17:27:52,911 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.fair.FSAppAttempt: Application
> application_1493983487923_0001 unreserved on node host:
> quickstart.cloudera:8041 #containers=2 available=1024 used=2048, currently
> has 0 at priority 1; currentReservation <memory:0, vCores:0>
>
>
>
>
> On Fri, May 5, 2017 at 5:11 PM, Vikram Patil <vik...@datatorrent.com>
> wrote:
>
>> Actually you are launching app in local mode.
>>
>> Instead of
>> launch -local /home/cloudera/myapexapp-1.0-SNAPSHOT.apa
>>
>> launch as:
>> launch /home/cloudera/myapexapp-1.0-SNAPSHOT.apa
>>
>> Then it should launch an application using Yarn.
>>
>> Thanks & Regards,
>> Vikram
>>
>> On Fri, May 5, 2017 at 5:04 PM, Sushil Apex <sushil.aa...@gmail.com>
>> wrote:
>>
>>> @tushar
>>>
>>> I have used apex-cli in one terminal and launched my application with
>>> below command, in another terminal, i checked for application id but it is
>>> coming zero active applications, I must be missing some thing here.
>>>
>>>
>>>
>>> apex> launch -local /home/cloudera/myapexapp-1.0-SNAPSHOT.apa
>>> 1. Kafka2HDFS2
>>> 2. MyFirstApplication
>>> Choose application: 1
>>>
>>>
>>>
>>> Apex CLI 3.5.0 06.12.2016 @ 22:11:51 PST rev: 6de8828 branch:
>>> 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8
>>> apex> list-apps *
>>> {"apps": []}
>>> 0 active, total 0 applications.
>>>
>>>
>>> @chaitanya
>>> hdfs and yarn are running, but the container folder is blank
>>>
>>> [cloudera@quickstart bin]$ cd /var/log/hadoop-yarn/container
>>> [cloudera@quickstart container]$ ls
>>> [cloudera@quickstart container]$
>>>
>>>
>>>
>>> On Fri, May 5, 2017 at 4:41 PM, Tushar Gosavi <tus...@datatorrent.com>
>>> wrote:
>>>
>>>> If log aggregations is enabled you can get the logs using yarn logs
>>>> command. you can get the application id using apex-cli.
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Fri, May 5, 2017 at 4:33 PM, Sushil Apex <sushil.aa...@gmail.com>
>>>> wrote:
>>>>
>>>>> To add here
>>>>> I am using Cloudera VM for hdfs and running Kafka from apache with
>>>>> single machine zookeeper
>>>>>
>>>>> On Fri, May 5, 2017 at 4:32 PM, Sushil Apex <sushil.aa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> @vikram, I am using apex cli
>>>>>> not sure where ts logs will be generated, checked
>>>>>> /var/log/hadoop-yarn but did found any updated logs for apex
>>>>>>
>>>>>> I have changed the application name but that didn't helped
>>>>>>
>>>>>> On Fri, May 5, 2017 at 4:03 PM, vikram patil <patilvik...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Can you share your Apex logs? Have you launched an application using
>>>>>>> apex cli?
>>>>>>>
>>>>>>> Also, can you try once by changing name of your application and
>>>>>>> relaunch again. Please kill or shut down your existing running
>>>>>>> application before this.
>>>>>>>
>>>>>>> -Vikram
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 5, 2017 at 3:56 PM, Sushil Apex <sushil.aa...@gmail.com>
>>>>>>> wrote:
>>>>>>> > Tried with EARLIEST option also, no luck :(
>>>>>>> >
>>>>>>> > On Fri, May 5, 2017 at 3:49 PM, Sushil Apex <
>>>>>>> sushil.aa...@gmail.com> wrote:
>>>>>>> >>
>>>>>>> >> @Vikram the folder /tmp/FromKafka is already created.
>>>>>>> >> @Chaitanya yes I am pusing messages in kafka topic using
>>>>>>> consoleProducer
>>>>>>> >> bin/kafka-console-producer.sh --broker-list localhost:9092
>>>>>>> --topic
>>>>>>> >> kafka2hdfs
>>>>>>> >>
>>>>>>> >> @Ajay
>>>>>>> >> will try with the initial offset to EARLIEST.
>>>>>>> >>
>>>>>>> >> Thank you
>>>>>>> >> Sushil
>>>>>>> >>
>>>>>>> >> On Fri, May 5, 2017 at 2:54 PM, AJAY GUPTA <ajaygit...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Also, you may be required to specify the initial offset to
>>>>>>> EARLIEST.
>>>>>>> >>>
>>>>>>> >>> Ajay
>>>>>>> >>>
>>>>>>> >>> On Fri, May 5, 2017 at 2:35 PM, vikram patil <
>>>>>>> patilvik...@gmail.com>
>>>>>>> >>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> Hi Sushil,
>>>>>>> >>>>
>>>>>>> >>>> Have you provided configuration specifying hdfs directory and
>>>>>>> file for
>>>>>>> >>>> an application?
>>>>>>> >>>> You may have to create /tmp/fromKafka directory in hdfs.
>>>>>>> >>>> Thanks & Regards,
>>>>>>> >>>> Vikram
>>>>>>> >>>>
>>>>>>> >>>> On Fri, May 5, 2017 at 2:30 PM, Sushil Apex <
>>>>>>> sushil.aa...@gmail.com>
>>>>>>> >>>> wrote:
>>>>>>> >>>> >
>>>>>>> >>>> > I am using Apex 3.5.0 and kafka 0.9
>>>>>>> >>>> > malhar library used is 3.6.0
>>>>>>> >>>> >
>>>>>>> >>>> > I am following the example from
>>>>>>> >>>> > https://github.com/DataTorrent/examples/blob/master/tutorial
>>>>>>> s/kafka/src/main/java/com/example/myapexapp/KafkaApp.java
>>>>>>> >>>> >
>>>>>>> >>>> > I am putting messages in Kafka topic using
>>>>>>> consolekafkaProducer
>>>>>>> >>>> > provided by Kafka, but I am not able to read these messages
>>>>>>> in Apex
>>>>>>> >>>> > DAG(created based on above link).
>>>>>>> >>>> >
>>>>>>> >>>> > I am running the apex apa file through apex-cli
>>>>>>> >>>> >
>>>>>>> >>>> > Apex CLI 3.5.0 06.12.2016 @ 22:11:51 PST rev: 6de8828 branch:
>>>>>>> >>>> > 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8
>>>>>>> >>>> > apex> launch --local /home/cloudera/myapexapp-1.0-S
>>>>>>> NAPSHOT.apa
>>>>>>> >>>> > 1. Kafka2HDFS
>>>>>>> >>>> > 2. MyFirstApplication
>>>>>>> >>>> > Choose application: 1
>>>>>>> >>>> >
>>>>>>> >>>> > and nothing happens after this, I can see the messages put in
>>>>>>> >>>> > consoleConsumer in kafka logs
>>>>>>> >>>> >
>>>>>>> >>>> > Properties used are
>>>>>>> >>>> >
>>>>>>> >>>> > <property>
>>>>>>> >>>> > <name>dt.operator.kafkaIn.prop.topics</name>
>>>>>>> >>>> > <value>kafka2hdfs</value>
>>>>>>> >>>> > </property>
>>>>>>> >>>> >
>>>>>>> >>>> > <property>
>>>>>>> >>>> > <name>dt.operator.kafkaIn.prop.consumer.zookeeper</name>
>>>>>>> >>>> > <value>localhost:2181</value>
>>>>>>> >>>> > </property>
>>>>>>> >>>> > <property>
>>>>>>> >>>> > <name>dt.operator.kafkaIn.prop.clusters</name>
>>>>>>> >>>> > <value>localhost:9092</value>
>>>>>>> >>>> > </property>
>>>>>>> >>>> > <property>
>>>>>>> >>>> > <name>dt.operator.kafkaIn.prop.initialPartitionCount</name>
>>>>>>> >>>> > <value>1</value>
>>>>>>> >>>> > </property>
>>>>>>> >>>> >
>>>>>>> >>>> >
>>>>>>> >>>> > Application Code
>>>>>>> >>>> >
>>>>>>> >>>> > KafkaSinglePortByteArrayInputOperator in
>>>>>>> >>>> > = dag.addOperator("kafkaIn", new
>>>>>>> >>>> > KafkaSinglePortByteArrayInputOperator());
>>>>>>> >>>> >
>>>>>>> >>>> > LineOutputOperator out = dag.addOperator("fileOut",
>>>>>>> new
>>>>>>> >>>> > LineOutputOperator());
>>>>>>> >>>> >
>>>>>>> >>>> > dag.addStream("dataf", in.outputPort, out.input);
>>>>>>> >>>> >
>>>>>>> >>>> >
>>>>>>> >>>> > I am not able to understand what config I am missing here?
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
