Re: Malhar 0.8.1.1 Kafka Operator and chroot

2016-08-25 Thread McCullough, Alex 
Hey Siyuan,

Looks like using the single ZK IP with the chroot works, but fails with the 
same error when adding more than one.

Can explain the exact impact of only having the single zk node listed? It can 
auto discover with ZK the other nodes, but if the one we connect to drops than 
Apex won’t know any other server to connect to?

Thanks,
Alex

From: "hsy...@gmail.com" 
Reply-To: "users@apex.apache.org" 
Date: Thursday, August 25, 2016 at 12:57 PM
To: "users@apex.apache.org" 
Subject: Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey Alex,

Does the workaround work? I just want to follow up to see my hypothesis for the 
root cause is correct. Thanks!

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:56 AM, hsy...@gmail.com 
mailto:hsy...@gmail.com>> wrote:
Hey Alex,

Yeah, I think there is a bug for multitenant kafka support in the code. I have 
created a ticket
https://issues.apache.org/jira/browse/APEXMALHAR-2199

For now can you try one thing:
Can you try to set your zookeeper to something like this:

   dt.operator.kafkaInputOperator.prop.consumer.zookeeper
   
10.##.##.#:2181/kafka2,10.##.##.##:2181/kafka2,10.##.##.##:2181/kafka2,10.##.##.#:2181/kafka2


or you can just try to set just one of the zookeeper nodes.
For kafka client it only needs to know one running node but you'll lose 
zookeeper HA

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:40 AM, McCullough, Alex 
mailto:alex.mccullo...@capitalone.com>> wrote:
ONE_TO_ONE



From: "hsy...@gmail.com" 
mailto:hsy...@gmail.com>>
Reply-To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Date: Wednesday, August 24, 2016 at 1:38 PM

To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Subject: Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey Alex,

Do you use ONE_TO_ONE or ONE_TO_MANY partition?

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:27 AM, McCullough, Alex 
mailto:alex.mccullo...@capitalone.com>> wrote:
Hey Siyuan,

We are using 3.4.0

Thanks,
Alex
From: "hsy...@gmail.com" 
mailto:hsy...@gmail.com>>
Reply-To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Date: Wednesday, August 24, 2016 at 12:47 PM
To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Subject: Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey McCullough,

What malhar version do you use?

Regards,
Siyuan

On Wed, Aug 24, 2016 at 9:07 AM, McCullough, Alex 
mailto:alex.mccullo...@capitalone.com>> wrote:
Hey All,

We are using the 0.8.1 kafka operator and the ZK connection string has a chroot 
on it. We get errors when launching and the app fails, is there a proper way in 
apex to append a chroot?


*the ip’s are masked with #, but that’s not how they appear in our code 
obviously*

When we add this to the property for ZK:



   dt.operator.kafkaInputOperator.prop.consumer.zookeeper
   
10.##.##.#:2181,10.##.##.##:2181,10.##.##.##:2181,10.##.##.#:2181/kafka2




We get this error (connecting to a cluster without chroot it works fine):


2016-08-24 11:55:13,448 [main-EventThread] INFO  zkclient.ZkClient 
processStateChanged - zookeeper state changed (SyncConnected)
2016-08-24 11:55:13,585 
[ZkClient-EventThread-30-10.##.##.#:2181,10.##.##.##:2181,10.##.##.#:2181/kafka2,10.##.##.##:2181]
 INFO  zkclient.ZkEventThread run - Terminate ZkClient event thread.
2016-08-24 11:55:13,596 [main] INFO  zookeeper.ZooKeeper close - Session: 
0x4558654aacf4263 closed
2016-08-24 11:55:13,596 [main-EventThread] INFO  zookeeper.ClientCnxn run - 
EventThread shut down
2016-08-24 11:55:13,597 [main] INFO  kafka.AbstractKafkaInputOperator 
definePartitions - [ONE_TO_ONE]: Initializing partition(s)
2016-08-24 11:55:13,602 [main] INFO  service.AbstractService noteFailure - 
Service com.datatorrent.stram.StreamingAppMasterService failed in state INITED; 
cause: java.lang.IllegalArgumentException: there has to be one idempotent 
storage manager
java.lang.IllegalArgumentException: there has to be one idempotent storage 
manager
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93)
at 
org.apache.apex.malhar.lib.wal.FSWindowDataManager.partitioned(FSWindowDataManager.java:251)
at 
com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:637)
at 
com.datatorrent.stram.plan.physical.PhysicalPlan.initPartitioning(PhysicalPlan.java:752)
at 
com.datatorrent.stram.plan.physical.PhysicalPlan.addLogicalOperator(PhysicalPlan.java:1676)
at 
com.datatorrent.stram.plan.physical.PhysicalPlan.(PhysicalPlan.java:378)
at 
com.datatorrent.stram.StreamingContainerManager.(StreamingContainerManager.java:418)
at 
com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:3023)
at 
com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterS

FileOutputOperator Rolling Windows Based on Time

2016-08-25 Thread McCullough, Alex 
Hey Everyone,

I am trying to set up my file output operator so that the files roll every hour.

My thought is that in the operator if I know the Streaming Window Size and the 
Application Window Count, I can easily enough calculate the value I should set 
for RotationWindows property.

I’ve been able to get all the attributes in to my operator, but it’s pretty 
ugly, pull them from the conf in App class and pass them to the operator, and 
things like this but it doesn’t feel like the proper pattern. I can get the App 
window count from the operator context if I set it as a property but I can’t 
get the streaming window millis and if I don’t set the app window count then I 
can’t get that either. Is there a recommended pattern to get application 
attributes inside of an operator?

Thanks,
Alex


The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Re: Sandbox Question - Set up more than 8 GIG RAM

2016-08-25 Thread Feldkamp, Brandon (CONT) 
Hey Jim,

I had to make adjustments to the default memory settings which I accomplished 
by doing the following:

The setting is located in /sfw/hadoop/conf/yarn-site.xml. You should see:

1.  yarn.nodemanager.resource.memory-mb (Amount of physical memory, in MB, 
that can be allocated for containers.)

2.  yarn.scheduler.maximum-allocation-mb (The maximum allocation for every 
container request at the RM, in MBs. Memory requests higher than this won't 
take effect, and will get capped to this value.)

Reference: 
https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

You’ll need to at least bounce the resource manager by executing “sudo 
/sfw/dtbox/dtbox-3.4.0/conf/init.d/hadoop-yarn-resourcemanager restart” to pick 
up the config change. I’m not sure if you need to restart any other services.

Hope this helps!
Brandon

On 2016-08-25 13:33 (-0400), Jim 
mailto:j...@facility.supplies>> wrote:
> Good afternoon,>
>
> I have a sandbox question.  I have three applications, that when run together 
> currently take around 16 gig of ram.>
>
> On my laptop, I have 32 gig of ram, and have allocated 20 GIG to the virtual 
> maching via virtual box.>
>
> However, when I look into the Hadoop console, I only see 8 GB ram available, 
> which prevents some of my apps from starting.>
>
> What do I change, and where do I change it in the sandbox (version 3.4.0), to 
> allow all the memory available to the virtual machine to be used for the 
> datatorrent applications?>
>
> Thanks,>
>
> JIm>
>
> [cid:image001.png@01D1FECC.F006DC60]>
>


The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Re: Malhar 0.8.1.1 Kafka Operator and chroot

2016-08-25 Thread McCullough, Alex 
Hey Siyuan,

Sorry for the delay ran into some unrelated cluster issues yesterday that 
prevented me from testing.

Working on testing this this afternoon and I will let you know.

Thanks,
Alex


From: "hsy...@gmail.com" 
Reply-To: "users@apex.apache.org" 
Date: Thursday, August 25, 2016 at 12:57 PM
To: "users@apex.apache.org" 
Subject: Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey Alex,

Does the workaround work? I just want to follow up to see my hypothesis for the 
root cause is correct. Thanks!

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:56 AM, hsy...@gmail.com 
mailto:hsy...@gmail.com>> wrote:
Hey Alex,

Yeah, I think there is a bug for multitenant kafka support in the code. I have 
created a ticket
https://issues.apache.org/jira/browse/APEXMALHAR-2199

For now can you try one thing:
Can you try to set your zookeeper to something like this:

   dt.operator.kafkaInputOperator.prop.consumer.zookeeper
   
10.##.##.#:2181/kafka2,10.##.##.##:2181/kafka2,10.##.##.##:2181/kafka2,10.##.##.#:2181/kafka2


or you can just try to set just one of the zookeeper nodes.
For kafka client it only needs to know one running node but you'll lose 
zookeeper HA

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:40 AM, McCullough, Alex 
mailto:alex.mccullo...@capitalone.com>> wrote:
ONE_TO_ONE



From: "hsy...@gmail.com" 
mailto:hsy...@gmail.com>>
Reply-To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Date: Wednesday, August 24, 2016 at 1:38 PM

To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Subject: Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey Alex,

Do you use ONE_TO_ONE or ONE_TO_MANY partition?

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:27 AM, McCullough, Alex 
mailto:alex.mccullo...@capitalone.com>> wrote:
Hey Siyuan,

We are using 3.4.0

Thanks,
Alex
From: "hsy...@gmail.com" 
mailto:hsy...@gmail.com>>
Reply-To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Date: Wednesday, August 24, 2016 at 12:47 PM
To: "users@apex.apache.org" 
mailto:users@apex.apache.org>>
Subject: Re: Malhar 0.8.1.1 Kafka Operator and chroot

Hey McCullough,

What malhar version do you use?

Regards,
Siyuan

On Wed, Aug 24, 2016 at 9:07 AM, McCullough, Alex 
mailto:alex.mccullo...@capitalone.com>> wrote:
Hey All,

We are using the 0.8.1 kafka operator and the ZK connection string has a chroot 
on it. We get errors when launching and the app fails, is there a proper way in 
apex to append a chroot?


*the ip’s are masked with #, but that’s not how they appear in our code 
obviously*

When we add this to the property for ZK:



   dt.operator.kafkaInputOperator.prop.consumer.zookeeper
   
10.##.##.#:2181,10.##.##.##:2181,10.##.##.##:2181,10.##.##.#:2181/kafka2




We get this error (connecting to a cluster without chroot it works fine):


2016-08-24 11:55:13,448 [main-EventThread] INFO  zkclient.ZkClient 
processStateChanged - zookeeper state changed (SyncConnected)
2016-08-24 11:55:13,585 
[ZkClient-EventThread-30-10.##.##.#:2181,10.##.##.##:2181,10.##.##.#:2181/kafka2,10.##.##.##:2181]
 INFO  zkclient.ZkEventThread run - Terminate ZkClient event thread.
2016-08-24 11:55:13,596 [main] INFO  zookeeper.ZooKeeper close - Session: 
0x4558654aacf4263 closed
2016-08-24 11:55:13,596 [main-EventThread] INFO  zookeeper.ClientCnxn run - 
EventThread shut down
2016-08-24 11:55:13,597 [main] INFO  kafka.AbstractKafkaInputOperator 
definePartitions - [ONE_TO_ONE]: Initializing partition(s)
2016-08-24 11:55:13,602 [main] INFO  service.AbstractService noteFailure - 
Service com.datatorrent.stram.StreamingAppMasterService failed in state INITED; 
cause: java.lang.IllegalArgumentException: there has to be one idempotent 
storage manager
java.lang.IllegalArgumentException: there has to be one idempotent storage 
manager
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93)
at 
org.apache.apex.malhar.lib.wal.FSWindowDataManager.partitioned(FSWindowDataManager.java:251)
at 
com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:637)
at 
com.datatorrent.stram.plan.physical.PhysicalPlan.initPartitioning(PhysicalPlan.java:752)
at 
com.datatorrent.stram.plan.physical.PhysicalPlan.addLogicalOperator(PhysicalPlan.java:1676)
at 
com.datatorrent.stram.plan.physical.PhysicalPlan.(PhysicalPlan.java:378)
at 
com.datatorrent.stram.StreamingContainerManager.(StreamingContainerManager.java:418)
at 
com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:3023)
at 
com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:551)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
com.datatorrent.stram.StreamingAppMaster.main(Stre

Re: Malhar 0.8.1.1 Kafka Operator and chroot

2016-08-25 Thread hsy...@gmail.com 
Hey Alex,

Does the workaround work? I just want to follow up to see my hypothesis for
the root cause is correct. Thanks!

Regards,
Siyuan

On Wed, Aug 24, 2016 at 10:56 AM, hsy...@gmail.com  wrote:

> Hey Alex,
>
> Yeah, I think there is a bug for multitenant kafka support in the code. I
> have created a ticket
> https://issues.apache.org/jira/browse/APEXMALHAR-2199
>
> For now can you try one thing:
> Can you try to set your zookeeper to something like this:
>
> 
>
>dt.operator.kafkaInputOperator.prop.consumer.zookeeper
>
>10.##.##.#:2181/kafka2,10.##.##.##:2181/kafka2,10.##.
> ##.##:2181/kafka2,10.##.##.#:2181/kafka2
>
> 
>
>
> or you can just try to set just one of the zookeeper nodes.
>
> For kafka client it only needs to know one running node but you'll lose
> zookeeper HA
>
>
> Regards,
>
> Siyuan
>
> On Wed, Aug 24, 2016 at 10:40 AM, McCullough, Alex <
> alex.mccullo...@capitalone.com> wrote:
>
>> ONE_TO_ONE
>>
>>
>>
>>
>>
>>
>>
>> *From: *"hsy...@gmail.com" 
>> *Reply-To: *"users@apex.apache.org" 
>> *Date: *Wednesday, August 24, 2016 at 1:38 PM
>>
>> *To: *"users@apex.apache.org" 
>> *Subject: *Re: Malhar 0.8.1.1 Kafka Operator and chroot
>>
>>
>>
>> Hey Alex,
>>
>>
>>
>> Do you use ONE_TO_ONE or ONE_TO_MANY partition?
>>
>>
>>
>> Regards,
>>
>> Siyuan
>>
>>
>>
>> On Wed, Aug 24, 2016 at 10:27 AM, McCullough, Alex <
>> alex.mccullo...@capitalone.com> wrote:
>>
>> Hey Siyuan,
>>
>>
>>
>> We are using 3.4.0
>>
>>
>>
>> Thanks,
>>
>> Alex
>>
>> *From: *"hsy...@gmail.com" 
>> *Reply-To: *"users@apex.apache.org" 
>> *Date: *Wednesday, August 24, 2016 at 12:47 PM
>> *To: *"users@apex.apache.org" 
>> *Subject: *Re: Malhar 0.8.1.1 Kafka Operator and chroot
>>
>>
>>
>> Hey McCullough,
>>
>>
>>
>> What malhar version do you use?
>>
>>
>>
>> Regards,
>>
>> Siyuan
>>
>>
>>
>> On Wed, Aug 24, 2016 at 9:07 AM, McCullough, Alex <
>> alex.mccullo...@capitalone.com> wrote:
>>
>> Hey All,
>>
>>
>>
>> We are using the 0.8.1 kafka operator and the ZK connection string has a
>> chroot on it. We get errors when launching and the app fails, is there a
>> proper way in apex to append a chroot?
>>
>>
>>
>>
>>
>> **the ip’s are masked with #, but that’s not how they appear in our code
>> obviously**
>>
>>
>>
>> When we add this to the property for ZK:
>>
>>
>>
>>
>>
>> 
>>
>>dt.operator.kafkaInputOperator.prop.consumer.
>> zookeeper
>>
>>10.##.##.#:2181,10.##.##.##:2181,10.##.##.##:2181,10.
>> ##.##.#:2181/kafka2
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> We get this error (connecting to a cluster without chroot it works fine):
>>
>>
>>
>>
>>
>> 2016-08-24 11:55:13,448 [main-EventThread] INFO  zkclient.ZkClient
>> processStateChanged - zookeeper state changed (SyncConnected)
>>
>> 2016-08-24 11:55:13,585 [ZkClient-EventThread-30-10.##
>> .##.#:2181,10.##.##.##:2181,10.##.##.#:2181/kafka2,10.##.##.##:2181]
>> INFO  zkclient.ZkEventThread run - Terminate ZkClient event thread.
>>
>> 2016-08-24 11:55:13,596 [main] INFO  zookeeper.ZooKeeper close - Session:
>> 0x4558654aacf4263 closed
>>
>> 2016-08-24 11:55:13,596 [main-EventThread] INFO  zookeeper.ClientCnxn run
>> - EventThread shut down
>>
>> 2016-08-24 11:55:13,597 [main] INFO  kafka.AbstractKafkaInputOperator
>> definePartitions - [ONE_TO_ONE]: Initializing partition(s)
>>
>> 2016-08-24 11:55:13,602 [main] INFO  service.AbstractService noteFailure
>> - Service com.datatorrent.stram.StreamingAppMasterService failed in
>> state INITED; cause: java.lang.IllegalArgumentException: there has to be
>> one idempotent storage manager
>>
>> java.lang.IllegalArgumentException: there has to be one idempotent
>> storage manager
>>
>> at com.google.common.base.Preconditions.checkArgument(Precondit
>> ions.java:93)
>>
>> at org.apache.apex.malhar.lib.wal.FSWindowDataManager.partition
>> ed(FSWindowDataManager.java:251)
>>
>> at com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.def
>> inePartitions(AbstractKafkaInputOperator.java:637)
>>
>> at com.datatorrent.stram.plan.physical.PhysicalPlan.initPartiti
>> oning(PhysicalPlan.java:752)
>>
>> at com.datatorrent.stram.plan.physical.PhysicalPlan.addLogicalO
>> perator(PhysicalPlan.java:1676)
>>
>> at com.datatorrent.stram.plan.physical.PhysicalPlan.(Phys
>> icalPlan.java:378)
>>
>> at com.datatorrent.stram.StreamingContainerManager.(Strea
>> mingContainerManager.java:418)
>>
>> at com.datatorrent.stram.StreamingContainerManager.getInstance(
>> StreamingContainerManager.java:3023)
>>
>> at com.datatorrent.stram.StreamingAppMasterService.serviceInit(
>> StreamingAppMasterService.java:551)
>>
>> at org.apache.hadoop.service.AbstractService.init(AbstractServi
>> ce.java:163)
>>
>> at com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMa
>> ster.java:102)
>>
>> 2016-08-24 11:55:13,604 [main] WARN  service.AbstractService stopQuietly
>> - When stopping the service com.datatorrent.stram.StreamingAppMasterService
>> : java.lang.NullPointerException
>>
>> java.lang.NullPoi