Re: Zookeeper general log or “monitor” command

2018-03-08 Thread Nikhil Bafna 
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.

Yes, that would work.

You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307

Checked the PR. IMO, the actual proposed change there doesn't solve the
intent of the mentioned problem (though it solves mine) and have left a
comment there with my viewpoint.


-
zodvik

On Fri, Mar 9, 2018 at 12:36 AM, Abraham Fine  wrote:

> You may also want to consider taking a look and reviewing the change
> proposed here: https://github.com/apache/zookeeper/pull/307
>
> If I understand correctly, a slow operation log with the threshold for
> logging set to 0 should accomplish what you are trying to accomplish.
>
> Thanks,
> Abe
>
> On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> > Cool, monitoring the transaction makes much sense I think.
> >
> > Interesting to see this tool, I didn't know about it yet. For some reason
> > it replicates most of the logic in FileSnap and FileTxnLog classes, but
> at
> > the same time it uses ZooKeeper as a dependency.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna  wrote:
> >
> > > As you mentioned, enabling trace/debug might impact performance.
> > >
> > >
> > > Here's the approach I'm using now.
> > >
> > >-
> > >
> > >Parse the Zookeeper transaction logs. The best repo I've found to do
> > >this is https://github.com/alenca/zklogtool. It's feature packed.
> The
> > >only downside is that it won't record the read queries.
> > >
> > >
> > > -
> > > zodvik
> > >
> > > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar 
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > Have you checked TRACE logging of client requests? (
> > > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > > html#sc_authOptions
> > > > )
> > > > Not sure how much different is from what you refer as 'debug logs',
> but
> > > it
> > > > might worth a try. It generates log messages like:
> > > >
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd20 type:getChildren2 cxid:0x6
> > > > zxid:0xfffe txntype:unknown reqpath:/
> > > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > > :Esessionid:0x8061f60cfd20 type:getData cxid:0x7
> > > > zxid:0xfffe txntype:unknown reqpath:/andor2
> > > >
> > > > Which looks like quite easy to parse and contains all required
> > > information.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > > On Thu, Mar 1, 2018 at 5:33 AM, Zod  wrote:
> > > >
> > > > > Unfortunately, parsing debug log is no-go since it doesn't print
> the
> > > > > complete command.
> > > > >
> > > > > I've 2 approached in mind as of now
> > > > >
> > > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > > "tailing".
> > > > > 2. Use ngrep on my local to monitor the network traffic to remote
> ZK
> > > > host &
> > > > > port 2181
> > > > >
> > > > > ​Will get back with what works better.​
> > > > >
> > > > > -
> > > > > zodvik
> > > > >
> > > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar 
> > > wrote:
> > > > >
> > > > > > Hi Zod,
> > > > > >
> > > > > > I think there's no such thing in ZooKeeper currently other than
> debug
> > > > > logs.
> > > > > > You might want to attach some processing engine to the debug log
> feed
> > > > and
> > > > > > parse and output data which is important for you into MySQL or
> Redis
> > > > > > database.
> > > > > >
> > > > > > The downside of this approach is debug logging has significant
> > > > > performance
> > > > > > impact on the entire ensemble, but it might be acceptable for you
> > > > > analysis.
> > > > > >
> > > > > > Regards,
> > > > > > Andor
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:
> > > > > >
> > > > > > > I want see all commands issued to my Zookeeper cluster,
> something
> > > > like
> > > > > > > general log in MySQL or "monitor" command in Redis. How can
> this be
> > > > > done?
> > > > > > >
> > > > > > > The intent is to see how Storm uses Zookeeper (for state
> > > management &
> > > > > > > ack-ing) and a good way to would be to run a sample topology
> and
> > > see
> > > > > all
> > > > > > > the commands issued to Zookeeper by Storm.
> > > > > > >
> > > > > > > I tried enabling debug log for Zookeeper, but that is
> insufficient
> > > > has
> > > > > a
> > > > > > > lot of noise. For example, issue a create /node prints
> > > > > > >
> > > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > > -
> > > > > > > zodvik
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: Zookeeper general log or “monitor” command

2018-03-08 Thread Abraham Fine 
You may also want to consider taking a look and reviewing the change proposed 
here: https://github.com/apache/zookeeper/pull/307

If I understand correctly, a slow operation log with the threshold for logging 
set to 0 should accomplish what you are trying to accomplish.

Thanks,
Abe

On Wed, Mar 7, 2018, at 04:30, Andor Molnar wrote:
> Cool, monitoring the transaction makes much sense I think.
> 
> Interesting to see this tool, I didn't know about it yet. For some reason
> it replicates most of the logic in FileSnap and FileTxnLog classes, but at
> the same time it uses ZooKeeper as a dependency.
> 
> Regards,
> Andor
> 
> 
> 
> On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna  wrote:
> 
> > As you mentioned, enabling trace/debug might impact performance.
> >
> >
> > Here's the approach I'm using now.
> >
> >-
> >
> >Parse the Zookeeper transaction logs. The best repo I've found to do
> >this is https://github.com/alenca/zklogtool. It's feature packed. The
> >only downside is that it won't record the read queries.
> >
> >
> > -
> > zodvik
> >
> > On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar  wrote:
> >
> > > Hi Zod,
> > >
> > > Have you checked TRACE logging of client requests? (
> > > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> > html#sc_authOptions
> > > )
> > > Not sure how much different is from what you refer as 'debug logs', but
> > it
> > > might worth a try. It generates log messages like:
> > >
> > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > :Esessionid:0x8061f60cfd20 type:getChildren2 cxid:0x6
> > > zxid:0xfffe txntype:unknown reqpath:/
> > > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > > :Esessionid:0x8061f60cfd20 type:getData cxid:0x7
> > > zxid:0xfffe txntype:unknown reqpath:/andor2
> > >
> > > Which looks like quite easy to parse and contains all required
> > information.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > > On Thu, Mar 1, 2018 at 5:33 AM, Zod  wrote:
> > >
> > > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > > complete command.
> > > >
> > > > I've 2 approached in mind as of now
> > > >
> > > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > > "tailing".
> > > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > > host &
> > > > port 2181
> > > >
> > > > ​Will get back with what works better.​
> > > >
> > > > -
> > > > zodvik
> > > >
> > > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar 
> > wrote:
> > > >
> > > > > Hi Zod,
> > > > >
> > > > > I think there's no such thing in ZooKeeper currently other than debug
> > > > logs.
> > > > > You might want to attach some processing engine to the debug log feed
> > > and
> > > > > parse and output data which is important for you into MySQL or Redis
> > > > > database.
> > > > >
> > > > > The downside of this approach is debug logging has significant
> > > > performance
> > > > > impact on the entire ensemble, but it might be acceptable for you
> > > > analysis.
> > > > >
> > > > > Regards,
> > > > > Andor
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:
> > > > >
> > > > > > I want see all commands issued to my Zookeeper cluster, something
> > > like
> > > > > > general log in MySQL or "monitor" command in Redis. How can this be
> > > > done?
> > > > > >
> > > > > > The intent is to see how Storm uses Zookeeper (for state
> > management &
> > > > > > ack-ing) and a good way to would be to run a sample topology and
> > see
> > > > all
> > > > > > the commands issued to Zookeeper by Storm.
> > > > > >
> > > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > > has
> > > > a
> > > > > > lot of noise. For example, issue a create /node prints
> > > > > >
> > > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > > -
> > > > > > zodvik
> > > > > >
> > > > >
> > > >
> > >
> >

Re: Zookeeper general log or “monitor” command

2018-03-07 Thread Andor Molnar 
Cool, monitoring the transaction makes much sense I think.

Interesting to see this tool, I didn't know about it yet. For some reason
it replicates most of the logic in FileSnap and FileTxnLog classes, but at
the same time it uses ZooKeeper as a dependency.

Regards,
Andor



On Wed, Mar 7, 2018 at 9:33 AM, Nikhil Bafna  wrote:

> As you mentioned, enabling trace/debug might impact performance.
>
>
> Here's the approach I'm using now.
>
>-
>
>Parse the Zookeeper transaction logs. The best repo I've found to do
>this is https://github.com/alenca/zklogtool. It's feature packed. The
>only downside is that it won't record the read queries.
>
>
> -
> zodvik
>
> On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar  wrote:
>
> > Hi Zod,
> >
> > Have you checked TRACE logging of client requests? (
> > http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.
> html#sc_authOptions
> > )
> > Not sure how much different is from what you refer as 'debug logs', but
> it
> > might worth a try. It generates log messages like:
> >
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd20 type:getChildren2 cxid:0x6
> > zxid:0xfffe txntype:unknown reqpath:/
> > TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> > :Esessionid:0x8061f60cfd20 type:getData cxid:0x7
> > zxid:0xfffe txntype:unknown reqpath:/andor2
> >
> > Which looks like quite easy to parse and contains all required
> information.
> >
> > Regards,
> > Andor
> >
> >
> > On Thu, Mar 1, 2018 at 5:33 AM, Zod  wrote:
> >
> > > Unfortunately, parsing debug log is no-go since it doesn't print the
> > > complete command.
> > >
> > > I've 2 approached in mind as of now
> > >
> > > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > > "tailing".
> > > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> > host &
> > > port 2181
> > >
> > > ​Will get back with what works better.​
> > >
> > > -
> > > zodvik
> > >
> > > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar 
> wrote:
> > >
> > > > Hi Zod,
> > > >
> > > > I think there's no such thing in ZooKeeper currently other than debug
> > > logs.
> > > > You might want to attach some processing engine to the debug log feed
> > and
> > > > parse and output data which is important for you into MySQL or Redis
> > > > database.
> > > >
> > > > The downside of this approach is debug logging has significant
> > > performance
> > > > impact on the entire ensemble, but it might be acceptable for you
> > > analysis.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:
> > > >
> > > > > I want see all commands issued to my Zookeeper cluster, something
> > like
> > > > > general log in MySQL or "monitor" command in Redis. How can this be
> > > done?
> > > > >
> > > > > The intent is to see how Storm uses Zookeeper (for state
> management &
> > > > > ack-ing) and a good way to would be to run a sample topology and
> see
> > > all
> > > > > the commands issued to Zookeeper by Storm.
> > > > >
> > > > > I tried enabling debug log for Zookeeper, but that is insufficient
> > has
> > > a
> > > > > lot of noise. For example, issue a create /node prints
> > > > >
> > > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > > -
> > > > > zodvik
> > > > >
> > > >
> > >
> >
>

Re: Zookeeper general log or “monitor” command

2018-03-07 Thread Nikhil Bafna 
As you mentioned, enabling trace/debug might impact performance.


Here's the approach I'm using now.

   -

   Parse the Zookeeper transaction logs. The best repo I've found to do
   this is https://github.com/alenca/zklogtool. It's feature packed. The
   only downside is that it won't record the read queries.


-
zodvik

On Mon, Mar 5, 2018 at 5:42 PM, Andor Molnar  wrote:

> Hi Zod,
>
> Have you checked TRACE logging of client requests? (
> http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_authOptions
> )
> Not sure how much different is from what you refer as 'debug logs', but it
> might worth a try. It generates log messages like:
>
> TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> :Esessionid:0x8061f60cfd20 type:getChildren2 cxid:0x6
> zxid:0xfffe txntype:unknown reqpath:/
> TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
> :Esessionid:0x8061f60cfd20 type:getData cxid:0x7
> zxid:0xfffe txntype:unknown reqpath:/andor2
>
> Which looks like quite easy to parse and contains all required information.
>
> Regards,
> Andor
>
>
> On Thu, Mar 1, 2018 at 5:33 AM, Zod  wrote:
>
> > Unfortunately, parsing debug log is no-go since it doesn't print the
> > complete command.
> >
> > I've 2 approached in mind as of now
> >
> > 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> > "tailing".
> > 2. Use ngrep on my local to monitor the network traffic to remote ZK
> host &
> > port 2181
> >
> > ​Will get back with what works better.​
> >
> > -
> > zodvik
> >
> > On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar  wrote:
> >
> > > Hi Zod,
> > >
> > > I think there's no such thing in ZooKeeper currently other than debug
> > logs.
> > > You might want to attach some processing engine to the debug log feed
> and
> > > parse and output data which is important for you into MySQL or Redis
> > > database.
> > >
> > > The downside of this approach is debug logging has significant
> > performance
> > > impact on the entire ensemble, but it might be acceptable for you
> > analysis.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:
> > >
> > > > I want see all commands issued to my Zookeeper cluster, something
> like
> > > > general log in MySQL or "monitor" command in Redis. How can this be
> > done?
> > > >
> > > > The intent is to see how Storm uses Zookeeper (for state management &
> > > > ack-ing) and a good way to would be to run a sample topology and see
> > all
> > > > the commands issued to Zookeeper by Storm.
> > > >
> > > > I tried enabling debug log for Zookeeper, but that is insufficient
> has
> > a
> > > > lot of noise. For example, issue a create /node prints
> > > >
> > > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > > -
> > > > zodvik
> > > >
> > >
> >
>

Re: Zookeeper general log or “monitor” command

2018-03-05 Thread Andor Molnar 
Hi Zod,

Have you checked TRACE logging of client requests? (
http://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#sc_authOptions)
Not sure how much different is from what you refer as 'debug logs', but it
might worth a try. It generates log messages like:

TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
:Esessionid:0x8061f60cfd20 type:getChildren2 cxid:0x6
zxid:0xfffe txntype:unknown reqpath:/
TRACE [CommitProcWorkThread-1:ZooTrace@88][] -
:Esessionid:0x8061f60cfd20 type:getData cxid:0x7
zxid:0xfffe txntype:unknown reqpath:/andor2

Which looks like quite easy to parse and contains all required information.

Regards,
Andor


On Thu, Mar 1, 2018 at 5:33 AM, Zod  wrote:

> Unfortunately, parsing debug log is no-go since it doesn't print the
> complete command.
>
> I've 2 approached in mind as of now
>
> 1. Modify of org.apache.zookeeper.server.LogFormatter to support
> "tailing".
> 2. Use ngrep on my local to monitor the network traffic to remote ZK host &
> port 2181
>
> ​Will get back with what works better.​
>
> -
> zodvik
>
> On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar  wrote:
>
> > Hi Zod,
> >
> > I think there's no such thing in ZooKeeper currently other than debug
> logs.
> > You might want to attach some processing engine to the debug log feed and
> > parse and output data which is important for you into MySQL or Redis
> > database.
> >
> > The downside of this approach is debug logging has significant
> performance
> > impact on the entire ensemble, but it might be acceptable for you
> analysis.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:
> >
> > > I want see all commands issued to my Zookeeper cluster, something like
> > > general log in MySQL or "monitor" command in Redis. How can this be
> done?
> > >
> > > The intent is to see how Storm uses Zookeeper (for state management &
> > > ack-ing) and a good way to would be to run a sample topology and see
> all
> > > the commands issued to Zookeeper by Storm.
> > >
> > > I tried enabling debug log for Zookeeper, but that is insufficient has
> a
> > > lot of noise. For example, issue a create /node prints
> > >
> > > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > > -
> > > zodvik
> > >
> >
>

Re: Zookeeper general log or “monitor” command

2018-02-28 Thread Zod 
Unfortunately, parsing debug log is no-go since it doesn't print the
complete command.

I've 2 approached in mind as of now

1. Modify of org.apache.zookeeper.server.LogFormatter to support "tailing".
2. Use ngrep on my local to monitor the network traffic to remote ZK host &
port 2181

​Will get back with what works better.​

-
zodvik

On Thu, Mar 1, 2018 at 1:22 AM, Andor Molnar  wrote:

> Hi Zod,
>
> I think there's no such thing in ZooKeeper currently other than debug logs.
> You might want to attach some processing engine to the debug log feed and
> parse and output data which is important for you into MySQL or Redis
> database.
>
> The downside of this approach is debug logging has significant performance
> impact on the entire ensemble, but it might be acceptable for you analysis.
>
> Regards,
> Andor
>
>
>
> On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:
>
> > I want see all commands issued to my Zookeeper cluster, something like
> > general log in MySQL or "monitor" command in Redis. How can this be done?
> >
> > The intent is to see how Storm uses Zookeeper (for state management &
> > ack-ing) and a good way to would be to run a sample topology and see all
> > the commands issued to Zookeeper by Storm.
> >
> > I tried enabling debug log for Zookeeper, but that is insufficient has a
> > lot of noise. For example, issue a create /node prints
> >
> > 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> > -
> > zodvik
> >
>

Re: Zookeeper general log or “monitor” command

2018-02-28 Thread Andor Molnar 
Hi Zod,

I think there's no such thing in ZooKeeper currently other than debug logs.
You might want to attach some processing engine to the debug log feed and
parse and output data which is important for you into MySQL or Redis
database.

The downside of this approach is debug logging has significant performance
impact on the entire ensemble, but it might be acceptable for you analysis.

Regards,
Andor



On Tue, Feb 27, 2018 at 11:26 PM, Zod  wrote:

> I want see all commands issued to my Zookeeper cluster, something like
> general log in MySQL or "monitor" command in Redis. How can this be done?
>
> The intent is to see how Storm uses Zookeeper (for state management &
> ack-ing) and a good way to would be to run a sample topology and see all
> the commands issued to Zookeeper by Storm.
>
> I tried enabling debug log for Zookeeper, but that is insufficient has a
> lot of noise. For example, issue a create /node prints
>
> 2018-02-27 18:05:34 ZooKeeperMain [DEBUG] Processing create
> -
> zodvik
>