Zookeeper log filling up disk

[Christie, Marcus Aaron]

Dev,

This message is just to document what happened today in the SGRC dev 
environment.  But if you have any insight on what caused this to happen, please 
share.

TLDR: Zookeeper log fills disk, Logstash is spamming ZK with requests, not sure 
what caused it but have reconfigured ZK logging to rotate files and not fill up 
disk.

So today in our dev environment the Zookeeper server’s log file filled up the 
disk. The size was about 190GB.  It wasn’t being rotated so it has possibly 
been growing for a while. On the other hand I saw in the log file that there 
were several messages a second, that looked like this:

2017-05-26 11:42:35,070 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket 
connection for client /127.0.0.1:46462 (no session established for client)
2017-05-26 11:42:35,070 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted 
socket connection from /127.0.0.1:46464
2017-05-26 11:42:35,071 [myid:] - WARN  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception 
causing close of session 0x0 due to java.io.IOException: Unreasonable length = 
1684371039
2017-05-26 11:42:35,071 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket 
connection for client /127.0.0.1:46464 (no session established for client)
2017-05-26 11:42:35,071 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted 
socket connection from /127.0.0.1:46466
2017-05-26 11:42:35,071 [myid:] - WARN  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception 
causing close of session 0x0 due to java.io.IOException: Unreasonable length = 
1684371039
2017-05-26 11:42:35,071 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket 
connection for client /127.0.0.1:46466 (no session established for client)
2017-05-26 11:42:35,072 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted 
socket connection from /127.0.0.1:46468
2017-05-26 11:42:35,072 [myid:] - WARN  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception 
causing close of session 0x0 due to java.io.IOException: Unreasonable length = 
1684371039
2017-05-26 11:42:35,072 [myid:] - INFO  
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket 
connection for client /127.0.0.1:46468 (no session established for client)
…


I shut down api-orch and gfac and Kafka (which is just pushing log messages to 
Logstash).  I then deleted the ./data directory in the Zookeeper installation 
and restarted it. Still getting those error messages above.

Eventually I found that Logstash was apparently trying to make Zookeeper 
connections. So I shut it down as well.  Once I shut down Logstash the error 
messages stop in the Zookeeper log.

It’s hard to say whether the problem was
1. Logstash inappropriately sending too large messages to Zookeeper
2. Or, Zookeeper’s log file fills up disk space causing Zookeeper’s database to 
become corrupted. Once the disk fills up, all sorts of weird behavior can start 
to manifest.

I’ve reconfigured the Zookeeper logging to log to a rotated file, that rotates 
at 10MB and keeps a max of 10 rotated log files.  That should prevent running 
out of disk space. Used this [1] as a resource.  I created issue AIRAVATA-2411 
[2] to incorporate this into our Ansible scripts.


Thanks,

Marcus


[1] 
https://community.hortonworks.com/content/supportkb/49091/zookeeperout-file-keeps-growing-until-restarted.html
[2] https://issues.apache.org/jira/browse/AIRAVATA-2411

Re: Spark context diagrams

[Ajinkya Dhamnaskar]

Hey Apoorv,

you can use draw.io.
Also, can you please add brief description explaining the diagram?

On Fri, May 26, 2017 at 11:38 AM, Apoorv Palkar 
wrote:

> ok will do.
>
>
>
> -Original Message-
> From: Pamidighantam, Sudhakar V 
> To: dev 
> Sent: Fri, May 26, 2017 11:36 am
> Subject: Re: Spark context diagrams
>
> Apoorv:
>
> Can you create these diagrams with creately or some software and annotate
> them better.
>
> It is a bit difficult for old eyes to read them..
>
> Thanks,
> Sudhakar.
>
> On May 26, 2017, at 11:25 AM, Apoorv Palkar  wrote:
>
> Hey I've been working on teh spark details and posted 2 diagrams on google
> docs in link below. Hopefully i can with the grove and have it be working
> with/as the potential orchestrator.
>
>
>
> https://docs.google.com/document/d/1kjIBC0ianDVJlSuPs8FanCTO8ili1
> VETA5xKeFqo1gY/edit?usp=sharing
>
>
>
>


-- 
Thanks and regards,

Ajinkya Dhamnaskar
Student ID : 0003469679
Masters (CS)
+1 (812) 369- 5416

Re: Spark context diagrams

[Apoorv Palkar]

ok will do.



-Original Message-
From: Pamidighantam, Sudhakar V 
To: dev 
Sent: Fri, May 26, 2017 11:36 am
Subject: Re: Spark context diagrams


Apoorv:


Can you create these diagrams with creately or some software and annotate them 
better. 


It is a bit difficult for old eyes to read them..


Thanks,
Sudhakar.



On May 26, 2017, at 11:25 AM, Apoorv Palkar  wrote:


Hey I've been working on teh spark details and posted 2 diagrams on google docs 
in link below. Hopefully i can with the grove and have it be working with/as 
the potential orchestrator.






https://docs.google.com/document/d/1kjIBC0ianDVJlSuPs8FanCTO8ili1VETA5xKeFqo1gY/edit?usp=sharing

Re: Spark context diagrams

[Pamidighantam, Sudhakar V]

Apoorv:

Can you create these diagrams with creately or some software and annotate them 
better.

It is a bit difficult for old eyes to read them..

Thanks,
Sudhakar.

On May 26, 2017, at 11:25 AM, Apoorv Palkar 
> wrote:

Hey I've been working on teh spark details and posted 2 diagrams on google docs 
in link below. Hopefully i can with the grove and have it be working with/as 
the potential orchestrator.



https://docs.google.com/document/d/1kjIBC0ianDVJlSuPs8FanCTO8ili1VETA5xKeFqo1gY/edit?usp=sharing

Spark context diagrams

[Apoorv Palkar]

Hey I've been working on teh spark details and posted 2 diagrams on google docs 
in link below. Hopefully i can with the grove and have it be working with/as 
the potential orchestrator.






https://docs.google.com/document/d/1kjIBC0ianDVJlSuPs8FanCTO8ili1VETA5xKeFqo1gY/edit?usp=sharing

Re: [VOTE] enables writes to Airavata GitHub mirror

[Christie, Marcus Aaron]

On May 26, 2017, at 9:12 AM, Suresh Marru 
> wrote:

I think that may not be desirable and ASF might not want to fully depend on a 
third party commercial service as the only write only repo. We can watch the 
commits and slowly educate all committers to only use GitHub as the write 
(except when it is down).


Sounds good. Thanks for the explanation Suresh.

Marcus

Re: [VOTE] enables writes to Airavata GitHub mirror

[Suresh Marru]

Any further thoughts from any one for or against? 

Or any objections to move forward to make a INFRA request to make this change? 

Hi Marcus,

> On May 23, 2017, at 2:27 PM, Christie, Marcus Aaron  wrote:
> 
> 
>> On May 23, 2017, at 10:44 AM, Suresh Marru > > wrote:
>> 
>> Yes and No. From a git (rather git pubsub) perspective yes both the repos 
>> allow writes and they synchronize with post commit hooks. But we should 
>> avoid writing to both since there could be code conflicts and the commits 
>> (reactionary to hooks) could fail which will need manual intervention to 
>> clear the clutter. So its better to pick one repo as the primary write and 
>> the other one only when primary one is unavailable. 
>> 
> 
> Can we turn off writes to the apache repo so that it’s not possible for there 
> to be conflicts?

I think that may not be desirable and ASF might not want to fully depend on a 
third party commercial service as the only write only repo. We can watch the 
commits and slowly educate all committers to only use GitHub as the write 
(except when it is down).

Suresh

> 
> Thanks,
> 
> Marcus