Re: Cassandra open port native_transport_port after a random time

2017-11-11 Thread Octavian Rinciog 
Hello

Thank you for your responses.

The question is still open: how can we debug this kind of problems and how
can we find what is cassandra doing/loading?

I don't mind to wait for loading data structures, but it will be great to
know what cassandra is loading and for how much time.

Thank you.




Pe 11 nov. 2017 01:54, "Jeff Jirsa"  a scris:

> I wouldn't expect a 128MB counter cache to take an hour to load. You've
> got some GC pressure (with quite a few ~500ms pauses during startup), but
> it still seems odd.
>
>
> On Fri, Nov 10, 2017 at 2:26 PM, Octavian Rinciog <
> octavian.rinc...@gmail.com> wrote:
>
>> Hello!
>>
>> As I said, the node opens the connections to the other nodes from the
>> cluster, but not with the clients on the 9042 port.
>>
>> But inspecting the logs, I have found the problem (INFO
>> [pool-5-thread-1] 2017-11-10 18:28:13,019 AutoSavingCache.java:173 -
>> Completed loading (5315293 ms; 893107 keys) CounterCache cache.) It
>> seems that it took only 1.5 hours to load counter cache.
>>
>>
>> Anyway I attached a section of logs.
>>
>> Thank you very much for your response.
>>
>>
>>
>>
>> 2017-11-10 22:21 GMT+02:00 kurt greaves :
>> > Can you upload some logs from startup? Obviously getting stuck starting
>> but
>> > it'd be nice to know where it got up to.
>> >
>> > On 11 Nov. 2017 04:49, "Octavian Rinciog" 
>> > wrote:
>> >>
>> >> Hello!
>> >>
>> >> I'm running a cluster of 10 Cassandra 3.10 and I saw a very strange
>> >> behavior: after restart, a node won't open immediately
>> >> native_transport_port (9042).
>> >>
>> >> After one node restart, the flow is :
>> >>
>> >> - node finishes to read all commitlog,
>> >> - update all its data,
>> >> - it's visible for other nodes in the cluster,
>> >> - wait for random time (from 1 minute to hours) to open 9042 port
>> >>
>> >> My logs are in DEBUG mode, and nothing is written about opening this
>> >> port. What is happening and how can I debug this problem?
>> >>
>> >> Output for several nodetool commands are:
>> >>
>> >> - nodetool enablebinary: does not return at all (it just hangs)
>> >> - nodetool compactionstats: 0 pending tasks
>> >> - nodetool netstats: Mode: STARTING. Not sending any streams.
>> >> - nodetool info: Gossip active : true Thrift active : false Native
>> >> Transport active: false
>> >>
>> >> Thank you.
>> >>
>> >>
>> >>
>> >> --
>> >> Octavian Rinciog
>> >>
>> >> -
>> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> >> For additional commands, e-mail: user-h...@cassandra.apache.org
>> >>
>> >
>>
>>
>>
>> --
>> Octavian Rinciog
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>
>

Re: Cassandra open port native_transport_port after a random time

2017-11-10 Thread Jeff Jirsa 
I wouldn't expect a 128MB counter cache to take an hour to load. You've got
some GC pressure (with quite a few ~500ms pauses during startup), but it
still seems odd.


On Fri, Nov 10, 2017 at 2:26 PM, Octavian Rinciog <
octavian.rinc...@gmail.com> wrote:

> Hello!
>
> As I said, the node opens the connections to the other nodes from the
> cluster, but not with the clients on the 9042 port.
>
> But inspecting the logs, I have found the problem (INFO
> [pool-5-thread-1] 2017-11-10 18:28:13,019 AutoSavingCache.java:173 -
> Completed loading (5315293 ms; 893107 keys) CounterCache cache.) It
> seems that it took only 1.5 hours to load counter cache.
>
>
> Anyway I attached a section of logs.
>
> Thank you very much for your response.
>
>
>
>
> 2017-11-10 22:21 GMT+02:00 kurt greaves :
> > Can you upload some logs from startup? Obviously getting stuck starting
> but
> > it'd be nice to know where it got up to.
> >
> > On 11 Nov. 2017 04:49, "Octavian Rinciog" 
> > wrote:
> >>
> >> Hello!
> >>
> >> I'm running a cluster of 10 Cassandra 3.10 and I saw a very strange
> >> behavior: after restart, a node won't open immediately
> >> native_transport_port (9042).
> >>
> >> After one node restart, the flow is :
> >>
> >> - node finishes to read all commitlog,
> >> - update all its data,
> >> - it's visible for other nodes in the cluster,
> >> - wait for random time (from 1 minute to hours) to open 9042 port
> >>
> >> My logs are in DEBUG mode, and nothing is written about opening this
> >> port. What is happening and how can I debug this problem?
> >>
> >> Output for several nodetool commands are:
> >>
> >> - nodetool enablebinary: does not return at all (it just hangs)
> >> - nodetool compactionstats: 0 pending tasks
> >> - nodetool netstats: Mode: STARTING. Not sending any streams.
> >> - nodetool info: Gossip active : true Thrift active : false Native
> >> Transport active: false
> >>
> >> Thank you.
> >>
> >>
> >>
> >> --
> >> Octavian Rinciog
> >>
> >> -
> >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: user-h...@cassandra.apache.org
> >>
> >
>
>
>
> --
> Octavian Rinciog
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Re: Cassandra open port native_transport_port after a random time

2017-11-10 Thread kurt greaves 
Can you upload some logs from startup? Obviously getting stuck starting but
it'd be nice to know where it got up to.

On 11 Nov. 2017 04:49, "Octavian Rinciog" 
wrote:

> Hello!
>
> I'm running a cluster of 10 Cassandra 3.10 and I saw a very strange
> behavior: after restart, a node won't open immediately
> native_transport_port (9042).
>
> After one node restart, the flow is :
>
> - node finishes to read all commitlog,
> - update all its data,
> - it's visible for other nodes in the cluster,
> - wait for random time (from 1 minute to hours) to open 9042 port
>
> My logs are in DEBUG mode, and nothing is written about opening this
> port. What is happening and how can I debug this problem?
>
> Output for several nodetool commands are:
>
> - nodetool enablebinary: does not return at all (it just hangs)
> - nodetool compactionstats: 0 pending tasks
> - nodetool netstats: Mode: STARTING. Not sending any streams.
> - nodetool info: Gossip active : true Thrift active : false Native
> Transport active: false
>
> Thank you.
>
>
>
> --
> Octavian Rinciog
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Cassandra open port native_transport_port after a random time

2017-11-10 Thread Octavian Rinciog 
Hello!

I'm running a cluster of 10 Cassandra 3.10 and I saw a very strange
behavior: after restart, a node won't open immediately
native_transport_port (9042).

After one node restart, the flow is :

- node finishes to read all commitlog,
- update all its data,
- it's visible for other nodes in the cluster,
- wait for random time (from 1 minute to hours) to open 9042 port

My logs are in DEBUG mode, and nothing is written about opening this
port. What is happening and how can I debug this problem?

Output for several nodetool commands are:

- nodetool enablebinary: does not return at all (it just hangs)
- nodetool compactionstats: 0 pending tasks
- nodetool netstats: Mode: STARTING. Not sending any streams.
- nodetool info: Gossip active : true Thrift active : false Native
Transport active: false

Thank you.



-- 
Octavian Rinciog

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org