Hi Frank,
I'm glad I could help in tracking down the problem.
Thanks for the quick responses!
regards,
Bart
On Monday, June 27, 2016 at 1:48:19 PM UTC+2, Frank Celler wrote:
>
> Hello Bart,
>
> thanks a lot for the details. They helped us to track down the issue. We
> will try to create a fix asap.
>
> Best
> Frank
>
> Am Montag, 27. Juni 2016 09:32:04 UTC+2 schrieb Bart DS:
>>
>> Hi Kaveh,
>>
>>
>> On Monday, June 27, 2016 at 9:07:40 AM UTC+2, Kaveh Vahedipour wrote:
>>>
>>> Hello Bart,
>>>
>>> Could you please provide us with some details like OS and docker
>>> versions your are running. The startup process like rough timings and such.
>>> In short I'd be extremely grateful with any kind of details you could
>>> provide us with really. I just went through the process again myself on OSX
>>> 10.11 and Ubuntu 16.04 and cannot create any unstable, let alone crashing,
>>> startups.
>>>
>>>
>> As mentioned in my initial post, I'm testing on a Linux Mint 17.3 x64
>> system with 8GB RAM and 256GB SSD
>> In the end I want to get it running inside docker indeed, but for now I'm
>> testing it natively because the referred to documentation is for running it
>> natively and it should also be easier than within docker I guess (cfr port
>> mappings and such)
>>
>> The startup process is exactly as explained in the refered documentation:
>> Start agent1 and wait until it's ready:
>>
>> arangod --server.endpoint tcp://127.0.0.1:5001 --server.authentication
>> false --agency.id 0 --agency.size 3 agency1
>>
>> Start agent2 and wait until it's ready:
>>
>> arangod --server.endpoint tcp://127.0.0.1:5002 --server.authentication
>> false --agency.id 1 --agency.size 3 agency2
>>
>> Start agent3 and wait until it's ready:
>>
>> arangod --server.endpoint tcp://127.0.0.1:5003 --server.authentication
>> false --agency.id 2 --agency.size 3 --agency.endpoint tcp://
>> 127.0.0.1:5001 --agency.endpoint tcp://127.0.0.1:5002 --agency.endpoint
>> tcp://127.0.0.1:5003 --agency.notify true agency3
>>
>> At this point I get following output for agent3:
>>
>> 2016-06-26T15:33:19Z [30429] INFO using SSL options:
>> SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
>> 2016-06-26T15:33:19Z [30429] INFO file-descriptors (nofiles) hard limit
>> is 4096, soft limit is 1024
>> 2016-06-26T15:33:19Z [30429] INFO ArangoDB 3.0.0 [linux] 64bit, using
>> VPack 0.1.30, ICU 54.1, V8 5.0.71.39, OpenSSL 1.0.2g-fips 1 Mar 2016
>> 2016-06-26T15:33:19Z [30429] INFO loaded database '_system' from
>> 'agency3/databases/database-1'
>> 2016-06-26T15:33:19Z [30429] INFO the server has 8 (hyper) cores, using 2
>> scheduler thread(s), 8 dispatcher thread(s)
>> 2016-06-26T15:33:19Z [30429] INFO running WAL recovery (1 logfiles)
>> 2016-06-26T15:33:19Z [30429] INFO replaying WAL logfile
>> 'agency3/journals/logfile-300418.db' (1 of 1)
>> 2016-06-26T15:33:19Z [30429] INFO WAL recovery finished successfully
>> 2016-06-26T15:33:19Z [30429] INFO JavaScript using startup
>> '/usr/share/arangodb3/js', application '/var/lib/arangodb3-apps'
>> 2016-06-26T15:33:24Z [30429] INFO In database '_system': Database is
>> up-to-date (30000/standalone/existing)
>> 2016-06-26T15:33:24Z [30429] INFO using endpoint 'http+tcp://
>> 127.0.0.1:5003' for non-encrypted requests
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO Authentication is turned off
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO ArangoDB (version 3.0.0 [linux]) is
>> ready for business. Have fun!
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:24Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:25Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] WARNING {agency} Unable to find matching
>> entry to previous entry (index,term) = (0,0)
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:26Z [30429] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:26Z [30429] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>>
>> And the agents start logging as well:
>>
>> Agent1:
>>
>> bart@laptop ~ $ arangod --server.endpoint tcp://127.0.0.1:5001
>> --server.authentication false --agency.id 0 --agency.size 3 agency1
>> 2016-06-26T15:33:38Z [30589] INFO using SSL options:
>> SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
>> 2016-06-26T15:33:38Z [30589] INFO file-descriptors (nofiles) hard limit
>> is 4096, soft limit is 1024
>> 2016-06-26T15:33:38Z [30589] INFO ArangoDB 3.0.0 [linux] 64bit, using
>> VPack 0.1.30, ICU 54.1, V8 5.0.71.39, OpenSSL 1.0.2g-fips 1 Mar 2016
>> 2016-06-26T15:33:38Z [30589] INFO loaded database '_system' from
>> 'agency1/databases/database-1'
>> 2016-06-26T15:33:38Z [30589] INFO the server has 8 (hyper) cores, using 2
>> scheduler thread(s), 8 dispatcher thread(s)
>> 2016-06-26T15:33:38Z [30589] INFO running WAL recovery (1 logfiles)
>> 2016-06-26T15:33:38Z [30589] INFO replaying WAL logfile
>> 'agency1/journals/logfile-2.db' (1 of 1)
>> 2016-06-26T15:33:38Z [30589] INFO WAL recovery finished successfully
>> 2016-06-26T15:33:38Z [30589] INFO JavaScript using startup
>> '/usr/share/arangodb3/js', application '/var/lib/arangodb3-apps'
>> 2016-06-26T15:33:43Z [30589] INFO In database '_system': Found 14 defined
>> task(s), 1 task(s) to run
>> 2016-06-26T15:33:43Z [30589] INFO In database '_system': state
>> standalone/existing, tasks updateUserModels
>> 2016-06-26T15:33:43Z [30589] INFO In database '_system': existing cleanup
>> successfully finished
>> 2016-06-26T15:33:44Z [30589] INFO using endpoint 'http+tcp://
>> 127.0.0.1:5001' for non-encrypted requests
>> 2016-06-26T15:33:44Z [30589] INFO Authentication is turned off
>> 2016-06-26T15:33:44Z [30589] INFO ArangoDB (version 3.0.0 [linux]) is
>> ready for business. Have fun!
>> 2016-06-26T15:33:45Z [30589] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:49Z [30589] WARNING {agency} I have a higher term than
>> RPC caller.
>> bart@laptop ~ $
>>
>> Agent2:
>>
>> bart@laptop ~ $ arangod --server.endpoint tcp://127.0.0.1:5002
>> --server.authentication false --agency.id 1 --agency.size 3 agency2
>> 2016-06-26T15:30:03Z [29228] INFO using SSL options:
>> SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
>> 2016-06-26T15:30:03Z [29228] INFO file-descriptors (nofiles) hard limit
>> is 4096, soft limit is 1024
>> 2016-06-26T15:30:03Z [29228] INFO created database directory 'agency2'.
>> 2016-06-26T15:30:03Z [29228] INFO WAL directory 'agency2/journals' does
>> not exist. creating it...
>> 2016-06-26T15:30:03Z [29228] INFO ArangoDB 3.0.0 [linux] 64bit, using
>> VPack 0.1.30, ICU 54.1, V8 5.0.71.39, OpenSSL 1.0.2g-fips 1 Mar 2016
>> 2016-06-26T15:30:03Z [29228] INFO loaded database '_system' from
>> 'agency2/databases/database-1'
>> 2016-06-26T15:30:03Z [29228] INFO the server has 8 (hyper) cores, using 2
>> scheduler thread(s), 8 dispatcher thread(s)
>> 2016-06-26T15:30:04Z [29228] INFO JavaScript using startup
>> '/usr/share/arangodb3/js', application '/var/lib/arangodb3-apps'
>> 2016-06-26T15:30:06Z [29228] INFO In database '_system': No version
>> information file found in database directory.
>> 2016-06-26T15:30:06Z [29228] INFO In database '_system': Found 14 defined
>> task(s), 12 task(s) to run
>> 2016-06-26T15:30:06Z [29228] INFO In database '_system': state
>> standalone/init, tasks setupGraphs, setupUsers, createUsersIndex,
>> addDefaultUserSystem, createModules, createRouting, insertRedirectionsAll,
>> setupAqlFunctions, createStatistics, createFrontend, setupQueues, setupJobs
>> 2016-06-26T15:30:06Z [29228] INFO In database '_system': init
>> successfully finished
>> 2016-06-26T15:30:06Z [29228] INFO using endpoint 'http+tcp://
>> 127.0.0.1:5002' for non-encrypted requests
>> 2016-06-26T15:30:06Z [29228] INFO Authentication is turned off
>> 2016-06-26T15:30:07Z [29228] INFO ArangoDB (version 3.0.0 [linux]) is
>> ready for business. Have fun!
>> 2016-06-26T15:33:01Z [29228] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:07Z [29228] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:07Z [29228] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:07Z [29228] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:07Z [29228] WARNING {agency} I have a higher term than
>> RPC caller.
>> 2016-06-26T15:33:07Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:08Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:08Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:08Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:09Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:09Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:10Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:10Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:11Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:11Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:12Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:12Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:12Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:12Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:13Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:13Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:14Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:14Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:15Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5001'
>> 2016-06-26T15:33:15Z [29228] INFO cannot create connection to server ''
>> at endpoint 'tcp://127.0.0.1:5003'
>> 2016-06-26T15:33:17Z [29228] WARNING {agency} Unable to find matching
>> entry to previous entry (index,term) = (6,1171)
>> 2016-06-26T15:33:17Z [29228] WARNING {agency} Unable to find matching
>> entry to previous entry (index,term) = (6,1171)
>> [...]
>>
>>
>> As you can see, this time it's agent1 that exits without any error
>> messages.
>> As mentioned in my last post, it's probably not really crashing but
>> rather simply exiting (I can't find any core dumps or other information
>> regarding a possible crash)
>>
>> If I then start Agent1 again, it can be that Agent2 or Agent3 exits or it
>> can be that everything stays up as it should be.
>>
>> In the latter case, if I start the first DBServer, it hangs like this:
>>
>> bart@laptop ~ $ arangod --server.authentication=false --server.endpoint
>> tcp://0.0.0.0:8529 --cluster.my-address tcp://127.0.0.1:8529
>> --cluster.my-local-info db1 --cluster.my-role PRIMARY
>> --cluster.agency-endpoint tcp://127.0.0.1:5001 primary1
>> 2016-06-26T15:37:06Z [31640] INFO using SSL options:
>> SSL_OP_CIPHER_SERVER_PREFERENCE, SSL_OP_TLS_ROLLBACK_BUG
>>
>> I think that's all info I can give you for now.
>>
>> regards,
>>
>> Bart
>>
>>
>>
>>
>>> KInd regards,
>>> Kaveh
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"ArangoDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
