HI Pawel,
I see - your cluster state (nodes + routing only, not meta data), seems to
be larger then 16KB when rendered to SMILE, which is quite big - does this
make sense?
Above 16KB an underlying paging system introduced in the ES 1.x branch
kicks in. At that breaks something in Marvel than normally ships very small
documents.
I'll work on a fix. Can you confirm your cluster state (again, without the
metadata) is indeed very large?
Cheers,
Boaz
On Thursday, June 5, 2014 10:56:00 AM UTC+2, Paweł Krzaczkowski wrote:
>
> Hi.
>
> After upgrading Marvel to 1.2.0 (running on Elasticsearch 1.2.1) i'm
> getting errors like
>
> [2014-06-05 10:47:25,346][INFO ][node ] [es-m-3]
> version[1.2.1], pid[68924], build[6c95b75/2014-06-03T15:02:52Z]
> [2014-06-05 10:47:25,347][INFO ][node ] [es-m-3]
> initializing ...
> [2014-06-05 10:47:25,367][INFO ][plugins ] [es-m-3]
> loaded [marvel, analysis-icu], sites [marvel, head, segmentspy, browser,
> paramedic]
> [2014-06-05 10:47:28,455][INFO ][node ] [es-m-3]
> initialized
> [2014-06-05 10:47:28,456][INFO ][node ] [es-m-3]
> starting ...
> [2014-06-05 10:47:28,597][INFO ][transport ] [es-m-3]
> bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
> 192.168.0.212:9300]}
> [2014-06-05 10:47:42,340][INFO ][cluster.service ] [es-m-3]
> new_master [es-m-3][0H3grrJxTJunU1U6FmkIEg][es-m-3][inet[
> 192.168.0.212/192.168.0.212:9300]]{data=false
> <http://192.168.0.212/192.168.0.212:9300%5D%5D%7Bdata=false>,
> master=true}, reason: zen-disco-join (elected_as_master)
> [2014-06-05 10:47:42,350][INFO ][discovery ] [es-m-3]
> freshmind/0H3grrJxTJunU1U6FmkIEg
> [2014-06-05 10:47:42,365][INFO ][http ] [es-m-3]
> bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
> 192.168.0.212:9200]}
> [2014-06-05 10:47:42,368][INFO ][node ] [es-m-3]
> started
> [2014-06-05 10:47:44,098][INFO ][cluster.service ] [es-m-3] added
> {[es-m-1][MHl5Ls-cRXCwc7OC-P0J5w][es-m-1][inet[/192.168.0.210:9300]]{data=false,
>
> machine=44454c4c-5300-1052-8038-b9c04f5a5a31, master=true},}, reason:
> zen-disco-receive(join from
> node[[es-m-1][MHl5Ls-cRXCwc7OC-P0J5w][es-m-1][inet[/192.168.0.210:9300]]{data=false,
>
> machine=44454c4c-5300-1052-8038-b9c04f5a5a31, master=true}])
> [2014-06-05 10:47:44,401][INFO ][gateway ] [es-m-3]
> recovered [28] indices into cluster_state
> [2014-06-05 10:47:48,683][ERROR][marvel.agent ] [es-m-3]
> exporter [es_exporter] has thrown an exception:
> java.lang.IllegalStateException: array not available
> at
> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
> at java.lang.Thread.run(Thread.java:745)
> [2014-06-05 10:47:58,738][ERROR][marvel.agent ] [es-m-3]
> exporter [es_exporter] has thrown an exception:
> java.lang.IllegalStateException: array not available
> at
> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
> at java.lang.Thread.run(Thread.java:745)
> [2014-06-05 10:48:36,572][INFO ][cluster.service ] [es-m-3] added
> {[es-m-2][e5uEqGRhS7uEioNxaYkwTg][es-m-2][inet[/192.168.0.211:9300]]{data=false,
>
> master=true},}, reason: zen-disco-receive(join from
> node[[es-m-2][e5uEqGRhS7uEioNxaYkwTg][es-m-2][inet[/192.168.0.211:9300]]{data=false,
>
> master=true}])
> [2014-06-05 10:48:38,859][ERROR][marvel.agent ] [es-m-3]
> exporter [es_exporter] has thrown an exception:
> java.lang.IllegalStateException: array not available
> at
> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
> at java.lang.Thread.run(Thread.java:745)
> [2014-06-05 10:49:47,283][INFO ][cluster.service ] [es-m-3] added
> {[es-d-1][9T5-BOVCSveOyh5dV9YvDg][es-d-1.localhost][inet[/192.168.0.213:9300]]{machine=44454c4c-5300-1052-8038-b9c04f5a5a31,
>
> master=false},}, reason: zen-disco-receive(join from
> node[[es-d-1][9T5-BOVCSveOyh5dV9YvDg][es-d-1.localhost][inet[/192.168.0.213:9300]]{machine=44454c4c-5300-1052-8038-b9c04f5a5a31,
>
> master=false}])
> [2014-06-05 10:49:47,350][INFO ][cluster.service ] [es-m-3] added
> {[es-d-2][ddoIhezQSjuGYWPmSWltIg][es-d-2][inet[/192.168.0.200:9300]]{machine=44454c4c-5200-1038-8030-c2c04f365931,
>
> master=false},}, reason: zen-disco-receive(join from
> node[[es-d-2][ddoIhezQSjuGYWPmSWltIg][es-d-2][inet[/192.168.0.200:9300]]{machine=44454c4c-5200-1038-8030-c2c04f365931,
>
> master=false}])
> [2014-06-05 10:49:49,055][ERROR][marvel.agent ] [es-m-3]
> exporter [es_exporter] has thrown an exception:
> java.lang.IllegalStateException: array not available
> at
> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
> at java.lang.Thread.run(Thread.java:745)
> [2014-06-05 10:49:59,090][ERROR][marvel.agent ] [es-m-3]
> exporter [es_exporter] has thrown an exception:
> java.lang.IllegalStateException: array not available
> at
> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
> at java.lang.Thread.run(Thread.java:745)
> [2014-06-05 10:50:09,267][ERROR][marvel.agent ] [es-m-3]
> exporter [es_exporter] has thrown an exception:
> java.lang.IllegalStateException: array not available
> at
> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
> at
> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
> at
> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
> at java.lang.Thread.run(Thread.java:745)
> [root@es-m-3 ~]#
>
> Marvel is running on a separate cluster.
> Cluster state is not being transmitted to marvel. Error occurs every time
> cluster state changes (node restart, shard allocation, etc.)
>
> ENV on master node
> [root@es-m-3 ~]# env
>
> MANPATH=/opt/local/man:/usr/share/man:/opt/local/gcc47/man:/opt/local/java/sun6/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man
> HZ=100
> SHELL=/usr/bin/bash
> TERM=xterm
> ES_HEAP_SIZE=512M
> ES_JAVA_OPTS=-d64 -server -Des.processors=1 -Des.node.name=es-m-3
> -Des.node.machine=44454c4c-5200-1038-8030-c2c04f365931
> COLUMNS=238
> PAGER=less
> MAIL=/var/mail/root
>
> PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
> PWD=/root
> JAVA_HOME=/opt/jdk1.7.0_55/
> LINES=74
> SHLVL=1
> HOME=/root
> TERMINFO=/opt/local/share/lib/terminfo
> LOGNAME=root
> FTPMODE=auto
> _=/opt/local/bin/env
>
> [root@es-m-3 ~]# /opt/jdk1.7.0_55/bin/java -d64 -version
> java version "1.7.0_55"
> Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
>
>
> Marvel 1.1.1 worked without any problems.
>
> Pawel
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9dff903e-2597-43a4-9b0c-23a43b23ba04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
