2019-03-23 11:00:01 UTC - Shivji Kumar Jha: I would love to join here and I am
sure many others here will relish this opportunity!
----
2019-03-23 17:50:53 UTC - Ben k: Hi guys . I have deployed pulsar on Aws
however I cant seem to connect to service URL when testing it. I followed
documentation , only thing I changed was instance type for large instances as
they aren't available in my region (us-west-2) . Wondering if anyone can help
or point me to somewhere if this is the wrong place . I can seem to connect to
the broker but closes broker with :
ERROR Checking/getting partition metadata when subscribing - 5.
----
2019-03-23 20:40:58 UTC - Ali Ahmed: @Ben k what’s your cluster specification ?
----
2019-03-23 20:50:21 UTC - Ben k: Hi @Ali Ahmed currently following this
documentation.
<https://pulsar.apache.org/docs/latest/deployment/aws-cluster/>
Orig instance types :
X3 zookeeper t2.small
X3 bookie i3.xlarge
X2 broker. C5.2xlarge
X1 proxy. C5.2xlarge
My config :
X3 zookeeper t2.small
X3 bookie i2.xlarge
X2 broker. C4.xlarge
X1 proxy. C4.xlarge
Is this what you need ?
----
2019-03-23 20:51:07 UTC - Ali Ahmed: you have only one bookie ?
----
2019-03-23 20:53:36 UTC - Ben k: 3 bookies
----
2019-03-23 20:55:42 UTC - Ali Ahmed: ok you will check the logs of the these
services , see if there any errors in brokers or bookies
----
2019-03-23 20:57:35 UTC - Ben k: Where do I find these logs ?
----
2019-03-23 21:00:08 UTC - Ali Ahmed: I believe rpm packages are being used so
it should under /var/log/
----
2019-03-23 21:21:02 UTC - Ben k: Nothing that seems alarming from what I can
see on all the nodes .
----
2019-03-23 21:25:26 UTC - Ali Ahmed: what about the logs in the proxy ?
----
2019-03-23 21:30:47 UTC - Ben k: I get native memory allocation failed to map
bytes . OpenJDK 64biy Server VM warning so it fails to start
pulsar.proxy.service
----
2019-03-23 21:31:04 UTC - Ben k: Does this mean I need a bigger instance for my
proxy ?
----
2019-03-23 21:31:54 UTC - Ali Ahmed: what’s the full stacktrace
----
2019-03-23 21:32:42 UTC - Ben k: Mar 23 21:14:56 localhost pulsar: # There is
insufficient memory for the Java Runtime Environment to continue.
Mar 23 21:14:56 localhost pulsar: # Native memory allocation (mmap) failed to
map 12884901888 bytes for committing reserved memory.
Mar 23 21:14:56 localhost pulsar: # An error report file with more information
is saved as:
Mar 23 21:14:56 localhost pulsar: # /opt/pulsar/hs_err_pid11841.log
Mar 23 21:14:56 localhost systemd: pulsar.proxy.service: main process exited,
code=exited, status=1/FAILURE
Mar 23 21:14:56 localhost systemd: Unit pulsar.proxy.service entered failed
state.
Mar 23 21:14:56 localhost systemd: pulsar.proxy.service failed.
Mar 23 21:14:57 localhost systemd: pulsar.proxy.service holdoff time over,
scheduling restart.
Mar 23 21:14:57 localhost systemd: Started Pulsar Proxy.
Mar 23 21:14:57 localhost systemd: Starting Pulsar Proxy...
Mar 23 21:14:57 localhost pulsar: OpenJDK 64-Bit Server VM warning: INFO:
os::commit_memory(0x00000004c0000000, 12884901888, 0) failed; error='Cannot
allocate memory' (errno=12)
Mar 23 21:14:57 localhost pulsar: #
Mar 23 21:14:57 localhost pulsar: # There is insufficient memory for the Java
Runtime Environment to continue.
Mar 23 21:14:57 localhost pulsar: # Native memory allocation (mmap) failed to
map 12884901888 bytes for committing reserved memory.
Mar 23 21:14:57 localhost pulsar: # An error report file with more information
is saved as:
Mar 23 21:14:57 localhost pulsar: # /opt/pulsar/hs_err_pid11955.log
Mar 23 21:14:57 localhost systemd: pulsar.proxy.service: main process exited,
code=exited, status=1/FAILURE
Mar 23 21:14:57 localhost systemd: Unit pulsar.proxy.service entered failed
state.
Mar 23 21:14:57 localhost systemd: pulsar.proxy.service failed.
Mar 23 21:14:58 localhost systemd: pulsar.proxy.service holdoff time over,
scheduling restart.
Mar 23 21:14:58 localhost systemd: start request repeated too quickly for
pulsar.proxy.service
Mar 23 21:14:58 localhost systemd: Failed to start Pulsar Proxy.
Mar 23 21:14:58 localhost systemd: Unit pulsar.proxy.service entered failed
state.
Mar 23 21:14:58 localhost systemd: pulsar.proxy.service failed.
----
2019-03-23 21:34:11 UTC - Ali Ahmed: you can probably change the config for the
memory
----
2019-03-23 21:34:15 UTC - Ali Ahmed: and run your instance
----
2019-03-23 21:37:05 UTC - Ben k: Where can I change the config for the memory .
Is this within the terraform.tfvars when I create the instances. Sorry I'm very
new to all this
----
2019-03-23 21:40:44 UTC - Ali Ahmed: it’s in the “deploy-pulsar.yaml”
```
- set_fact:
max_heap_memory: "12g"
max_direct_memory: "12g"
```
----
2019-03-23 21:44:04 UTC - Chris DiGiovanni: Sorry for the delay, that seemed to
fix that issue. Though now it seems to upload for a but then stall out. Never
really finishes. Getting errors like the below
----
2019-03-23 21:46:21 UTC - Chris DiGiovanni:
----
2019-03-23 21:47:05 UTC - Chris DiGiovanni: I can bounce the brokers, once that
has been done I can try to kick off another offload, it will upload some
data... then stall.
----
2019-03-23 22:42:48 UTC - Ben k: I get the same error on bookies now but not on
the proxy . I'm guessing the max heap in the deploy tank is shared among all
types of instances ?? . It fails to start bookie service but starts proxy now
----
2019-03-23 22:43:48 UTC - Ben k: I increased max heap and direct memory to 20g
----
2019-03-23 22:44:28 UTC - Ali Ahmed: I don’t think you that much memory in your
instance types I would recommend decreasing them.
----
2019-03-23 22:47:53 UTC - Ben k: Do you think 5g is ok ?
----
2019-03-23 22:49:03 UTC - Ali Ahmed: if it’s for basic testing you can go much
lower, your instance must support max_heap_memory + max_direct_memory
----
2019-03-23 22:50:58 UTC - Ben k: Oh ok . Trying that now . Appreciate the help
----
2019-03-23 23:09:45 UTC - Ben k: Thanks @Ali Ahmed that worked decreasing it .
Thanks for all your help , saved my dissertation
man-surfing : Matteo Merli
----
