Re: [systemd-devel] Systemctl causes Spark native thread creation issue

[Rao Vz]

Changed to infinity and the error seems gone. Will keep monitoring for a
few days ..

Thanks guys!

Rao

On Tue, Feb 21, 2017 at 3:42 AM, Lennart Poettering 
wrote:

> On Mon, 20.02.17 16:44, Rao Vz (raoa...@gmail.com) wrote:
>
> > Hi, Guys
> >
> > We have a Apache Spark cluster of 3 nodes, one is master and slave, the
> > other two are slaves. When starting Spark worker with "systemctl start
> > spark-worker", when running out apps, sometimes but not always it
> generates
> > "java.lang.OutOfMemoryError: unable to create new native thread" error in
> > Spark worker logs.
>
> I figure the error is misleading and is not about memory at all, and
> you need to bump the default TasksMax= field or even turn it off by
> setting it to infinity.
>
> Lennart
>
> --
> Lennart Poettering, Red Hat
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Systemctl causes Spark native thread creation issue

[Lennart Poettering]

On Mon, 20.02.17 16:44, Rao Vz (raoa...@gmail.com) wrote:

> Hi, Guys
> 
> We have a Apache Spark cluster of 3 nodes, one is master and slave, the
> other two are slaves. When starting Spark worker with "systemctl start
> spark-worker", when running out apps, sometimes but not always it generates
> "java.lang.OutOfMemoryError: unable to create new native thread" error in
> Spark worker logs.

I figure the error is misleading and is not about memory at all, and
you need to bump the default TasksMax= field or even turn it off by
setting it to infinity.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Systemctl causes Spark native thread creation issue

[Cristian Rodríguez]

El 20-02-2017 a las 18:44, Rao Vz escribió:
> Hi, Guys

> Any help is appreciated.
> 
Most likely you went over TasksMax.


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] Systemctl causes Spark native thread creation issue

[Rao Vz]

Hi, Guys

We have a Apache Spark cluster of 3 nodes, one is master and slave, the
other two are slaves. When starting Spark worker with "systemctl start
spark-worker", when running out apps, sometimes but not always it generates
"java.lang.OutOfMemoryError: unable to create new native thread" error in
Spark worker logs.

If instead starting Spark worker directly (/opt/spark/sbin/start-slave.sh
spark://masterip:7077), it never causes any such error.

We tried tweaking ulimit and java options but did not have any luck.

The unit file (spark-worker.service) is like below:
[Unit]
Description=Spark Worker
After=network.target

[Service]
Type=forking
ExecStart=/opt/spark/sbin/start-slave.sh spark://masterIP:7077
ExecStop=/opt/spark/sbin/stop-slave.sh
StandardOutput=journal
StandardError=journal
LimitNOFILE=infinity
LimitMEMLOCK=infinity
LimitNPROC=infinity
LimitAS=infinity
CPUAccounting=true
CPUShares=100
Restart=always

[Install]
WantedBy=multi-user.target

Any help is appreciated.

Thanks,
Rao
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel