[
https://issues.apache.org/jira/browse/YARN-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-9669:
----------------------------
Description:
Docker container can run on Hadoop cluster fine today in Hadoop 3.1.1+ release
and trunk code base. We are getting feedback from potential customers for
advance features that may take Docker support to the next level. The
frequently requested features are:
h2. Improve cgroups support
h3. Support systemd based cgorup
YARN-9660 provides instruction to setup cgroups fs to control Docker container
with cgroup. The next step is to support Docker cgroups with systemd natively.
This will reduce some configuration steps on Docker side to make the user
experience more seamless.
h3. Disk throttling
Docker supports IO throttling with several docker parameters:
{code}--device-read-bps=""
--device-write-bps=""
--device-read-iops=""
--device-write-iops=""{code}
This looks like a good feature for YARN to specify IO throughput for stress
test simulation.
h2. Docker volume support
Docker can create data container of fixed size. This is useful to limit disk
usage for containers. This feature provides resource management to data, and
ability to create docker volume and anchor docker containers to colocate with
data container.
h2. Protection against overlay network spike
Badly designed docker container can cause system to become non-responsive, if
network spikes occurs or forking uncontrollably. Some protection maybe useful
to guard against network flood and fork bomb. Active prevention can shutdown
the crashing docker container before system hangs.
h2. Docker-compose support
A few people have expressed interests to see the possibility to consume docker
compose yaml file, and start a dock swarm like cluster. YARN service REST API
can have alignment in this area. In term, support docker swarm like cluster in
Hadoop.
h2. OCI Runtime support
Deeper integration with Runc can provide more standard API integration with
Docker core engine. This is continuation work to follow up on YARN-9560 and
associated JIRAs.
was:
Docker container can run on Hadoop cluster fine today in Hadoop 3.1.1+ release
and trunk code base. We are getting feedback from potential customers for
advance features that may take Docker support to the next level. The
frequently requested features are:
h3. Improve cgroups support
h4. Support systemd based cgorup
YARN-9660 provides instruction to setup cgroups fs to control Docker container
with cgroup. The next step is to support Docker cgroups with systemd natively.
This will reduce some configuration steps on Docker side to make the user
experience more seamless.
h4. Disk throttling
Docker supports IO throttling with several docker parameters:
{code}--device-read-bps=""
--device-write-bps=""
--device-read-iops=""
--device-write-iops=""{code}
This looks like a good feature for YARN to specify IO throughput for stress
test simulation.
h3. Docker volume support
Docker can create data container of fixed size. This is useful to limit disk
usage for containers. This feature provides resource management to data, and
ability to create docker volume and anchor docker containers to colocate with
data container.
h3. Protection against overlay network spike
Badly designed docker container can cause system to become non-responsive, if
network spikes occurs or forking uncontrollably. Some protection maybe useful
to guard against network flood and fork bomb. Active prevention can shutdown
the crashing docker container before system hangs.
> YARN Container phase 3
> ----------------------
>
> Key: YARN-9669
> URL: https://issues.apache.org/jira/browse/YARN-9669
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Eric Yang
> Priority: Major
>
> Docker container can run on Hadoop cluster fine today in Hadoop 3.1.1+
> release and trunk code base. We are getting feedback from potential
> customers for advance features that may take Docker support to the next
> level. The frequently requested features are:
> h2. Improve cgroups support
> h3. Support systemd based cgorup
> YARN-9660 provides instruction to setup cgroups fs to control Docker
> container with cgroup. The next step is to support Docker cgroups with
> systemd natively. This will reduce some configuration steps on Docker side
> to make the user experience more seamless.
> h3. Disk throttling
> Docker supports IO throttling with several docker parameters:
> {code}--device-read-bps=""
> --device-write-bps=""
> --device-read-iops=""
> --device-write-iops=""{code}
> This looks like a good feature for YARN to specify IO throughput for stress
> test simulation.
> h2. Docker volume support
> Docker can create data container of fixed size. This is useful to limit disk
> usage for containers. This feature provides resource management to data, and
> ability to create docker volume and anchor docker containers to colocate with
> data container.
> h2. Protection against overlay network spike
> Badly designed docker container can cause system to become non-responsive, if
> network spikes occurs or forking uncontrollably. Some protection maybe
> useful to guard against network flood and fork bomb. Active prevention can
> shutdown the crashing docker container before system hangs.
> h2. Docker-compose support
> A few people have expressed interests to see the possibility to consume
> docker compose yaml file, and start a dock swarm like cluster. YARN service
> REST API can have alignment in this area. In term, support docker swarm like
> cluster in Hadoop.
> h2. OCI Runtime support
> Deeper integration with Runc can provide more standard API integration with
> Docker core engine. This is continuation work to follow up on YARN-9560 and
> associated JIRAs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
