[jira] [Commented] (MESOS-7675) Isolate network ports.

[Jie Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137399#comment-16137399
 ] 

Jie Yu commented on MESOS-7675:
---

commit 40906e31a44848b826a94fbcde668661fe2028d4
Author: James Peach 
Date:   Tue Aug 22 13:37:55 2017 -0700

Moved the libnl3 configure checks into a macro.

Since the `network/ports` isolator will depend on libnl3, move those
checks into a separate macro so that we can call it again when we
add a configure option to enable it.

Review: https://reviews.apache.org/r/60902/

commit f7a38d7b1b1de6d52d5134364f257679de69505b
Author: James Peach 
Date:   Tue Aug 22 13:37:51 2017 -0700

Used common port range interval code in the port_mapping isolator.

Switched the port_mapping isolator over to start using the
common values code to parse port ranges into an IntervalSet.

Review: https://reviews.apache.org/r/61538/

commit daa77c66cd211b2f33c4fe4bd3dd0aa7f78430a8
Author: James Peach 
Date:   Tue Aug 22 13:37:49 2017 -0700

Added IntervalSet to Ranges conversion helpers.

Added a new `common/values.hpp` header file to expose IntervalSet
to Ranges conversion helper declarations.

The most common use of Range resources is for representing network
ports. Since ports are bounded to uint16_t it is awkward to store
them in a IntervalSet. To address this, convert the
IntervalSet helpers to templates so that we can convert between
IntervalSets of the appropriate type.

Review: https://reviews.apache.org/r/60836/

Last login: Sun Jul 30 16:55:41 on console
Jies-MacBook-Pro:~ jie$ tmux











































-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/source_context.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/struct.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/timestamp.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/type.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/wrappers.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/compiler/plugin.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/include/google/protobuf/compiler/profile.proto
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf/protobuf-targets.cmake
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf/protobuf-targets-noconfig.cmake
-- Up-to-date: /Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf/protobuf-config-version.cmake
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf/protobuf-config.cmake
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf/protobuf-module.cmake
-- Installing: 
/Users/jie/workspace/csi-spec/build/3rdparty/lib/cmake/protobuf/protobuf-options.cmake
[ 66%] Completed 'protobuf-3.3.2'
[ 66%] Built target protobuf-3.3.2
Scanning dependencies of target protoc
[ 66%] Built target protoc
[ 75%] Generating csi.proto
[ 83%] Running C++ protocol buffer compiler
Scanning dependencies of target csi
[ 91%] Building CXX object CMakeFiles/csi.dir/csi.pb.cc.o
[100%] Linking CXX static library libcsi.a
[100%] Built target csi
bash-3.2$ make
[ 66%] Built target protobuf-3.3.2
[ 66%] Built target protoc
[100%] Built target csi
bash-3.2$ make
[ 66%] Built target protobuf-3.3.2
[ 66%] Built target protoc
[100%] Built target csi
bash-3.2$ exit

bash-3.2$ cmake ..
-- Could NOT find Protobuf (missing: Protobuf_LIBRARIES Protobuf_INCLUDE_DIR)
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/jie/workspace/csi-spec/build
bash-3.2$ make
[ 66%] Built target protobuf-3.3.2
[ 66%] Built target protoc
[100%] Built target csi
bash-3.2$
0:vim  1:bash- 2:bash* 3:bash  4:bash   
   "Jies-MacBook-Pro.loca" 13:32 03-Aug-17  

"Jies-MacBook-Pro.loca" 18:09 02-Aug-17
  [Restored Aug 3, 2017, 2:17:52 PM]
Last login: Thu Aug  3 14:17:40 on console
Jies-MacBook-Pro:~ jie$ ls
Applications   DesktopDocuments  Downloads  Dropbox
Google Drive   LibraryMovies Music  Pictures   
Public VirtualBox VMs workspace
Jies-MacBook-Pro:~ jie$ tmux

Review: https://reviews.apache.org/r/60836/

commit 16cbd203bf5626ec1377a3b4ce772ce6dbaeb78a
Author: James Peach 
Date:   Tue Aug 22 13:37:45 2017 -0700

Use a consistent preprocessor check for ENABLE_PORT_MAPPING_ISOLATOR.

There's 

[jira] [Commented] (MESOS-7675) Isolate network ports.

[James Peach (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072946#comment-16072946
 ] 

James Peach commented on MESOS-7675:


Updated review chain:

| [r/60592|https://reviews.apache.org/r/60592] | Configure the `network/ports` 
isolator watch interval. |
| [r/60594|https://reviews.apache.org/r/60594] | Add a`network/ports` isolator 
nested container test. |
| [r/60593|https://reviews.apache.org/r/60593] | Test the `network/ports` 
isolator recovery. |
| [r/60591|https://reviews.apache.org/r/60591] | Optionally isolate only the 
agent network ports. |
| [r/60496|https://reviews.apache.org/r/60496] | WIP: Add socket checking to 
the network ports isolator. |
| [r/60495|https://reviews.apache.org/r/60495] | WIP: Network ports isolator 
listen socket utilities. |
| [r/60492|https://reviews.apache.org/r/60492] | Add network/ports isolator 
skeleton. |
| [r/60494|https://reviews.apache.org/r/60494] | Expose LinuxLauncher cgroups 
helper. |
| [r/60493|https://reviews.apache.org/r/60493] | Remove diagnostic socket IPv4 
assumptions. |
| [r/60491|https://reviews.apache.org/r/60491] | Capture the inode when 
scanning for sockets. |

> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using cgroup isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MESOS-7675) Isolate network ports.

[James Peach (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068609#comment-16068609
 ] 

James Peach commented on MESOS-7675:


TODO optionally check only listening sockets that are advertised by master.

> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using cgroup isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MESOS-7675) Isolate network ports.

[James Peach (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068513#comment-16068513
 ] 

James Peach commented on MESOS-7675:


{quote}
Would this monitor only the network ports advertised as `ports` resources? 
Wondering about interaction with ephemeral ports.
{quote}

It ensures that any ports that processes are listening on are within the 
allocated {{ports}} resources. So ephemeral ports bound by connecting to other 
services aren't checked.

> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using cgroup isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MESOS-7675) Isolate network ports.

[James DeFelice (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068462#comment-16068462
 ] 

James DeFelice commented on MESOS-7675:
---

Would this monitor only the network ports advertised as `ports` resources? 
Wondering about interaction with ephemeral ports.

> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using cgroup isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MESOS-7675) Isolate network ports.

[James Peach (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066082#comment-16066082
 ] 

James Peach commented on MESOS-7675:


Posted some reviews for early feedback.

These are trivial fixes that we need for the isolator:

| [r/60494|https://reviews.apache.org/r/60494] | Expose LinuxLauncher cgroups 
helper. |
| [r/60493|https://reviews.apache.org/r/60493] | Remove diagnostic socket IPv4 
assumptions. |
| [r/60491|https://reviews.apache.org/r/60491] | Capture the inode when 
scanning for sockets. |

This is the isolator itself:

| [r/60496|https://reviews.apache.org/r/60496] | WIP: Add socket checking to 
the network ports isolator. |
| [r/60495|https://reviews.apache.org/r/60495] | WIP: Network ports isolator 
listen socket utilities. |
| [r/60492|https://reviews.apache.org/r/60492] | Add network/ports isolator 
skeleton. |

There are a couple of issues I'd like to get feedback on
* What's the right way to only isolate tasks with host networking?
* Should we do the socket scanning in a background process?
* What should we do about the command executor using unallocated ports?





> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using cgroup isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MESOS-7675) Isolate network ports.

[James Peach (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062527#comment-16062527
 ] 

James Peach commented on MESOS-7675:


{quote}
Also, this seems like we need to perform the algorithm for the lifetime of 
every task running on the agent?
{quote}

Yes, this would behave similarly to the {{posix/disk}} isolator where we 
periodically scan to check the resource usage. I couldn't find any way to get 
netlink notifications on listening sockets.

{quote}
I am assuming this would work only for tasks on the host network.
{quote}

You could do the same algorithm with a network namespace, though it would be a 
bit more involved and int most cases it wouldn't be especially helpful. For now 
I'm only proposing to do this for the host network.

> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using cgroup isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (MESOS-7675) Isolate network ports.

[Avinash Sridharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062520#comment-16062520
 ] 

Avinash Sridharan commented on MESOS-7675:
--

[~jpe...@apache.org] I am assuming this would work only for tasks on the host 
network. Also, this seems like we need to perform the algorithm for the 
lifetime of every task running on the agent? How do you propose we do this. By 
doing a periodic scan?

PS: By group isolation, did you mean cgroup isolation?

> Isolate network ports.
> --
>
> Key: MESOS-7675
> URL: https://issues.apache.org/jira/browse/MESOS-7675
> Project: Mesos
>  Issue Type: Improvement
>  Components: agent
>Reporter: James Peach
>Assignee: James Peach
>Priority: Minor
>
> If a task uses network ports, there is no isolator that can enforce that it 
> only listens on the ports that it has resources for. Implement a ports 
> isolator that can limit tasks to listen only on allocated TCP ports.
> Roughly, the algorithm for this follows what standard tools like {{lsof}} and 
> {{ss}} do.
> * Find all the listening TCP sockets (using netlink)
> * Index the sockets by their node (from the netlink information)
> * Find all the open sockets on the system (by scanning {{/proc/\*/fd/\*}} 
> links)
> * For each open socket, check whether its node (given in the link target) in 
> the set of listen sockets that we scanned
> * If the socket is a listening socket and the corresponding PID is in the 
> task, send a resource limitation for the task
> Matching pids to tasks depends on using group isolation, otherwise we would 
> have to build a full process tree, which would be nice to avoid.
> Scanning all the open sockets can be avoided by using the {{net_cls}} 
> isolator with kernel + libnl3 patches to publish the socket classid when we 
> find the listening socket.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)