Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-11-27 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/
---

(Updated Nov. 27, 2017, 10:14 p.m.)


Review request for mesos, Greg Mann, Jie Yu, Joseph Wu, and Jan Schlicht.


Changes
---

Rebased.


Bugs: MESOS-8097
https://issues.apache.org/jira/browse/MESOS-8097


Repository: mesos


Description
---

A local resource provide can store checkpoints, whose lifecycles should
be tied to the agent, under
`/meta/slaves//resource_providers///
`.


Diffs (updated)
-

  src/resource_provider/daemon.cpp 80b5c71e266bda2f15d5dfb9b1c15f01c6aa8e93 
  src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
  src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
  src/resource_provider/storage/provider.hpp 
6de88c2329b358fcf48bc39ddda0132170991c3c 
  src/resource_provider/storage/provider.cpp 
49c042cdb1837860aaedde2e48f318ed5ac8b1d1 
  src/slave/paths.hpp 66dfa4544772d78ccc9229dc861da60c79913f24 
  src/slave/paths.cpp b03ffeeed83cb73228cca27769262fb08df38fb5 


Diff: https://reviews.apache.org/r/63018/diff/7/

Changes: https://reviews.apache.org/r/63018/diff/6-7/


Testing
---

make


Thanks,

Chun-Hung Hsiao

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-11-23 Thread Jie Yu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/#review191820
---


Ship it!




Ship It!

- Jie Yu


On Nov. 21, 2017, 5:34 a.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63018/
> ---
> 
> (Updated Nov. 21, 2017, 5:34 a.m.)
> 
> 
> Review request for mesos, Greg Mann, Jie Yu, Joseph Wu, and Jan Schlicht.
> 
> 
> Bugs: MESOS-8097
> https://issues.apache.org/jira/browse/MESOS-8097
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> A local resource provide can store checkpoints, whose lifecycles should
> be tied to the agent, under
> `/meta/slaves//resource_providers///
> `.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
>   src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
>   src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
>   src/resource_provider/storage/provider.hpp 
> 6de88c2329b358fcf48bc39ddda0132170991c3c 
>   src/resource_provider/storage/provider.cpp 
> 46224997430ac0c568904d80014166a6f059907f 
>   src/slave/paths.hpp 66dfa4544772d78ccc9229dc861da60c79913f24 
>   src/slave/paths.cpp b03ffeeed83cb73228cca27769262fb08df38fb5 
> 
> 
> Diff: https://reviews.apache.org/r/63018/diff/6/
> 
> 
> Testing
> ---
> 
> make
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-11-20 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/
---

(Updated Nov. 21, 2017, 5:34 a.m.)


Review request for mesos, Greg Mann, Jie Yu, Joseph Wu, and Jan Schlicht.


Changes
---

Removed `getResourceProviderAgentRootDir()`.


Bugs: MESOS-8097
https://issues.apache.org/jira/browse/MESOS-8097


Repository: mesos


Description (updated)
---

A local resource provide can store checkpoints, whose lifecycles should
be tied to the agent, under
`/meta/slaves//resource_providers///
`.


Diffs (updated)
-

  src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
  src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
  src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
  src/resource_provider/storage/provider.hpp 
6de88c2329b358fcf48bc39ddda0132170991c3c 
  src/resource_provider/storage/provider.cpp 
46224997430ac0c568904d80014166a6f059907f 
  src/slave/paths.hpp 66dfa4544772d78ccc9229dc861da60c79913f24 
  src/slave/paths.cpp b03ffeeed83cb73228cca27769262fb08df38fb5 


Diff: https://reviews.apache.org/r/63018/diff/6/

Changes: https://reviews.apache.org/r/63018/diff/5-6/


Testing
---

make


Thanks,

Chun-Hung Hsiao

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-11-16 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/
---

(Updated Nov. 17, 2017, 1:36 a.m.)


Review request for mesos, Greg Mann, Jie Yu, Joseph Wu, and Jan Schlicht.


Bugs: MESOS-8097
https://issues.apache.org/jira/browse/MESOS-8097


Repository: mesos


Description
---

A local resource provide can store checkpoints, whose lifecycles should
be tied to the agent, under
`/meta/slaves//resource_providers///
`. Data that persist across agents can be stored under
`/resource_providers//`.


Diffs (updated)
-

  src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
  src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
  src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
  src/resource_provider/storage/provider.hpp 
6de88c2329b358fcf48bc39ddda0132170991c3c 
  src/resource_provider/storage/provider.cpp 
46224997430ac0c568904d80014166a6f059907f 
  src/slave/paths.hpp 66dfa4544772d78ccc9229dc861da60c79913f24 
  src/slave/paths.cpp b03ffeeed83cb73228cca27769262fb08df38fb5 


Diff: https://reviews.apache.org/r/63018/diff/5/

Changes: https://reviews.apache.org/r/63018/diff/4-5/


Testing
---

make


Thanks,

Chun-Hung Hsiao

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-11-03 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/
---

(Updated Nov. 4, 2017, 12:52 a.m.)


Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.


Changes
---

Added `getResourceProviderAgentRootDir()` function.


Bugs: MESOS-8097
https://issues.apache.org/jira/browse/MESOS-8097


Repository: mesos


Description (updated)
---

A local resource provide can store checkpoints, whose lifecycles should
be tied to the agent, under
`/meta/slaves//resource_providers///
`. Data that persist across agents can be stored under
`/resource_providers//`.


Diffs (updated)
-

  src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
  src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
  src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
  src/resource_provider/storage/provider.hpp 
6de88c2329b358fcf48bc39ddda0132170991c3c 
  src/resource_provider/storage/provider.cpp 
46224997430ac0c568904d80014166a6f059907f 
  src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
  src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 


Diff: https://reviews.apache.org/r/63018/diff/4/

Changes: https://reviews.apache.org/r/63018/diff/3-4/


Testing
---

make


Thanks,

Chun-Hung Hsiao

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-10-27 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/
---

(Updated Oct. 27, 2017, 10:07 p.m.)


Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.


Changes
---

Removed the cross-agent `resource_providers` directory.


Bugs: MESOS-8097
https://issues.apache.org/jira/browse/MESOS-8097


Repository: mesos


Description (updated)
---

A local resource provide can store checkpoints, whose lifecycles should
be tied to the slave id, under
`/meta/slaves//resource_providers///
`.


Diffs (updated)
-

  src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
  src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
  src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
  src/resource_provider/storage/provider.hpp 
6de88c2329b358fcf48bc39ddda0132170991c3c 
  src/resource_provider/storage/provider.cpp 
46224997430ac0c568904d80014166a6f059907f 
  src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
  src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 


Diff: https://reviews.apache.org/r/63018/diff/3/

Changes: https://reviews.apache.org/r/63018/diff/2-3/


Testing
---

make


Thanks,

Chun-Hung Hsiao

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-10-27 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/#review189484
---




src/slave/paths.hpp
Lines 55 (patched)


Working on a patch that removes this `resource_providers` directory. The 
fact that there're two `resource_providers` directories in this file, one lives 
across agents but the other doesn't, are confusing.


- Chun-Hung Hsiao


On Oct. 20, 2017, 8:28 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63018/
> ---
> 
> (Updated Oct. 20, 2017, 8:28 p.m.)
> 
> 
> Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.
> 
> 
> Bugs: MESOS-8097
> https://issues.apache.org/jira/browse/MESOS-8097
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> A local resource provide can use
> `work_dir/resource_providers///` to store whatever data
> whose lifecycle is not tied to the agent ID, and store checkpoints,
> whose lifecycles should be tied to the agent ID, under
> `work_dir/meta/slaves/latest/resource_providers///`.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
>   src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
>   src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
>   src/resource_provider/storage/provider.hpp 
> 6de88c2329b358fcf48bc39ddda0132170991c3c 
>   src/resource_provider/storage/provider.cpp 
> 46224997430ac0c568904d80014166a6f059907f 
>   src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
>   src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 
> 
> 
> Diff: https://reviews.apache.org/r/63018/diff/2/
> 
> 
> Testing
> ---
> 
> make
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-10-20 Thread Chun-Hung Hsiao 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/
---

(Updated Oct. 20, 2017, 8:28 p.m.)


Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.


Changes
---

Addressed Jie's comments.


Bugs: MESOS-8097
https://issues.apache.org/jira/browse/MESOS-8097


Repository: mesos


Description
---

A local resource provide can use
`work_dir/resource_providers///` to store whatever data
whose lifecycle is not tied to the agent ID, and store checkpoints,
whose lifecycles should be tied to the agent ID, under
`work_dir/meta/slaves/latest/resource_providers///`.


Diffs (updated)
-

  src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
  src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
  src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
  src/resource_provider/storage/provider.hpp 
6de88c2329b358fcf48bc39ddda0132170991c3c 
  src/resource_provider/storage/provider.cpp 
46224997430ac0c568904d80014166a6f059907f 
  src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
  src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 


Diff: https://reviews.apache.org/r/63018/diff/2/

Changes: https://reviews.apache.org/r/63018/diff/1-2/


Testing
---

make


Thanks,

Chun-Hung Hsiao

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-10-17 Thread Jie Yu 


> On Oct. 18, 2017, 12:37 a.m., Jie Yu wrote:
> > src/slave/paths.hpp
> > Lines 59 (patched)
> > 
> >
> > It would be nice to call out what will be stored under this directory. 
> > For instance, csi endpoint, and volume bind mounts.
> 
> Chun-Hung Hsiao wrote:
> I didn't do that because the CSI endpoints and volumes are storage 
> specific. Do you think it is a good idea to note some examples specific to 
> storage local resource provider?

ah, ic. good point. I guess I am asking because we have two directories for 
resource providers. How do one decide to which directory to put RP related data?


- Jie


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/#review188183
---


On Oct. 16, 2017, 6:25 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63018/
> ---
> 
> (Updated Oct. 16, 2017, 6:25 p.m.)
> 
> 
> Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.
> 
> 
> Bugs: MESOS-8097
> https://issues.apache.org/jira/browse/MESOS-8097
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> A local resource provide can use
> `work_dir/resource_providers///` to store whatever data
> whose lifecycle is not tied to the agent ID, and store checkpoints,
> whose lifecycles should be tied to the agent ID, under
> `work_dir/meta/slaves/latest/resource_providers///`.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
>   src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
>   src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
>   src/resource_provider/storage/provider.hpp 
> 6de88c2329b358fcf48bc39ddda0132170991c3c 
>   src/resource_provider/storage/provider.cpp 
> 46224997430ac0c568904d80014166a6f059907f 
>   src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
>   src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 
> 
> 
> Diff: https://reviews.apache.org/r/63018/diff/1/
> 
> 
> Testing
> ---
> 
> make
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-10-17 Thread Chun-Hung Hsiao 


> On Oct. 18, 2017, 12:37 a.m., Jie Yu wrote:
> > src/slave/paths.hpp
> > Lines 59 (patched)
> > 
> >
> > It would be nice to call out what will be stored under this directory. 
> > For instance, csi endpoint, and volume bind mounts.

I didn't do that because the CSI endpoints and volumes are storage specific. Do 
you think it is a good idea to note some examples specific to storage local 
resource provider?


- Chun-Hung


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/#review188183
---


On Oct. 16, 2017, 6:25 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63018/
> ---
> 
> (Updated Oct. 16, 2017, 6:25 p.m.)
> 
> 
> Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.
> 
> 
> Bugs: MESOS-8097
> https://issues.apache.org/jira/browse/MESOS-8097
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> A local resource provide can use
> `work_dir/resource_providers///` to store whatever data
> whose lifecycle is not tied to the agent ID, and store checkpoints,
> whose lifecycles should be tied to the agent ID, under
> `work_dir/meta/slaves/latest/resource_providers///`.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
>   src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
>   src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
>   src/resource_provider/storage/provider.hpp 
> 6de88c2329b358fcf48bc39ddda0132170991c3c 
>   src/resource_provider/storage/provider.cpp 
> 46224997430ac0c568904d80014166a6f059907f 
>   src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
>   src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 
> 
> 
> Diff: https://reviews.apache.org/r/63018/diff/1/
> 
> 
> Testing
> ---
> 
> make
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>

Re: Review Request 63018: Added filesystem layout for local resource providers.

2017-10-17 Thread Jie Yu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63018/#review188183
---


Fix it, then Ship it!





src/slave/paths.hpp
Lines 59 (patched)


It would be nice to call out what will be stored under this directory. For 
instance, csi endpoint, and volume bind mounts.



src/slave/paths.hpp
Lines 84-85 (patched)


A single `state` file because operations and resources have to be 
checkpointed atomically.


- Jie Yu


On Oct. 16, 2017, 6:25 p.m., Chun-Hung Hsiao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63018/
> ---
> 
> (Updated Oct. 16, 2017, 6:25 p.m.)
> 
> 
> Review request for mesos, Jie Yu, Joseph Wu, and Jan Schlicht.
> 
> 
> Bugs: MESOS-8097
> https://issues.apache.org/jira/browse/MESOS-8097
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> A local resource provide can use
> `work_dir/resource_providers///` to store whatever data
> whose lifecycle is not tied to the agent ID, and store checkpoints,
> whose lifecycles should be tied to the agent ID, under
> `work_dir/meta/slaves/latest/resource_providers///`.
> 
> 
> Diffs
> -
> 
>   src/resource_provider/daemon.cpp d584eb9d7aa75522aec97277674321061b90fbed 
>   src/resource_provider/local.hpp ebaa07d03ad77d516066ee2d4b60864be0611b5f 
>   src/resource_provider/local.cpp ad98f333c5668ca81de6e7ed3fc8f59323b151da 
>   src/resource_provider/storage/provider.hpp 
> 6de88c2329b358fcf48bc39ddda0132170991c3c 
>   src/resource_provider/storage/provider.cpp 
> 46224997430ac0c568904d80014166a6f059907f 
>   src/slave/paths.hpp f000508d414daf9f943561f89c7105503a8a98b3 
>   src/slave/paths.cpp fd546525b900cb6524fb9196d19616ef18de0f30 
> 
> 
> Diff: https://reviews.apache.org/r/63018/diff/1/
> 
> 
> Testing
> ---
> 
> make
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>