Re: Review Request 41713: Added HDFS URI fetcher plugin.

[haosdent huang]

> On Jan. 3, 2016, 10:50 a.m., Adam B wrote:
> > src/uri/fetchers/hadoop.cpp, line 54
> > 
> >
> > Why is this hardcoded again? We moved away from that for 
> > https://issues.apache.org/jira/browse/MESOS-1711 but now tachyon/maprfs 
> > users will be broken (again).
> > Can we make the supported schemes a plugin flag/parameter, so users can 
> > adjust as needed?
> 
> Jie Yu wrote:
> Yeah, I'll follow up with a patch to allow users to configure schemes 
> through a flag.

Seems hadoop also support below scheme:

file,hdfs,webhdfs,swebhdfs,har,viewfs,ftp,s3a,wasb,swift


- haosdent


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112444
---


On Dec. 31, 2015, 5:35 a.m., Jie Yu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> ---
> 
> (Updated Dec. 31, 2015, 5:35 a.m.)
> 
> 
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and 
> Vinod Kone.
> 
> 
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin 
> initialization logic so that flags can be used to initialize each plugin.
> 
> 
> Diffs
> -
> 
>   include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c 
>   include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23 
>   src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6 
>   src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98 
>   src/uri/fetcher.hpp PRE-CREATION 
>   src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886 
>   src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937 
>   src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892 
>   src/uri/fetchers/hadoop.hpp PRE-CREATION 
>   src/uri/fetchers/hadoop.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41713/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Jie Yu
> 
>

Re: Review Request 41713: Added HDFS URI fetcher plugin.

[Adam B]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112444
---


Hardcoded hadoop schemes? Eek! Guess we'll have to reopen MESOS-1711?


src/uri/fetchers/hadoop.cpp (line 54)


Why is this hardcoded again? We moved away from that for 
https://issues.apache.org/jira/browse/MESOS-1711 but now tachyon/maprfs users 
will be broken (again).
Can we make the supported schemes a plugin flag/parameter, so users can 
adjust as needed?


- Adam B


On Dec. 30, 2015, 9:35 p.m., Jie Yu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> ---
> 
> (Updated Dec. 30, 2015, 9:35 p.m.)
> 
> 
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and 
> Vinod Kone.
> 
> 
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin 
> initialization logic so that flags can be used to initialize each plugin.
> 
> 
> Diffs
> -
> 
>   include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c 
>   include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23 
>   src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6 
>   src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98 
>   src/uri/fetcher.hpp PRE-CREATION 
>   src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886 
>   src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937 
>   src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892 
>   src/uri/fetchers/hadoop.hpp PRE-CREATION 
>   src/uri/fetchers/hadoop.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41713/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Jie Yu
> 
>

Re: Review Request 41713: Added HDFS URI fetcher plugin.

[Jie Yu]

> On Jan. 3, 2016, 10:50 a.m., Adam B wrote:
> > src/uri/fetchers/hadoop.cpp, line 54
> > 
> >
> > Why is this hardcoded again? We moved away from that for 
> > https://issues.apache.org/jira/browse/MESOS-1711 but now tachyon/maprfs 
> > users will be broken (again).
> > Can we make the supported schemes a plugin flag/parameter, so users can 
> > adjust as needed?

Yeah, I'll follow up with a patch to allow users to configure schemes through a 
flag.


- Jie


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112444
---


On Dec. 31, 2015, 5:35 a.m., Jie Yu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> ---
> 
> (Updated Dec. 31, 2015, 5:35 a.m.)
> 
> 
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and 
> Vinod Kone.
> 
> 
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin 
> initialization logic so that flags can be used to initialize each plugin.
> 
> 
> Diffs
> -
> 
>   include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c 
>   include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23 
>   src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6 
>   src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98 
>   src/uri/fetcher.hpp PRE-CREATION 
>   src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886 
>   src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937 
>   src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892 
>   src/uri/fetchers/hadoop.hpp PRE-CREATION 
>   src/uri/fetchers/hadoop.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41713/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Jie Yu
> 
>

Re: Review Request 41713: Added HDFS URI fetcher plugin.

[Jie Yu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/
---

(Updated Dec. 31, 2015, 5:35 a.m.)


Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and 
Vinod Kone.


Changes
---

Addressed review comments.


Bugs: MESOS-3925
https://issues.apache.org/jira/browse/MESOS-3925


Repository: mesos


Description
---

Added HDFS URI fetcher plugin. This patch also adjusted the plugin 
initialization logic so that flags can be used to initialize each plugin.


Diffs (updated)
-

  include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c 
  include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23 
  src/Makefile.am bf71fe6ee65233bbc47a41f610d68ff9760e19f6 
  src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98 
  src/uri/fetcher.hpp PRE-CREATION 
  src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886 
  src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937 
  src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892 
  src/uri/fetchers/hadoop.hpp PRE-CREATION 
  src/uri/fetchers/hadoop.cpp PRE-CREATION 

Diff: https://reviews.apache.org/r/41713/diff/


Testing
---

make check


Thanks,

Jie Yu

Re: Review Request 41713: Added HDFS URI fetcher plugin.

[Timothy Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/#review112105
---

Ship it!


Ship It!


src/uri/fetcher.cpp (line 57)


Missing starting single quote



src/uri/fetcher.cpp (line 63)


I think we should warn when there is already a plugin for the scheme.


- Timothy Chen


On Dec. 28, 2015, 6:07 p.m., Jie Yu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41713/
> ---
> 
> (Updated Dec. 28, 2015, 6:07 p.m.)
> 
> 
> Review request for mesos, Adam B, Ben Mahler, Ian Downes, Timothy Chen, and 
> Vinod Kone.
> 
> 
> Bugs: MESOS-3925
> https://issues.apache.org/jira/browse/MESOS-3925
> 
> 
> Repository: mesos
> 
> 
> Description
> ---
> 
> Added HDFS URI fetcher plugin. This patch also adjusted the plugin 
> initialization logic so that flags can be used to initialize each plugin.
> 
> 
> Diffs
> -
> 
>   include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c 
>   include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23 
>   src/Makefile.am 8af0115caa67ac8f3193d8f0d0f1a4ae739af275 
>   src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98 
>   src/uri/fetcher.hpp PRE-CREATION 
>   src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886 
>   src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937 
>   src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892 
>   src/uri/fetchers/hadoop.hpp PRE-CREATION 
>   src/uri/fetchers/hadoop.cpp PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/41713/diff/
> 
> 
> Testing
> ---
> 
> make check
> 
> 
> Thanks,
> 
> Jie Yu
> 
>

Review Request 41713: Added HDFS URI fetcher plugin.

[Jie Yu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41713/
---

Review request for mesos, Adam B, Ben Mahler, Ian Downes, and Vinod Kone.


Bugs: MESOS-3925
https://issues.apache.org/jira/browse/MESOS-3925


Repository: mesos


Description
---

Added HDFS URI fetcher plugin. This patch also adjusted the plugin 
initialization logic so that flags can be used to initialize each plugin.


Diffs
-

  include/mesos/uri/fetcher.hpp 6cf7b37818ceff64ffe56138ebd4bc7e2abe5a7c 
  include/mesos/uri/uri.hpp aa3ab5d24bb8c501dd19e93d7563cb0afd889f23 
  src/Makefile.am 8af0115caa67ac8f3193d8f0d0f1a4ae739af275 
  src/tests/uri_fetcher_tests.cpp d54a7e7b08a741b1d5d371c11747681ae4ddef98 
  src/uri/fetcher.hpp PRE-CREATION 
  src/uri/fetcher.cpp 8bd7cf3d638a737c40183294cb4548a4851e0886 
  src/uri/fetchers/curl.hpp 401829d1990ec817ea069ce45f902c521cd1a937 
  src/uri/fetchers/curl.cpp 6c607a6a800374b6e14b37bfdc6b04e7a4e87892 
  src/uri/fetchers/hadoop.hpp PRE-CREATION 
  src/uri/fetchers/hadoop.cpp PRE-CREATION 

Diff: https://reviews.apache.org/r/41713/diff/


Testing
---

make check


Thanks,

Jie Yu