Re: Review Request 43469: Added a parameter to specify the maximum number of tokens for tokenize.

[Ben Mahler]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43469/#review118988
-----------------------------------------------------------




3rdparty/libprocess/3rdparty/stout/include/stout/strings.hpp (lines 136 - 171)
<https://reviews.apache.org/r/43469/#comment180287>

    How about the following to simplify the logic and make it consistent with 
strings::split? Note that I noticed a bug in strings::split for n=1 and filed 
[MESOS-4656](https://issues.apache.org/jira/browse/MESOS-4656).
    
    ```
    inline std::vector<std::string> tokenize(
        const std::string& s,
        const std::string& delims,
        const Option<size_t>& maxTokens = None())
    {
      size_t offset = 0;
      std::vector<std::string> tokens;
    
      while (maxTokens.isNone() || maxTokens.get() > 0) {
        size_t nonDelim = s.find_first_not_of(delims, offset);
    
        if (nonDelim == std::string::npos) {
          break; // Nothing left
        }
    
        size_t delim = s.find_first_of(delims, nonDelim);
    
        // Finish tokenizing if this is the last token,
        // or we've found enough tokens.
        if (delim == std::string::npos ||
            (maxTokens.isSome() && tokens.size() == maxTokens.get() - 1)) {
          tokens.push_back(s.substr(nonDelim));
          break;
        }
    
        tokens.push_back(s.substr(nonDelim, delim - nonDelim));
        offset = delim;
      }
    
      return tokens;
    }
    ```


- Ben Mahler


On Feb. 11, 2016, 8:32 a.m., Guangya Liu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43469/
> -----------------------------------------------------------
> 
> (Updated Feb. 11, 2016, 8:32 a.m.)
> 
> 
> Review request for mesos and Ben Mahler.
> 
> 
> Bugs: MESOS-3833
>     https://issues.apache.org/jira/browse/MESOS-3833
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Added a parameter to specify the maximum number of tokens for tokenize.
> 
> 
> Diffs
> -----
> 
>   3rdparty/libprocess/3rdparty/stout/include/stout/strings.hpp 
> c20e2ffe128059f9999e63ac2630a2c73e512fc6 
>   3rdparty/libprocess/3rdparty/stout/tests/strings_tests.cpp 
> 8e7d3eeb22645c7600df1ee215a9de6a26166096 
> 
> Diff: https://reviews.apache.org/r/43469/diff/
> 
> 
> Testing
> -------
> 
> LiuGuangyas-MacBook-Pro:3rdparty gyliu$ ./stout-tests   
> --gtest_filter="StringsTest.Tokenize*" --verbose
> Note: Google Test filter = StringsTest.Tokenize*
> [==========] Running 15 tests from 1 test case.
> [----------] Global test environment set-up.
> [----------] 15 tests from StringsTest
> [ RUN      ] StringsTest.Tokenize
> [       OK ] StringsTest.Tokenize (0 ms)
> [ RUN      ] StringsTest.TokenizeStringWithDelimsAtStart
> [       OK ] StringsTest.TokenizeStringWithDelimsAtStart (0 ms)
> [ RUN      ] StringsTest.TokenizeStringWithDelimsAtEnd
> [       OK ] StringsTest.TokenizeStringWithDelimsAtEnd (0 ms)
> [ RUN      ] StringsTest.TokenizeStringWithDelimsAtStartAndEnd
> [       OK ] StringsTest.TokenizeStringWithDelimsAtStartAndEnd (0 ms)
> [ RUN      ] StringsTest.TokenizeWithMultipleDelims
> [       OK ] StringsTest.TokenizeWithMultipleDelims (0 ms)
> [ RUN      ] StringsTest.TokenizeEmptyString
> [       OK ] StringsTest.TokenizeEmptyString (0 ms)
> [ RUN      ] StringsTest.TokenizeDelimOnlyString
> [       OK ] StringsTest.TokenizeDelimOnlyString (0 ms)
> [ RUN      ] StringsTest.TokenizeNullByteDelim
> [       OK ] StringsTest.TokenizeNullByteDelim (0 ms)
> [ RUN      ] StringsTest.TokenizeNZero
> [       OK ] StringsTest.TokenizeNZero (0 ms)
> [ RUN      ] StringsTest.TokenizeNDelimOnlyString
> [       OK ] StringsTest.TokenizeNDelimOnlyString (0 ms)
> [ RUN      ] StringsTest.TokenizeN
> [       OK ] StringsTest.TokenizeN (0 ms)
> [ RUN      ] StringsTest.TokenizeNStringWithDelimsAtStart
> [       OK ] StringsTest.TokenizeNStringWithDelimsAtStart (0 ms)
> [ RUN      ] StringsTest.TokenizeNStringWithDelimsAtEnd
> [       OK ] StringsTest.TokenizeNStringWithDelimsAtEnd (0 ms)
> [ RUN      ] StringsTest.TokenizeNStringWithDelimsAtStartAndEnd
> [       OK ] StringsTest.TokenizeNStringWithDelimsAtStartAndEnd (0 ms)
> [ RUN      ] StringsTest.TokenizeNWithMultipleDelims
> [       OK ] StringsTest.TokenizeNWithMultipleDelims (0 ms)
> [----------] 15 tests from StringsTest (0 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 15 tests from 1 test case ran. (0 ms total)
> [  PASSED  ] 15 tests.
> 
> 
> Thanks,
> 
> Guangya Liu
> 
>