[issue17343] Add a version of str.split which returns an iterator

Juancarlo Añez added the comment: def isplit(text, sep=None, maxsplit=-1): """ A lowmemory-footprint version of: iter(text.split(sep, maxsplit)) Adapted from https://stackoverflow.com/a/9770397 """ if maxsplit == 0: yield text else: rsep =

[issue17343] Add a version of str.split which returns an iterator

Paweł Miech added the comment: Making string.split iterator sounds like an interesting task. I found this issue because recently we talked in project that string.split returns a list and it can cause increased memory usage footprint for some tasks when there is large response to parse.

[issue17343] Add a version of str.split which returns an iterator

Change by Brett Cannon : -- nosy: -brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue17343] Add a version of str.split which returns an iterator

Martin Winks added the comment: > Perhaps the use case is already served by re.finditer() def split_whitespace_ascii(s: str): return (pt.group(0) for pt in re.finditer(r"[A-Za-z']+", s)) solution above does not cover all possible data and is incorrect for bytes-like objects. writing

[issue17343] Add a version of str.split which returns an iterator

Changes by Serhiy Storchaka : -- resolution: -> rejected stage: needs patch -> resolved status: pending -> closed ___ Python tracker

[issue17343] Add a version of str.split which returns an iterator

Changes by Serhiy Storchaka : -- status: open -> pending ___ Python tracker ___

[issue17343] Add a version of str.split which returns an iterator

Raymond Hettinger added the comment: No one has submitted a patch for this or has expressed an interest in a long time. Perhaps the use case is already served by re.finditer() Unassigning. Feel free to push this forward or to close due to lack on interest. -- assignee: rhettinger

[issue17343] Add a version of str.split which returns an iterator

Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___ ___

[issue17343] Add a version of str.split which returns an iterator

Georg Brandl added the comment: I'm guessing Terry wanted to say os.listdir instead of os.walk. -- nosy: +georg.brandl ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___

[issue17343] Add a version of str.split which returns an iterator

Serhiy Storchaka added the comment: May be str.iter_indices() or even just str.indices()? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___

[issue17343] Add a version of str.split which returns an iterator

Terry J. Reedy added the comment: I'm guessing Terry wanted to say os.listdir instead of os.walk. yes, sorry. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___

[issue17343] Add a version of str.split which returns an iterator

Raymond Hettinger added the comment: If someone wants whip-up a patch for str.iter_index(), I would be happy to review it. Be sure to add a test case to make sure that the results are non-overlapping: list(''.iter_index('aa')) == [0, 2] -- assignee: - rhettinger

[issue17343] Add a version of str.split which returns an iterator

Raymond Hettinger added the comment: -1 on os.walk returning an iterator. The API is already a bit challenging for some and our experience with itertools.groupby() is that returning an inner iterator can be very confusing. -- nosy: +rhettinger ___

[issue17343] Add a version of str.split which returns an iterator

Alex Gaynor added the comment: Raymond: Is that for the wrong ticket, or was the message incorrect? :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___

[issue17343] Add a version of str.split which returns an iterator

Raymond Hettinger added the comment: Alex, it was response to Terry's message: http://bugs.python.org/issue17343#msg183782 FWIW, I'm +1 on an iterator version of str.split(). I'm not sure yet that it would be worthwhile to propagate the idea to other string-like objects though. --

[issue17343] Add a version of str.split which returns an iterator

Gregory P. Smith added the comment: It'd perhaps have been better if things like memoryview were never exposed to the user at all as a distinct type and became an internal implementation detail behind PyBytes and PyUnicode objects (they could hold a reference to something else or collapse

[issue17343] Add a version of str.split which returns an iterator

Changes by Tshepang Lekhonkhobe tshep...@gmail.com: -- nosy: +tshepang ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___ ___

[issue17343] Add a version of str.split which returns an iterator

Terry J. Reedy added the comment: I personally would have changed both str.split and os.walk to return iterators in 3.0, like many other builtins. The rationale for os.walk continuing to produce a list is that there would be little time saving as the list is not *that* long and most uses look

[issue17343] Add a version of str.split which returns an iterator

Serhiy Storchaka added the comment: Indeed, a bytearray version would require the talked about but not implemented due to complexity (in pep3118) support for locking a buffer from other mutations. I rather think that a bytearray version can't pre-scan the data. Note that an array for

[issue17343] Add a version of str.split which returns an iterator

Serhiy Storchaka added the comment: There is no string view that I know of. Interesting idea, though, thanks to the immutability of strings. Would much have to be different other than boundary checking and __hash__ (and hoping extension authors are changing things in-place)?

[issue17343] Add a version of str.split which returns an iterator

Changes by Brett Cannon br...@python.org: -- nosy: +brett.cannon ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___ ___ Python-bugs-list

[issue17343] Add a version of str.split which returns an iterator

Changes by Santoso Wijaya santoso.wij...@gmail.com: -- nosy: +santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue17343 ___ ___

[issue17343] Add a version of str.split which returns an iterator

Gregory P. Smith added the comment: Indeed, a bytearray version would require the talked about but not implemented due to complexity (in pep3118) support for locking a buffer from other mutations. best concentrate on bytes then. Do we have a memoryview equivalent for PyUnicode? If not, we

[issue17343] Add a version of str.split which returns an iterator

Brett Cannon added the comment: There is no string view that I know of. Interesting idea, though, thanks to the immutability of strings. Would much have to be different other than boundary checking and __hash__ (and hoping extension authors are changing things in-place)? I say go ahead and

[issue17343] Add a version of str.split which returns an iterator

New submission from Alex Gaynor: str.split returns a list, which is inefficient when you just want to process items one be one. You could emulate this with str.find and tracking indexes manually, but this should really be a builtin behavior. -- messages: 183411 nosy: alex priority:

[issue17343] Add a version of str.split which returns an iterator

Gregory P. Smith added the comment: The bytes (and bytearray?) version of this should generate memoryview's instead of new bytes objects to avoid a copy. While not required, It'd be useful if the implementation of this pre-scanned the data internally so that the length of the generated

[issue17343] Add a version of str.split which returns an iterator

Serhiy Storchaka added the comment: While not required, It'd be useful if the implementation of this pre-scanned the data internally so that the length of the generated sequence was known up front. This could imply an internal bitset of vector of split indices is kept for the life of the