Making tokenize stream would be really difficult. The reason being that the pattern can be an arbitrary regular expression which is a lot of cases would require to materialize the entire stream anyway.
Hence, I'm suggestion that we define our own string:tokenize function which accepts only separators instead of regular expressions and make this work in a streaming fashion. Would that also do the job? ** Changed in: zorba Milestone: None => 2.2 -- You received this bug notification because you are a member of Zorba Coders, which is the registrant for Zorba. https://bugs.launchpad.net/bugs/898074 Title: fn:tokenize() doesn't stream Status in Zorba - The XQuery Processor: New Bug description: The following query: let $content := file:read-text("huge_file.txt") return tokenize($content, "\s") doesn't stream the result. To manage notifications about this bug go to: https://bugs.launchpad.net/zorba/+bug/898074/+subscriptions -- Mailing list: https://launchpad.net/~zorba-coders Post to : firstname.lastname@example.org Unsubscribe : https://launchpad.net/~zorba-coders More help : https://help.launchpad.net/ListHelp