[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2012-01-04 Thread noreply
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Approved = Merged

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-24 Thread William Candillon
Review: Approve


-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread William Candillon
Is there an example that works with streaming?
I wasn't able to make the following work:
import module namespace http = 
http://www.zorba-xquery.com/modules/http-client;;

declare namespace h = http://expath.org/ns/http-client;;

let $item := http:send-request(h:request 
href=https://stream.twitter.com/1/statuses/sample.json?delimited=length;
  method=GET
  username=wcandillon
  password=wnvbb86g
  override-media-type=text/plain
   /  
   ,   
   
https://stream.twitter.com/1/statuses/sample.json?delimited=length;,
   ()  
)[2]
for $tweet in tokenize($item,a)
return $tweet

Where:
import module namespace http = 
http://www.zorba-xquery.com/modules/http-client;;

declare namespace h = http://expath.org/ns/http-client;;

let $item := http:send-request(h:request 
href=https://stream.twitter.com/1/statuses/sample.json?delimited=length;
  method=GET
  username=wcandillon
  password=wnvbb86g
  override-media-type=text/plain
   /  
   ,   
   
https://stream.twitter.com/1/statuses/sample.json?delimited=length;,
   ()  
)[2]
return $item

streams fine.
What am I missing?
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
As discussed in this thread, only the new tokenize function of the string 
module streams.
Use the following instead

import module namespace s = http://www.zorba-xquery.com/modules/string;;

s:tokenize($item, a)
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread William Candillon
Review: Approve

Works like a charm.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Needs review = Approved

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
Attempt to merge into lp:zorba failed due to conflicts: 

text conflict in ChangeLog
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Approved = Needs review

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
Review: Approve


-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Commit Message changed to:

implementation of string:split function that doesn't accept regular expressions 
but allows for streamable processing of the input (resolves bug #898074)

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread William Candillon
Review: Approve

Works great.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Needs review = Approved

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
Validation queue starting for merge proposal.
Log at: 
http://zorbatest.lambda.nu:8080/remotequeue/tokenize-2011-12-23T20-55-01.864Z/log.html
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Approved = Needs review

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
Validation queue starting for merge proposal.
Log at: 
http://zorbatest.lambda.nu:8080/remotequeue/tokenize-2011-12-23T21-25-57.422Z/log.html
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
The attempt to merge lp:~zorba-coders/zorba/tokenize into lp:zorba failed. 
Below is the output from the failed tests.


CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:273 
(message):
  Validation queue job tokenize-2011-12-23T21-25-57.422Z is finished.  The
  final status was:

  

  1 tests did not succeed - changes not commited.


Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Approved = Needs review

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86829
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
Matthias Brantner has proposed merging lp:~zorba-coders/zorba/tokenize into 
lp:zorba.

Requested reviews:
  Matthias Brantner (matthias-brantner)
  Paul J. Lucas (paul-lucas)
  William Candillon (wcandillon)
Related bugs:
  Bug #898074 in Zorba: fn:tokenize() doesn't stream
  https://bugs.launchpad.net/zorba/+bug/898074

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835

implementation of string:split function that doesn't accept regular expressions 
but allows for streamable processing of the input (resolves bug #898074)
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'ChangeLog'
--- ChangeLog	2011-12-23 19:38:53 +
+++ ChangeLog	2011-12-23 21:56:35 +
@@ -12,6 +12,8 @@
 set multiple times via the c++ api).
   * Fixed bug #905050 (setting and getting the context item type via the c++ api)
   * Added createDayTimeDuration, createYearMonthDuration, createDocumentNode, createCommentNode, createPiNode to api's ItemFactory.
+  * Added split function to the string module that allows for streamable tokenization but doesn't have regular expression
+support.
   * zerr is not predeclared anymore to be http://www.zorba-xquery.com/errors
 
 version 2.1

=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-21 14:40:33 +
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-23 21:56:35 +
@@ -58,7 +58,7 @@
   URI http://www.zorba-xquery.com/modules/reflection;)
 DECLARE_ZORBA_MODULE(FILE schema.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/schema;)
-DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.0
+DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.1
   URI http://www.zorba-xquery.com/modules/string;)
 DECLARE_ZORBA_MODULE(FILE xml.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/xml;)

=== modified file 'modules/com/zorba-xquery/www/modules/string.xq'
--- modules/com/zorba-xquery/www/modules/string.xq	2011-08-03 15:12:40 +
+++ modules/com/zorba-xquery/www/modules/string.xq	2011-12-23 21:56:35 +
@@ -25,7 +25,7 @@
  :)
 module namespace string = http://www.zorba-xquery.com/modules/string;;
 declare namespace ver = http://www.zorba-xquery.com/options/versioning;;
-declare option ver:module-version 2.0;
+declare option ver:module-version 2.1;
 
 (:~
  : This function materializes a streamable string.
@@ -63,3 +63,23 @@
  :
  :)
 declare function string:is-streamable($s as xs:string) as xs:boolean external;
+
+(:~
+ : Returns a sequence of strings constructed by splitting the input wherever the given
+ : separator is found.
+ :
+ : The function is different from fn:tokenize. It doesn't allow
+ : the separator to be a regular expression. This restriction allows for more
+ : performant implementation. Specifically, the function processes
+ : streamable strings as input in a streamable way which is particularly useful
+ : to tokenize huge strings (e.g. if returned by the file module's read-text
+ : function).
+ :
+ : @param $s the input string to split
+ : @param $separator the separator used for splitting the input string $s
+ :
+ : @return a sequence of strings constructed by splitting the input
+ :)
+declare function string:split(
+  $s as xs:string,
+  $separator as xs:string) as xs:string* external;

=== modified file 'src/functions/pregenerated/func_strings.cpp'
--- src/functions/pregenerated/func_strings.cpp	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/func_strings.cpp	2011-12-23 21:56:35 +
@@ -320,6 +320,16 @@
   return new StringIsStreamableIterator(sctx, loc, argv);
 }
 
+PlanIter_t fn_zorba_string_split::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc loc,
+  std::vectorPlanIter_t argv,
+  AnnotationHolder ann) const
+{
+  return new StringSplitIterator(sctx, loc, argv);
+}
+
 void populate_context_strings(static_context* sctx)
 {
   {
@@ -890,6 +900,19 @@
 
   }
 
+
+  {
+
+
+DECL_WITH_KIND(sctx, fn_zorba_string_split,
+(createQName(http://www.zorba-xquery.com/modules/string,,split;), 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_STAR),
+FunctionConsts::FN_ZORBA_STRING_SPLIT_2);
+
+  }
+
 }
 
 

=== modified file 'src/functions/pregenerated/func_strings.h'
--- src/functions/pregenerated/func_strings.h	2011-12-22 14:14:53 +
+++ src/functions/pregenerated/func_strings.h	2011-12-23 21:56:35 +
@@ -481,6 +481,21 @@
 };
 
 
+//fn-zorba-string:split
+class fn_zorba_string_split : public function
+{
+public:
+  fn_zorba_string_split(const signature sig, FunctionConsts::FunctionKind kind)
+: 
+function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+
+
 } //namespace zorba
 
 

=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h	

[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Needs review = Approved

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
Review: Approve


-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Commit Message changed to:

implementation of string:split function that doesn't accept regular expressions 
but allows for streamable processing of the input (resolves bug #898074)

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
There are additional revisions which have not been approved in review. Please 
seek review and approval of these new revisions.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Approved = Needs review

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
Matthias Brantner has proposed merging lp:~zorba-coders/zorba/tokenize into 
lp:zorba.

Requested reviews:
  Paul J. Lucas (paul-lucas)
  Matthias Brantner (matthias-brantner)
  William Candillon (wcandillon)
Related bugs:
  Bug #898074 in Zorba: fn:tokenize() doesn't stream
  https://bugs.launchpad.net/zorba/+bug/898074

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840

implementation of string:split function that doesn't accept regular expressions 
but allows for streamable processing of the input (resolves bug #898074)
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'ChangeLog'
--- ChangeLog	2011-12-23 19:38:53 +
+++ ChangeLog	2011-12-23 22:04:30 +
@@ -12,6 +12,8 @@
 set multiple times via the c++ api).
   * Fixed bug #905050 (setting and getting the context item type via the c++ api)
   * Added createDayTimeDuration, createYearMonthDuration, createDocumentNode, createCommentNode, createPiNode to api's ItemFactory.
+  * Added split function to the string module that allows for streamable tokenization but doesn't have regular expression
+support.
   * zerr is not predeclared anymore to be http://www.zorba-xquery.com/errors
 
 version 2.1

=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-21 14:40:33 +
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-23 22:04:30 +
@@ -58,7 +58,7 @@
   URI http://www.zorba-xquery.com/modules/reflection;)
 DECLARE_ZORBA_MODULE(FILE schema.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/schema;)
-DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.0
+DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.1
   URI http://www.zorba-xquery.com/modules/string;)
 DECLARE_ZORBA_MODULE(FILE xml.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/xml;)

=== modified file 'modules/com/zorba-xquery/www/modules/string.xq'
--- modules/com/zorba-xquery/www/modules/string.xq	2011-08-03 15:12:40 +
+++ modules/com/zorba-xquery/www/modules/string.xq	2011-12-23 22:04:30 +
@@ -25,7 +25,7 @@
  :)
 module namespace string = http://www.zorba-xquery.com/modules/string;;
 declare namespace ver = http://www.zorba-xquery.com/options/versioning;;
-declare option ver:module-version 2.0;
+declare option ver:module-version 2.1;
 
 (:~
  : This function materializes a streamable string.
@@ -63,3 +63,23 @@
  :
  :)
 declare function string:is-streamable($s as xs:string) as xs:boolean external;
+
+(:~
+ : Returns a sequence of strings constructed by splitting the input wherever the given
+ : separator is found.
+ :
+ : The function is different from fn:tokenize. It doesn't allow
+ : the separator to be a regular expression. This restriction allows for more
+ : performant implementation. Specifically, the function processes
+ : streamable strings as input in a streamable way which is particularly useful
+ : to tokenize huge strings (e.g. if returned by the file module's read-text
+ : function).
+ :
+ : @param $s the input string to split
+ : @param $separator the separator used for splitting the input string $s
+ :
+ : @return a sequence of strings constructed by splitting the input
+ :)
+declare function string:split(
+  $s as xs:string,
+  $separator as xs:string) as xs:string* external;

=== modified file 'src/functions/pregenerated/func_strings.cpp'
--- src/functions/pregenerated/func_strings.cpp	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/func_strings.cpp	2011-12-23 22:04:30 +
@@ -320,6 +320,16 @@
   return new StringIsStreamableIterator(sctx, loc, argv);
 }
 
+PlanIter_t fn_zorba_string_split::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc loc,
+  std::vectorPlanIter_t argv,
+  AnnotationHolder ann) const
+{
+  return new StringSplitIterator(sctx, loc, argv);
+}
+
 void populate_context_strings(static_context* sctx)
 {
   {
@@ -890,6 +900,19 @@
 
   }
 
+
+  {
+
+
+DECL_WITH_KIND(sctx, fn_zorba_string_split,
+(createQName(http://www.zorba-xquery.com/modules/string,,split;), 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_STAR),
+FunctionConsts::FN_ZORBA_STRING_SPLIT_2);
+
+  }
+
 }
 
 

=== modified file 'src/functions/pregenerated/func_strings.h'
--- src/functions/pregenerated/func_strings.h	2011-12-22 14:14:53 +
+++ src/functions/pregenerated/func_strings.h	2011-12-23 22:04:30 +
@@ -481,6 +481,21 @@
 };
 
 
+//fn-zorba-string:split
+class fn_zorba_string_split : public function
+{
+public:
+  fn_zorba_string_split(const signature sig, FunctionConsts::FunctionKind kind)
+: 
+function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+
+
 } //namespace zorba
 
 

=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h	

[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Commit Message changed to:

implementation of string:split function that doesn't accept regular expressions 
but allows for streamable processing of the input (resolves bug #898074)

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Needs review = Approved

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
Validation queue starting for merge proposal.
Log at: 
http://zorbatest.lambda.nu:8080/remotequeue/tokenize-2011-12-23T22-04-07.45Z/log.html
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
Voting does not meet specified criteria. Required: Approve  1, Disapprove  1. 
Got: 3 Pending.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread Zorba Build Bot
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Approved = Needs review

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86840
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-23 Thread William Candillon
Review: Approve


-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86835
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Matthias Brantner
Matthias Brantner has proposed merging lp:~zorba-coders/zorba/tokenize into 
lp:zorba.

Requested reviews:
  William Candillon (wcandillon)
  Paul J. Lucas (paul-lucas)
Related bugs:
  Bug #898074 in Zorba: fn:tokenize() doesn't stream
  https://bugs.launchpad.net/zorba/+bug/898074

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626

implementation of string:tokenize function that doesn't accept regular 
expressions but allows for streamable processing of the input (resolves bug 
#898074)
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'ChangeLog'
--- ChangeLog	2011-12-21 14:40:33 +
+++ ChangeLog	2011-12-21 21:44:29 +
@@ -9,6 +9,8 @@
 set multiple times via the c++ api).
   * Fixed bug #905050 (setting and getting the context item type via the c++ api)
   * Added createDayTimeDuration, createYearMonthDuration, createDocumentNode, createCommentNode, createPiNode to api's ItemFactory.
+  * Added tokenize function to the string module that allows for streamable tokenization but doesn't have regular expression
+support.
 
 version 2.1
 

=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-21 14:40:33 +
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-21 21:44:29 +
@@ -58,7 +58,7 @@
   URI http://www.zorba-xquery.com/modules/reflection;)
 DECLARE_ZORBA_MODULE(FILE schema.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/schema;)
-DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.0
+DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.1
   URI http://www.zorba-xquery.com/modules/string;)
 DECLARE_ZORBA_MODULE(FILE xml.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/xml;)

=== modified file 'modules/com/zorba-xquery/www/modules/string.xq'
--- modules/com/zorba-xquery/www/modules/string.xq	2011-08-03 15:12:40 +
+++ modules/com/zorba-xquery/www/modules/string.xq	2011-12-21 21:44:29 +
@@ -25,7 +25,7 @@
  :)
 module namespace string = http://www.zorba-xquery.com/modules/string;;
 declare namespace ver = http://www.zorba-xquery.com/options/versioning;;
-declare option ver:module-version 2.0;
+declare option ver:module-version 2.1;
 
 (:~
  : This function materializes a streamable string.
@@ -63,3 +63,23 @@
  :
  :)
 declare function string:is-streamable($s as xs:string) as xs:boolean external;
+
+(:~
+ : Returns a sequence of strings constructed by splitting the input wherever the given
+ : separator is found.
+ :
+ : The function is different from fn:tokenize. It doesn't allow
+ : the separator to be a regular expression. This restriction allows for more
+ : performant implementation. Specifically, the function processes
+ : streamable strings as input in a streamable way which is particularly useful
+ : to tokenize huge strings (e.g. if returned by the file module's read-text
+ : function).
+ :
+ : @param $s the input string to tokenize
+ : @param $separator the separator used for splitting the input string $s
+ :
+ : @return a sequence of strings constructed by splitting the input
+ :)
+declare function string:tokenize(
+  $s as xs:string,
+  $separator as xs:string) as xs:string* external;

=== modified file 'src/functions/pregenerated/func_strings.cpp'
--- src/functions/pregenerated/func_strings.cpp	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/func_strings.cpp	2011-12-21 21:44:29 +
@@ -320,6 +320,16 @@
   return new StringIsStreamableIterator(sctx, loc, argv);
 }
 
+PlanIter_t fn_zorba_string_tokenize::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc loc,
+  std::vectorPlanIter_t argv,
+  AnnotationHolder ann) const
+{
+  return new StringTokenizeIterator(sctx, loc, argv);
+}
+
 void populate_context_strings(static_context* sctx)
 {
   {
@@ -890,6 +900,19 @@
 
   }
 
+
+  {
+
+
+DECL_WITH_KIND(sctx, fn_zorba_string_tokenize,
+(createQName(http://www.zorba-xquery.com/modules/string,,tokenize;), 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_STAR),
+FunctionConsts::FN_ZORBA_STRING_TOKENIZE_2);
+
+  }
+
 }
 
 

=== modified file 'src/functions/pregenerated/func_strings.h'
--- src/functions/pregenerated/func_strings.h	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/func_strings.h	2011-12-21 21:44:29 +
@@ -423,6 +423,19 @@
 };
 
 
+//fn-zorba-string:tokenize
+class fn_zorba_string_tokenize : public function
+{
+public:
+  fn_zorba_string_tokenize(const signature sig, FunctionConsts::FunctionKind kind)
+: function(sig, kind) {
+
+}
+
+  CODEGEN_DECL();
+};
+
+
 } //namespace zorba
 
 

=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/function_enum.h	2011-12-21 21:44:29 +
@@ 

[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Matthias Brantner
The proposal to merge lp:~zorba-coders/zorba/tokenize into lp:zorba has been 
updated.

Status: Needs review = Approved

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Zorba Build Bot
Validation queue starting for merge proposal.
Log at: 
http://zorbatest.lambda.nu:8080/remotequeue/tokenize-2011-12-21T21-46-05.289Z/log.html
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Matthias Brantner
Once you finished the implementation of the transcoding stream buffer, I don't 
even want to do this check anymore. This must not happen with the stream buffer.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Paul J. Lucas
 Once you finished the implementation of the transcoding stream buffer, I don't
 even want to do this check anymore. This must not happen with the stream
 buffer.

I don't understand how it must not happen. It can always happen. However, I 
think you're saying that you assume the check will happen in the transcoder.  
While it will be doing checks, bad input can still happen.

In the mean time, using an assert() is still too Draconian.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86626
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


[Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Matthias Brantner
Matthias Brantner has proposed merging lp:~zorba-coders/zorba/tokenize into 
lp:zorba.

Requested reviews:
  Paul J. Lucas (paul-lucas)
  William Candillon (wcandillon)
Related bugs:
  Bug #898074 in Zorba: fn:tokenize() doesn't stream
  https://bugs.launchpad.net/zorba/+bug/898074

For more details, see:
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647

implementation of string:tokenize function that doesn't accept regular 
expressions but allows for streamable processing of the input (resolves bug 
#898074)
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.
=== modified file 'ChangeLog'
--- ChangeLog	2011-12-21 14:40:33 +
+++ ChangeLog	2011-12-22 04:03:27 +
@@ -9,6 +9,8 @@
 set multiple times via the c++ api).
   * Fixed bug #905050 (setting and getting the context item type via the c++ api)
   * Added createDayTimeDuration, createYearMonthDuration, createDocumentNode, createCommentNode, createPiNode to api's ItemFactory.
+  * Added tokenize function to the string module that allows for streamable tokenization but doesn't have regular expression
+support.
 
 version 2.1
 

=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-21 14:40:33 +
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt	2011-12-22 04:03:27 +
@@ -58,7 +58,7 @@
   URI http://www.zorba-xquery.com/modules/reflection;)
 DECLARE_ZORBA_MODULE(FILE schema.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/schema;)
-DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.0
+DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.1
   URI http://www.zorba-xquery.com/modules/string;)
 DECLARE_ZORBA_MODULE(FILE xml.xq VERSION 2.0
   URI http://www.zorba-xquery.com/modules/xml;)

=== modified file 'modules/com/zorba-xquery/www/modules/string.xq'
--- modules/com/zorba-xquery/www/modules/string.xq	2011-08-03 15:12:40 +
+++ modules/com/zorba-xquery/www/modules/string.xq	2011-12-22 04:03:27 +
@@ -25,7 +25,7 @@
  :)
 module namespace string = http://www.zorba-xquery.com/modules/string;;
 declare namespace ver = http://www.zorba-xquery.com/options/versioning;;
-declare option ver:module-version 2.0;
+declare option ver:module-version 2.1;
 
 (:~
  : This function materializes a streamable string.
@@ -63,3 +63,23 @@
  :
  :)
 declare function string:is-streamable($s as xs:string) as xs:boolean external;
+
+(:~
+ : Returns a sequence of strings constructed by splitting the input wherever the given
+ : separator is found.
+ :
+ : The function is different from fn:tokenize. It doesn't allow
+ : the separator to be a regular expression. This restriction allows for more
+ : performant implementation. Specifically, the function processes
+ : streamable strings as input in a streamable way which is particularly useful
+ : to tokenize huge strings (e.g. if returned by the file module's read-text
+ : function).
+ :
+ : @param $s the input string to tokenize
+ : @param $separator the separator used for splitting the input string $s
+ :
+ : @return a sequence of strings constructed by splitting the input
+ :)
+declare function string:tokenize(
+  $s as xs:string,
+  $separator as xs:string) as xs:string* external;

=== modified file 'src/functions/pregenerated/func_strings.cpp'
--- src/functions/pregenerated/func_strings.cpp	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/func_strings.cpp	2011-12-22 04:03:27 +
@@ -320,6 +320,16 @@
   return new StringIsStreamableIterator(sctx, loc, argv);
 }
 
+PlanIter_t fn_zorba_string_tokenize::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc loc,
+  std::vectorPlanIter_t argv,
+  AnnotationHolder ann) const
+{
+  return new StringTokenizeIterator(sctx, loc, argv);
+}
+
 void populate_context_strings(static_context* sctx)
 {
   {
@@ -890,6 +900,19 @@
 
   }
 
+
+  {
+
+
+DECL_WITH_KIND(sctx, fn_zorba_string_tokenize,
+(createQName(http://www.zorba-xquery.com/modules/string,,tokenize;), 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+GENV_TYPESYSTEM.STRING_TYPE_STAR),
+FunctionConsts::FN_ZORBA_STRING_TOKENIZE_2);
+
+  }
+
 }
 
 

=== modified file 'src/functions/pregenerated/func_strings.h'
--- src/functions/pregenerated/func_strings.h	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/func_strings.h	2011-12-22 04:03:27 +
@@ -423,6 +423,19 @@
 };
 
 
+//fn-zorba-string:tokenize
+class fn_zorba_string_tokenize : public function
+{
+public:
+  fn_zorba_string_tokenize(const signature sig, FunctionConsts::FunctionKind kind)
+: function(sig, kind) {
+
+}
+
+  CODEGEN_DECL();
+};
+
+
 } //namespace zorba
 
 

=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h	2011-12-21 14:40:33 +
+++ src/functions/pregenerated/function_enum.h	2011-12-22 04:03:27 +
@@ 

Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Matthias Brantner
I have replaced the assertion with a graceful error.
-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp


Re: [Zorba-coders] [Merge] lp:~zorba-coders/zorba/tokenize into lp:zorba

2011-12-21 Thread Paul J. Lucas
Review: Approve


-- 
https://code.launchpad.net/~zorba-coders/zorba/tokenize/+merge/86647
Your team Zorba Coders is subscribed to branch lp:zorba.

-- 
Mailing list: https://launchpad.net/~zorba-coders
Post to : zorba-coders@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zorba-coders
More help   : https://help.launchpad.net/ListHelp