[Zope-Checkins] SVN: Zope/branches/2.12/src/Products/ZCTextIndex/ Merged Zope/branches/tyam-unicodeSplitterPatch 104723:104761

2009-10-13 Thread Takeshi Yamamoto 
Log message for revision 105033:
  Merged Zope/branches/tyam-unicodeSplitterPatch 104723:104761

Changed:
  U   Zope/branches/2.12/src/Products/ZCTextIndex/Lexicon.py
  U   Zope/branches/2.12/src/Products/ZCTextIndex/tests/testLexicon.py

-=-
Modified: Zope/branches/2.12/src/Products/ZCTextIndex/Lexicon.py
===
--- Zope/branches/2.12/src/Products/ZCTextIndex/Lexicon.py  2009-10-13 
09:18:10 UTC (rev 105032)
+++ Zope/branches/2.12/src/Products/ZCTextIndex/Lexicon.py  2009-10-13 
09:23:58 UTC (rev 105033)
@@ -76,7 +76,8 @@
 def termToWordIds(self, text):
 last = _text2list(text)
 for element in self._pipeline:
-last = element.process(last)
+process = getattr(element, process_post_glob, element.process) 
+last = process(last)
 wids = []
 for word in last:
 wids.append(self._wids.get(word, 0))

Modified: Zope/branches/2.12/src/Products/ZCTextIndex/tests/testLexicon.py
===
--- Zope/branches/2.12/src/Products/ZCTextIndex/tests/testLexicon.py
2009-10-13 09:18:10 UTC (rev 105032)
+++ Zope/branches/2.12/src/Products/ZCTextIndex/tests/testLexicon.py
2009-10-13 09:23:58 UTC (rev 105033)
@@ -94,6 +94,28 @@
 wids = lexicon.termToWordIds('boxes')
 self.assertEqual(wids, [0])
 
+def testTermToWordIdsWithProcess_post_glob(self):
+This test is for added process_post_glob
+class AddedSplitter(Splitter):
+def process_post_glob(self, lst):
+assert lst == ['dogs']
+return ['dogs']
+lexicon = Lexicon(AddedSplitter())
+wids = lexicon.sourceToWordIds('cats and dogs')
+wids = lexicon.termToWordIds('dogs')
+self.assertEqual(wids, [3])
+
+def testMissingTermToWordIdsWithProcess_post_glob(self):
+This test is for added process_post_glob
+class AddedSplitter(Splitter):
+def process_post_glob(self, lst):
+assert lst == ['dogs']
+return ['fox']
+lexicon = Lexicon(AddedSplitter())
+wids = lexicon.sourceToWordIds('cats and dogs')
+wids = lexicon.termToWordIds('dogs')
+self.assertEqual(wids, [0])
+
 def testOnePipelineElement(self):
 lexicon = Lexicon(Splitter(), StupidPipelineElement('dogs', 'fish'))
 wids = lexicon.sourceToWordIds('cats and dogs')

___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins

[Zope-Checkins] SVN: Zope/trunk/src/Products/ZCTextIndex/ Merged Zope/branches/tyam-unicodeSplitterPatch 104723:104761

2009-10-13 Thread Takeshi Yamamoto 
Log message for revision 105051:
  Merged Zope/branches/tyam-unicodeSplitterPatch 104723:104761

Changed:
  U   Zope/trunk/src/Products/ZCTextIndex/Lexicon.py
  U   Zope/trunk/src/Products/ZCTextIndex/tests/testLexicon.py

-=-
Modified: Zope/trunk/src/Products/ZCTextIndex/Lexicon.py
===
--- Zope/trunk/src/Products/ZCTextIndex/Lexicon.py  2009-10-13 16:05:59 UTC 
(rev 105050)
+++ Zope/trunk/src/Products/ZCTextIndex/Lexicon.py  2009-10-13 16:28:22 UTC 
(rev 105051)
@@ -76,7 +76,8 @@
 def termToWordIds(self, text):
 last = _text2list(text)
 for element in self._pipeline:
-last = element.process(last)
+process = getattr(element, process_post_glob, element.process) 
+last = process(last)
 wids = []
 for word in last:
 wids.append(self._wids.get(word, 0))

Modified: Zope/trunk/src/Products/ZCTextIndex/tests/testLexicon.py
===
--- Zope/trunk/src/Products/ZCTextIndex/tests/testLexicon.py2009-10-13 
16:05:59 UTC (rev 105050)
+++ Zope/trunk/src/Products/ZCTextIndex/tests/testLexicon.py2009-10-13 
16:28:22 UTC (rev 105051)
@@ -94,6 +94,28 @@
 wids = lexicon.termToWordIds('boxes')
 self.assertEqual(wids, [0])
 
+def testTermToWordIdsWithProcess_post_glob(self):
+This test is for added process_post_glob
+class AddedSplitter(Splitter):
+def process_post_glob(self, lst):
+assert lst == ['dogs']
+return ['dogs']
+lexicon = Lexicon(AddedSplitter())
+wids = lexicon.sourceToWordIds('cats and dogs')
+wids = lexicon.termToWordIds('dogs')
+self.assertEqual(wids, [3])
+
+def testMissingTermToWordIdsWithProcess_post_glob(self):
+This test is for added process_post_glob
+class AddedSplitter(Splitter):
+def process_post_glob(self, lst):
+assert lst == ['dogs']
+return ['fox']
+lexicon = Lexicon(AddedSplitter())
+wids = lexicon.sourceToWordIds('cats and dogs')
+wids = lexicon.termToWordIds('dogs')
+self.assertEqual(wids, [0])
+
 def testOnePipelineElement(self):
 lexicon = Lexicon(Splitter(), StupidPipelineElement('dogs', 'fish'))
 wids = lexicon.sourceToWordIds('cats and dogs')

___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins

[Zope-Checkins] SVN: Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/tests/testLexicon.py Adding two tests for an additional hook on Lexicon.py.

2009-10-02 Thread Takeshi Yamamoto 
Log message for revision 104742:
  Adding two tests for an additional hook on Lexicon.py.

Changed:
  U   
Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/tests/testLexicon.py

-=-
Modified: 
Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/tests/testLexicon.py
===
--- 
Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/tests/testLexicon.py
   2009-10-02 10:25:08 UTC (rev 104741)
+++ 
Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/tests/testLexicon.py
   2009-10-02 10:28:47 UTC (rev 104742)
@@ -94,6 +94,28 @@
 wids = lexicon.termToWordIds('boxes')
 self.assertEqual(wids, [0])
 
+def testTermToWordIdsWithProcess_post_glob(self):
+This test is for added process_post_grob
+class AddedSplitter(Splitter):
+def process_post_glob(self, lst):
+assert lst == ['dogs']
+return ['dogs']
+lexicon = Lexicon(AddedSplitter())
+wids = lexicon.sourceToWordIds('cats and dogs')
+wids = lexicon.termToWordIds('dogs')
+self.assertEqual(wids, [3])
+
+def testMissingTermToWordIdsWithProcess_post_glob(self):
+This test is for added process_post_grob
+class AddedSplitter(Splitter):
+def process_post_glob(self, lst):
+assert lst == ['dogs']
+return ['fox']
+lexicon = Lexicon(AddedSplitter())
+wids = lexicon.sourceToWordIds('cats and dogs')
+wids = lexicon.termToWordIds('dogs')
+self.assertEqual(wids, [0])
+
 def testOnePipelineElement(self):
 lexicon = Lexicon(Splitter(), StupidPipelineElement('dogs', 'fish'))
 wids = lexicon.sourceToWordIds('cats and dogs')

___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins

[Zope-Checkins] SVN: Zope/branches/tyam-unicodeSplitterPatch/lib/python/Products/ZCTextIndex/Lexicon.py added a hook for CJK process

2009-10-01 Thread Takeshi Yamamoto 
Log message for revision 104648:
  added a hook for CJK process

Changed:
  U   
Zope/branches/tyam-unicodeSplitterPatch/lib/python/Products/ZCTextIndex/Lexicon.py

-=-
Modified: 
Zope/branches/tyam-unicodeSplitterPatch/lib/python/Products/ZCTextIndex/Lexicon.py
===
--- 
Zope/branches/tyam-unicodeSplitterPatch/lib/python/Products/ZCTextIndex/Lexicon.py
  2009-09-30 10:00:33 UTC (rev 104647)
+++ 
Zope/branches/tyam-unicodeSplitterPatch/lib/python/Products/ZCTextIndex/Lexicon.py
  2009-09-30 10:20:17 UTC (rev 104648)
@@ -80,7 +80,8 @@
 def termToWordIds(self, text):
 last = _text2list(text)
 for element in self._pipeline:
-last = element.process(last)
+process = getattr(element, process_post_glob, element.process)
+last = process(last)
 wids = []
 for word in last:
 wids.append(self._wids.get(word, 0))

___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins

[Zope-Checkins] SVN: Zope/branches/tyam-unicodeSplitterPatch/ Delete c1-based branch since since final is released.

2009-10-01 Thread Takeshi Yamamoto 
Log message for revision 104722:
  Delete c1-based branch since since final is released.
  

Changed:
  D   Zope/branches/tyam-unicodeSplitterPatch/

-=-
___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins

[Zope-Checkins] SVN: Zope/branches/tyam-unicodeSplitterPatch/ Making a branch from 2.12.0 to make a hook for CJK processing.

2009-10-01 Thread Takeshi Yamamoto 
Log message for revision 104723:
  Making a branch from 2.12.0 to make a hook for CJK processing.
  

Changed:
  A   Zope/branches/tyam-unicodeSplitterPatch/

-=-
___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins

[Zope-Checkins] SVN: Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/Lexicon.py added a hook for CJK processing.

2009-10-01 Thread Takeshi Yamamoto 
Log message for revision 104724:
  added a hook for CJK processing.

Changed:
  U   
Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/Lexicon.py

-=-
Modified: 
Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/Lexicon.py
===
--- Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/Lexicon.py 
2009-10-02 04:15:49 UTC (rev 104723)
+++ Zope/branches/tyam-unicodeSplitterPatch/src/Products/ZCTextIndex/Lexicon.py 
2009-10-02 05:01:33 UTC (rev 104724)
@@ -76,7 +76,8 @@
 def termToWordIds(self, text):
 last = _text2list(text)
 for element in self._pipeline:
-last = element.process(last)
+process = getattr(element, process_post_glob, element.process) 
+last = process(last)
 wids = []
 for word in last:
 wids.append(self._wids.get(word, 0))

___
Zope-Checkins maillist  -  Zope-Checkins@zope.org
https://mail.zope.org/mailman/listinfo/zope-checkins