Andi,
thanks for the fixes and cleanup! I've updated from SVN and wrote the unit
test. Runs withtout errors (Tracebacks etc.) now, however there may be some
slight type mismatch issue still: the comparison of objects retrieved from
JArray and ArrayList respectively with the objects from the initial JavaList
does not match in case of the ArrayList (test yields 4 failures - see below).
That's with my newly built JCC2.12/PyLucene3.5 from branch_3x.
As far as I understand
elem0 = jArray[0] -> yields python object
elem0 = arrayList.get(0) -> yields wrapped Java object
Not sure if that's intended. In that case the test should be fixed ,-)
Attached is a patch with the test - this time with .txt extension - hope it
get's through...
regards
Thomas
======================================================================
FAIL: test_ArrayList (__main__.Test_CollectionsBoolList)
create ArrayList in JVM (from the JavaSet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_Collections.py", line 208, in test_ArrayList
elem0,type(elem0), listElem0, type(listElem0)))
AssertionError: should be equal: true (<type 'Object'>) <-> True (<type 'bool'>
======================================================================
FAIL: test_ArrayList (__main__.Test_CollectionsFloatList)
create ArrayList in JVM (from the JavaSet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_Collections.py", line 208, in test_ArrayList
elem0,type(elem0), listElem0, type(listElem0)))
AssertionError: should be equal: 1.5 (<type 'Object'>) <-> 1.5 (<type 'float'>)
======================================================================
FAIL: test_ArrayList (__main__.Test_CollectionsListBase)
create ArrayList in JVM (from the JavaSet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_Collections.py", line 208, in test_ArrayList
elem0,type(elem0), listElem0, type(listElem0)))
AssertionError: should be equal: 0 (<type 'Object'>) <-> 0 (<type 'int'>)
======================================================================
FAIL: test_ArrayList (__main__.Test_CollectionsStringList)
create ArrayList in JVM (from the JavaSet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_Collections.py", line 208, in test_ArrayList
elem0,type(elem0), listElem0, type(listElem0)))
AssertionError: should be equal: a (<type 'Object'>) <-> a (<type 'str'>)
----------------------------------------------------------------------
Ran 42 tests in 0.031s
FAILED (failures=4)
> -----Ursprüngliche Nachricht-----
> Von: Andi Vajda [mailto:va...@apache.org]
> Gesendet: Donnerstag, 3. Mai 2012 01:00
> An: pylucene-dev@lucene.apache.org
> Betreff: Re: AW: AW: AW: PyLucene use JCC shared object by default
>
>
> Hi Thomas,
>
> On Wed, 2 May 2012, Thomas Koch wrote:
>
> > could you download the patch from the link?
>
> Yes, I got your patch just fine.
>
> I fixed a few bugs today having to do with converting sequences to JArray
> and added support for auto-boxing primitive types when converting a
> sequence to an object JArray. Now your collections-demo.py all works fine !
>
> With these fixes the Python toArray() methods can return a Python
> sequence object directly, there is no need to do the JArray conversion in
> Python anymore.
>
> I simplified the collections.py file a bit to reflect the fixes and all
> changes,
> including the PythonList/PythonListIterator code is now checked in.
>
> Could you please convert collections-demo.py into a proper unit test module
> like the unit tests in pylucene/test so that it gets integrated into the test
> suite ?
>
> Thanks !
>
> Andi..
>
> >
> > Just one more thing ... in the initial implementation of PythonList I did
> > the
> toArray() method in Python and the toArray(Object[]) method in Java - just
> as was done for the PythonSet:
> >
> > + public native List subList(int fromIndex, int toIndex);
> > + public native Object[] toArray();
> > +
> > + public Object[] toArray(Object[] a)
> > + {
> > + Object[] array = toArray();
> > +
> > + if (a.length < array.length)
> > + a = (Object[])
> Array.newInstance(a.getClass().getComponentType(),
> > + array.length);
> > +
> > + System.arraycopy(array, 0, a, 0, array.length);
> > +
> > + return a;
> > + }
> >
> > (from patch of Feb 22nd I sent to you)
> >
> >> From the current patch you can see that the latter part is missing now -
> the toArray(Object[]) method is now done in Python as well, i.e. it simply
> calls toArray():
> >
> >
> ==========================================================
> =========
> > --- java/org/apache/pylucene/util/PythonSet.java (revision 1332162)
> > +++ java/org/apache/pylucene/util/PythonSet.java (working copy)
> > @@ -62,14 +62,6 @@
> >
> > public Object[] toArray(Object[] a)
> > {
> > - Object[] array = toArray();
> > -
> > - if (a.length < array.length)
> > - a = (Object[])
> > Array.newInstance(a.getClass().getComponentType(),
> > - array.length);
> > -
> > - System.arraycopy(array, 0, a, 0, array.length);
> > -
> > - return a;
> > + return toArray();
> > }
> > }
> >
> > As far as I remember that was part of your changes in between (I probably
> never touched PythonSet). Anyway I could imagine that this is related to the
> current problem.
> >
> > However, the ArrayList never calls the 2nd toArray method. The ArrayList
> constructor actually triggers the "simple" toArray method:
> >
> > /**
> > * Constructs a list containing the elements of the specified
> > * collection, in the order they are returned by the collection's
> > * iterator.
> > *
> > * @param c the collection whose elements are to be placed into this list
> > * @throws NullPointerException if the specified collection is null
> > */
> > public ArrayList(Collection<? extends E> c) {
> > elementData = c.toArray();
> > size = elementData.length;
> > // c.toArray might (incorrectly) not return Object[] (see 6260652)
> > if (elementData.getClass() != Object[].class)
> > elementData = Arrays.copyOf(elementData, size, Object[].class);
> > }
> >
> > The ArrayList source code is attached (from OpenJDK6 sources).
> >
> > So maybe that's the wrong path ... Anyhow I feel the mapping of
> toArray(Object[]) to toArray() does not fully comply with the Java API
> description:
> > http://docs.oracle.com/javase/6/docs/api/java/util/List.html#toArray(T
> > [])
> >
> >
> > Hope that helps...
> >
> >
> > Regards,
> > Thomas
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Andi Vajda [mailto:va...@apache.org]
> >> Gesendet: Montag, 30. April 2012 19:32
> >> An: pylucene-dev@lucene.apache.org
> >> Betreff: Re: AW: AW: PyLucene use JCC shared object by default
> >>
> >>
> >> On Mon, 30 Apr 2012, Thomas Koch wrote:
> >>
> >>> Dear Andi, I again had a look at the patch I submitted recently and
> >>> would like to get back to it. An updated version of the patch is
> >>> attached to this email - the patch is against the branch_3x repo
> >>> http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x
> >>
> >> Oh, and there is no attachment in your email. Maybe it got eaten up
> >> by some mail server. Please, make sure it's of a text mimetype or
> >> mail it to me directly.
> >>
> >> Thanks !
> >>
> >> Andi..
> >>
> >>>
> >>> The patch mainly
> >>> - adds two java classes: PythonList, PythonListIterator
> >>> - adds according Python classes (JavaListIterator and JavaList in
> >> collections.py)
> >>>
> >>> Purpose:
> >>> - provide a Java-based List implementation in JCC/PyLucene (similar
> >>> to existing PythonSet/JavaSet)
> >>> - allow to pass python lists via Java Collections into PyLucene
> >>>
> >>> Let's try summarize shortly: PythonSet /JavaSet was already
> >>> existing, but
> >> nothing similar for Lists. I made an implementation of PythonList
> >> /JavaList and with your help this is now basically working. Except of
> >> an open issue that affects both JavaSet and JavaList: initialization
> >> of an ArrayList with a JavaSet (or JavaList) may cause trouble.
> >>>
> >>> As you said: "There is a bug somewhere with constructing an
> >>> ArrayList from
> >> a python collection like JavaSet or JavaList."
> >>>
> >>> I tried to change the toArray() method as you suggested, but that
> >>> didn't
> >> help. As far as I understood, there are two options to box python
> >> values into a typed JArray:
> >>>
> >>> 1) use the object based JArray class and box python values by
> >>> wrapping
> >> them with the corresponding Java object (e.g. type<int> ->
> lucene.Integer):
> >>>
> >>>>>> x =
> >>>>>> lucene.JArray('object')([lucene.Boolean(True),lucene.Boolean(Fals
> >>>>>> e)
> >>>>>> ])
> >>> JArray<object>[<Object: true>, <Object: false>]
> >>>>>> type(x[0])
> >>> <type 'Object'>
> >>>
> >>> 2) use the correct array type (int, float, etc.) and pass the list
> >>> of Python
> >> elements or literals) to the JArray constructur, e.g.
> >>>
> >>>>>> y = lucene.JArray('bool')([True,False])
> >>> JArray<bool>[True, False]
> >>>>>> type(y[0])
> >>> <type 'bool'>
> >>>
> >>> I tried both of them (see _pyList2JArray methods in collections.py)
> >>> but
> >> none of them did the trick. Actually the 'empty objects in ArrayList'
> >> problem remains when handling with strings (the ArrayList object that
> >> is initialized with a JavaSet or JavaList of string items will have a
> >> number of objects as the original JavaSet/JavaList, but all objects
> >> are the same - ooks like an array of empty objects). Furthermore another
> issue with integer lists comes into play:
> >> here the initialization of ArrayList with the Collection fails with
> >> a Java stacktrace (lucene.JavaError: org.apache.jcc.PythonException).
> >>>
> >>> The most simple test case is as follows:
> >>>
> >>> --%< --
> >>> import lucene
> >>> lucene.initVM()
> >>> from lucene.collections import JavaList
> >>>
> >>> # using strings: the ArrayList is created, but initialized with
> >>> empty objects jl = JavaList(['a','b']) al = lucene.ArrayList(jl)
> >>> assert (not al.get(0).equals(al.get(1))), "unique values"
> >>>
> >>> # using ints: the ArrayList is not created, but an error occurs instead:
> >>> # Java stacktrace: org.apache.jcc.PythonException: ('while calling
> >>> toArray') jl = JavaList(range(3)) al = lucene.ArrayList(jl) --%< --
> >>>
> >>> I currently feel like having to stab around in the dark to find out
> >>> what's going on here and would welcome any suggestions. Needs some
> >> JCC
> >>> expert I guess ,-)
> >>>
> >>> Of course we can leave the patch out - but still there's the same
> >>> issue with
> >> JavaSet.
> >>>
> >>>
> >>> kind regards
> >>>
> >>> Thomas
> >>> --
> >>> OrbiTeam Software GmbH & Co. KG, Germany http://www.orbiteam.de
> >>>
> >>>
> >>>> -----Ursprüngliche Nachricht-----
> >>>> Von: Andi Vajda [mailto:va...@apache.org]
> >>>> Gesendet: Mittwoch, 18. April 2012 20:37
> >>>> An: pylucene-dev@lucene.apache.org
> >>>> Betreff: Re: AW: PyLucene use JCC shared object by default
> >>>>
> >>>>
> >>>> Hi Thomas,
> >>>> ...
> >>>> Lucene 3.6 just got released a few days ago. Apart from your patch,
> >>>> the PyLucene 3.6 release is ready. I'm about to go offline (email
> >>>> only) for a
> >> week.
> >>>> Let's revisit this patch then (first week of May). It's not
> >>>> blocking the
> >> release
> >>>> right now as, even if I sent out a release candidate for a vote,
> >>>> the three business days required for this would take this into the time
> I'm away.
> >>>> ...
> >>>> Andi..
> >>>
> >>>
> >
Index: test/test_Collections.py
===================================================================
--- test/test_Collections.py (revision 0)
+++ test/test_Collections.py (revision 0)
@@ -0,0 +1,245 @@
+# ====================================================================
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ====================================================================
+
+import unittest
+import lucene
+from lucene.collections import JavaSet, JavaList
+
+
+class Test_CollectionsSetBase(unittest.TestCase):
+ """base test case for JavaSet (uses integers)
+ subclass may redefine method 'createTestSet'
+ """
+
+ def createTestSet(self):
+ """creates the test set for this test case
+ """
+ return set(range(9))
+
+ def setUp(self):
+ self.testSet = self.createTestSet()
+ self.javaSet = JavaSet(self.testSet)
+ # print "created testSet: %s JavaSet %s" % (self.testSet,self.javaSet)
+
+ def tearDown(self):
+ del self.testSet
+ del self.javaSet
+
+ def test_Contains(self):
+ elem0 = list(self.testSet)[0]
+ self.assertTrue(self.javaSet.contains(elem0))
+
+ def test_Size(self):
+ self.assertEqual(len(self.testSet), self.javaSet.size())
+
+ def test_Add(self):
+ """must fail to add an existing element
+ """
+ elem0 = list(self.testSet)[0]
+ self.assertFalse(self.javaSet.add(elem0))
+ self.assertEqual(len(self.testSet),
+ self.javaSet.size(),
+ "size has not changed")
+
+ def test_HashSet(self):
+ """create HashSet in JVM (from the JavaSet)
+ """
+ hashSet = lucene.HashSet(self.javaSet)
+ # print "created HashSet:", hashSet, type(hashSet)
+ self.assertEqual(self.javaSet.size(),
+ hashSet.size(),
+ "HashSet has same size")
+ elem0 = list(self.testSet)[0]
+ self.assertTrue(hashSet.contains(elem0))
+
+ def test_JArray(self):
+ """create JArray in JVM (from the JavaSet)
+ """
+ jArray = self.javaSet.toArray()
+ # print "created JArray:", jArray, type(jArray)
+ self.assertEqual(self.javaSet.size(),len(jArray),
+ "JArray has same size")
+ elem0 = jArray[0]
+ elem1 = jArray[1]
+ # print "JArray: first element: %s (%s)" % (elem0,type(elem0))
+ # print "JArray: second element: %s (%s)"% (elem1,type(elem1))
+
+ def test_ArrayList(self):
+ """create ArrayList in JVM (from the JavaSet)
+ """
+ arrayList = lucene.ArrayList(self.javaSet)
+ # print "created ArrayList:", arrayList, type(arrayList)
+ self.assertEqual(self.javaSet.size(), arrayList.size(),
+ "ArrayList has same size")
+ elem0 = arrayList.get(0)
+ elem1 = arrayList.get(1)
+ # print "ArrayList: first element: %s (%s) indexOf=%d" %
(elem0,type(elem0), arrayList.indexOf(elem0))
+ # print "ArrayList: second element: %s (%s) indexOf=%d" %
(elem1,type(elem1), arrayList.indexOf(elem1))
+ self.assertFalse(elem0.equals(elem1),
+ "ArrayList: first element must NOT equal second
element")
+ self.assertNotEqual(elem0, elem1,
+ "ArrayList: first element must NOT equal second
element")
+
+
+
+
+class Test_CollectionsStringSet(Test_CollectionsSetBase):
+
+ def createTestSet(self):
+ return set(['a','b','c'])
+
+
+class Test_CollectionsFloatSet(Test_CollectionsSetBase):
+
+ def createTestSet(self):
+ return set([1.5, 4.5, -0.5])
+
+
+class Test_CollectionsBoolList(Test_CollectionsSetBase):
+
+ def createTestSet(self):
+ return set([True,False])
+
+
+
+
+class Test_CollectionsListBase(unittest.TestCase):
+ """base test case for JavaList (uses integers)
+ subclass may redefine method 'createTestList'
+ """
+
+ def createTestList(self):
+ """creates the test list for this test case
+ """
+ return range(9)
+
+ def setUp(self):
+ self.testList = self.createTestList()
+ self.javaList = JavaList(self.testList)
+ # print "created testList: %s JavaList %s" %
(self.testList,self.javaList)
+
+ def tearDown(self):
+ del self.testList
+ del self.javaList
+
+ def test_Contains(self):
+ elem0 = self.testList[0]
+ self.assertTrue(self.javaList.contains(elem0))
+
+ def test_Size(self):
+ self.assertEqual(len(self.testList), self.javaList.size())
+
+ def test_Pos(self):
+ """elements must have same position
+ """
+ elem0 = self.testList[0]
+ elem1 = self.testList[1]
+ pos0 = self.javaList.indexOf(elem0)
+ pos1 = self.javaList.indexOf(elem1)
+ self.assertEqual(pos0, 0, "indexOf first element")
+ self.assertEqual(pos1, 1, "indexOf second element")
+
+ def test_HashSet(self):
+ """create HashSet in JVM (from the JavaSet)
+ """
+ hashSet = lucene.HashSet(self.javaList)
+ # print "created HashSet:", hashSet, type(hashSet)
+ self.assertEqual(self.javaList.size(),
+ hashSet.size(),
+ "HashSet has same size")
+ elem0 = self.testList[0]
+ self.assertTrue(hashSet.contains(elem0))
+
+ def test_JArray(self):
+ """create JArray in JVM (from the JavaSet)
+ """
+ jArray = self.javaList.toArray()
+ # print "created JArray:", jArray, type(jArray)
+ self.assertEqual(self.javaList.size(),len(jArray),
+ "JArray has same size")
+ elem0 = jArray[0]
+ elem1 = jArray[1]
+ listElem0 = self.testList[0]
+ listElem1 = self.testList[1]
+
+ self.assertEqual(elem0, listElem0,
+ "should be equal: %s (%s) <-> %s (%s)" % (
+ elem0,type(elem0), listElem0, type(listElem0)))
+
+ self.assertEqual(elem1, listElem1,
+ "should be equal: %s (%s) <-> %s (%s)" % (
+ elem1,type(elem1), listElem1, type(listElem1)))
+
+ self.assertEqual(type(elem0), type(listElem0),
+ "should have same type: %s <-> %s" % (
+ type(elem0), type(listElem0)))
+
+ self.assertNotEqual(elem0, elem1,
+ "JArray: first element must NOT equal second
element")
+
+
+ def test_ArrayList(self):
+ """create ArrayList in JVM (from the JavaSet)
+ """
+ arrayList = lucene.ArrayList(self.javaList)
+ # print "created ArrayList:", arrayList, type(arrayList)
+ self.assertEqual(self.javaList.size(), arrayList.size(),
+ "ArrayList has same size")
+ elem0 = arrayList.get(0)
+ elem1 = arrayList.get(1)
+ self.assertEqual(0, arrayList.indexOf(elem0), "same index position")
+ self.assertEqual(1, arrayList.indexOf(elem1), "same index position")
+ listElem0 = self.testList[0]
+ listElem1 = self.testList[1]
+
+ self.assertEqual(elem0, listElem0,
+ "should be equal: %s (%s) <-> %s (%s)" % (
+ elem0,type(elem0), listElem0, type(listElem0)))
+
+ self.assertEqual(elem1, listElem1,
+ "should be equal: %s (%s) <-> %s (%s)" % (
+ elem1,type(elem1), listElem1, type(listElem1)))
+
+ self.assertEqual(type(elem0), type(listElem0),
+ "should have same type: %s <-> %s" % (
+ type(elem0), type(listElem0)))
+
+ self.assertNotEqual(elem0, elem1,
+ "ArrayList: first element must NOT equal second
element")
+
+
+
+
+class Test_CollectionsStringList(Test_CollectionsListBase):
+
+ def createTestList(self):
+ return ['a','b','c']
+
+
+class Test_CollectionsFloatList(Test_CollectionsListBase):
+
+ def createTestList(self):
+ return [1.5, 4.5, -0.5]
+
+
+class Test_CollectionsBoolList(Test_CollectionsListBase):
+
+ def createTestList(self):
+ return [True,False]
+
+
+if __name__ == '__main__':
+ import lucene
+ lucene.initVM()
+ unittest.main()
