> However, I wonder if it possible for you to turn whatever made you
> notice it into a test-case? It is possible that this bug exists on the
> LISP you are using and not mine, but in any case we should use each such
> bug as an opportunity to expand the suite of automated tests.
It was hard, but I managed to add one. It won't work on all implementations
since the spec doesn't guarantee a way to generate a non-simple string.
It's very plain to see that the old code was wrong. In a branch where VAR
was guaranteed to be a STRING it used SCHAR, which is only defined for
SIMPLE-STRING (a subtype of STRING). I wonder how it could get away
with that for so long.
> As you will see, we have a tiny number of tests around unicode issues,
> which arose out of my using some Esperanto stuff. However, I am sure
> our tests are not sufficient.
It would appear so :)
> The ideal thing would be to write a test, see that it fails without our
> patch, and then that it and all the other relevant tests are green with
> the patch in place.
I don't know how other Lisps handle this string stuff, but on SBCL (1.0.14,
probably earlier versions, too) the new test will fail. Note that if you
have a Lisp where the test doesn't fail (despite incorrect use of SCHAR),
you won't have any problems with the erroneous code, either.
Now, attached are three patch files containing the following:
* the new test case
* the fix
* a refactoring of the UTF{16,32}LE serializers
If you want multiple revisions in one file in the future, please say so.
Leslie
--
My personal blog: http://blog.viridian-project.de/
New patches:
[add test for STRING types (as opposed to SIMPLE-STRING types)
[EMAIL PROTECTED] {
hunk ./tests/testserializer.lisp 209
+ (in-out-equal (make-array 3 :initial-element #\â :fill-pointer 2
+ :element-type 'character)) ; test non-simple Unicode string, where supported
}
Context:
[Alex's patch for 8.3
[EMAIL PROTECTED]
I entered here the patch from Alex of 2088/02/16
which apparently makes this compatible with Postgres 8.3.
It is green for me on all tests on Posgres 8.1, so
I am committing it.
]
[mtype change in dcm
[EMAIL PROTECTED]
[moved cache-instance into initial-persistent-setup
[EMAIL PROTECTED]
because it was bypassed by recreate-instance otherwise
]
[accessor name in tests change
[EMAIL PROTECTED]
[db-postmodern: pm-btree initialization fixed
[EMAIL PROTECTED]
[recreate-instance stuff improved
[EMAIL PROTECTED]
[db-postmodern: removed specialized map-index
[EMAIL PROTECTED]
because pure cursor version works fine and is more robust
]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>*-20071124163701]
[db-postmodern fix map-index optimization bug
Henrik Hjelte <[EMAIL PROTECTED]>**20080104151644]
[db-postmodern: cursors re-implemented
[EMAIL PROTECTED]
[controller-doc-improvement
[EMAIL PROTECTED]
[tutorial
[EMAIL PROTECTED]
[non-keyword-accessors
[EMAIL PROTECTED]
allows lispworks to run tests.
]
[function-call-key-form
[EMAIL PROTECTED]
[documentation type fix
[EMAIL PROTECTED]
[Fix the use of internal symbol of sb-kernel in memutils
Leonardo Varuzza <[EMAIL PROTECTED]>**20071230000120
memutil.lisp use the functions sb-kernel::copy-*-from-system-area, which
aren't exported in the latest version of sbcl.
Fix it adding the :: when appropriate
]
[db-bdb bugfix: when bdb key comparison compared only the first half of utf16 strings
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[db-postmodern: optimized map-index for -by-value case
[EMAIL PROTECTED]
[db-postmodern: optimized form-slot-key for persistent-slot-reader
[EMAIL PROTECTED]
it uses SBCL internal function now, for other implementation it's less optimized.
]
[db-postmodern: small example update
[EMAIL PROTECTED]
[added sh script for flushing logs sample
[EMAIL PROTECTED]
[db-postmodern removed possiblity of using NIL as a key in btrees
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163828]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163701]
[removed a little compiler warning (typo)
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151929]
[remove kind-hints parameter from add-index
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151046
Probably a coming feature from Ian, but
right now it breaks the generic function add-index
and thus postmodern, so I removed it for now.
]
[Ensure set-db-synch is defined before pset is loaded
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[Fix to from-end traversal of new map-index
[EMAIL PROTECTED]
[New map-index implementation
[EMAIL PROTECTED]
[Cheaper get-instance-by-value
[EMAIL PROTECTED]
[TAG ELEPHANT-0-9-1
[EMAIL PROTECTED]
Patch bundle hash:
51a929ea7a81660eaadc54de91aa69886fb51da6
New patches:
[Fix Unicode serializers SERIALIZE-TO-UTF16LE and SERIALIZE-TO-UTF32LE; they attempted to use SCHAR on STRING types, whereas it is defined for SIMPLE-STRING only.
[EMAIL PROTECTED] {
hunk ./src/elephant/unicode2.lisp 131
- (let ((code (char-code (schar string i))))
+ (let ((code (char-code (char string i))))
hunk ./src/elephant/unicode2.lisp 171
- (let ((code (char-code (schar string i))))
+ (let ((code (char-code (char string i))))
}
Context:
[Alex's patch for 8.3
[EMAIL PROTECTED]
I entered here the patch from Alex of 2088/02/16
which apparently makes this compatible with Postgres 8.3.
It is green for me on all tests on Posgres 8.1, so
I am committing it.
]
[mtype change in dcm
[EMAIL PROTECTED]
[moved cache-instance into initial-persistent-setup
[EMAIL PROTECTED]
because it was bypassed by recreate-instance otherwise
]
[accessor name in tests change
[EMAIL PROTECTED]
[db-postmodern: pm-btree initialization fixed
[EMAIL PROTECTED]
[recreate-instance stuff improved
[EMAIL PROTECTED]
[db-postmodern: removed specialized map-index
[EMAIL PROTECTED]
because pure cursor version works fine and is more robust
]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>*-20071124163701]
[db-postmodern fix map-index optimization bug
Henrik Hjelte <[EMAIL PROTECTED]>**20080104151644]
[db-postmodern: cursors re-implemented
[EMAIL PROTECTED]
[controller-doc-improvement
[EMAIL PROTECTED]
[tutorial
[EMAIL PROTECTED]
[non-keyword-accessors
[EMAIL PROTECTED]
allows lispworks to run tests.
]
[function-call-key-form
[EMAIL PROTECTED]
[documentation type fix
[EMAIL PROTECTED]
[Fix the use of internal symbol of sb-kernel in memutils
Leonardo Varuzza <[EMAIL PROTECTED]>**20071230000120
memutil.lisp use the functions sb-kernel::copy-*-from-system-area, which
aren't exported in the latest version of sbcl.
Fix it adding the :: when appropriate
]
[db-bdb bugfix: when bdb key comparison compared only the first half of utf16 strings
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[db-postmodern: optimized map-index for -by-value case
[EMAIL PROTECTED]
[db-postmodern: optimized form-slot-key for persistent-slot-reader
[EMAIL PROTECTED]
it uses SBCL internal function now, for other implementation it's less optimized.
]
[db-postmodern: small example update
[EMAIL PROTECTED]
[added sh script for flushing logs sample
[EMAIL PROTECTED]
[db-postmodern removed possiblity of using NIL as a key in btrees
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163828]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163701]
[removed a little compiler warning (typo)
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151929]
[remove kind-hints parameter from add-index
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151046
Probably a coming feature from Ian, but
right now it breaks the generic function add-index
and thus postmodern, so I removed it for now.
]
[Ensure set-db-synch is defined before pset is loaded
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[Fix to from-end traversal of new map-index
[EMAIL PROTECTED]
[New map-index implementation
[EMAIL PROTECTED]
[Cheaper get-instance-by-value
[EMAIL PROTECTED]
[TAG ELEPHANT-0-9-1
[EMAIL PROTECTED]
Patch bundle hash:
b8048b7280a4642fb6323d73f3d6fbc849eb78ec
New patches:
[Refactor UTF{16,32}LE serializers.
[EMAIL PROTECTED] {
hunk ./src/elephant/unicode2.lisp 115
- (let ((needed (+ size (* characters 2))))
- (when (> needed allocated)
- (resize-buffer-stream bstream needed))
- (etypecase string
- (simple-string
- (loop for i fixnum from 0 below characters do
- (let ((code (char-code (schar string i))))
- (when (> code #xFFFF) (fail))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size))
-;; (coerce (ldb (byte 8 8) code) '(signed 8)))
- (ldb (byte 8 8) code))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1))
-;; (coerce (ldb (byte 8 0) code) '(signed 8))))))
- (ldb (byte 8 0) code)))))
- (string
- (loop for i fixnum from 0 below characters do
- (let ((code (char-code (schar string i))))
- (when (> code #xFFFF) (fail))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size))
-;; (coerce (ldb (byte 8 8) code) '(signed 8)))
- (ldb (byte 8 8) code))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1))
-;; (coerce (ldb (byte 8 0) code) '(signed 8)))))))
- (ldb (byte 8 0) code))))))
- (incf size (* characters 2))
- (succeed))))))
+ (let ((needed (+ size (* characters 2)))
+ (char (etypecase string
+ (simple-string #'schar)
+ (string #'char))))
+ (when (> needed allocated)
+ (resize-buffer-stream bstream needed))
+ (loop for i fixnum from 0 below characters do
+ (let ((code (char-code (funcall char string i))))
+ (when (> code #xFFFF) (fail))
+ (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size))
+ ;; (coerce (ldb (byte 8 8) code) '(signed 8)))
+ (ldb (byte 8 8) code))
+ (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1))
+ ;; (coerce (ldb (byte 8 0) code) '(signed 8))))))
+ (ldb (byte 8 0) code))))
+ (incf size (* characters 2))
+ (succeed))))))
hunk ./src/elephant/unicode2.lisp 144
- (let ((needed (+ size (* 4 characters))))
+ (let ((needed (+ size (* 4 characters)))
+ (char (etypecase string
+ (simple-string #'schar)
+ (string #'char))))
hunk ./src/elephant/unicode2.lisp 150
- (etypecase string
- (simple-string
hunk ./src/elephant/unicode2.lisp 151
- (let ((code (char-code (schar string i))))
+ (let ((code (char-code (funcall char string i))))
hunk ./src/elephant/unicode2.lisp 161
- (string
- (loop for i fixnum from 0 below characters do
- (let ((code (char-code (schar string i))))
- (when (> code #x10FFFF) (error "Invalid unicode code type"))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 0))
- (ldb (byte 8 24) code))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 1))
- (ldb (byte 8 16) code))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 2))
- (ldb (byte 8 8) code))
- (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 3))
- (ldb (byte 8 0) code))))))
hunk ./src/elephant/unicode2.lisp 162
- t))))
+ t)))
}
Context:
[Alex's patch for 8.3
[EMAIL PROTECTED]
I entered here the patch from Alex of 2088/02/16
which apparently makes this compatible with Postgres 8.3.
It is green for me on all tests on Posgres 8.1, so
I am committing it.
]
[mtype change in dcm
[EMAIL PROTECTED]
[moved cache-instance into initial-persistent-setup
[EMAIL PROTECTED]
because it was bypassed by recreate-instance otherwise
]
[accessor name in tests change
[EMAIL PROTECTED]
[db-postmodern: pm-btree initialization fixed
[EMAIL PROTECTED]
[recreate-instance stuff improved
[EMAIL PROTECTED]
[db-postmodern: removed specialized map-index
[EMAIL PROTECTED]
because pure cursor version works fine and is more robust
]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>*-20071124163701]
[db-postmodern fix map-index optimization bug
Henrik Hjelte <[EMAIL PROTECTED]>**20080104151644]
[db-postmodern: cursors re-implemented
[EMAIL PROTECTED]
[controller-doc-improvement
[EMAIL PROTECTED]
[tutorial
[EMAIL PROTECTED]
[non-keyword-accessors
[EMAIL PROTECTED]
allows lispworks to run tests.
]
[function-call-key-form
[EMAIL PROTECTED]
[documentation type fix
[EMAIL PROTECTED]
[Fix the use of internal symbol of sb-kernel in memutils
Leonardo Varuzza <[EMAIL PROTECTED]>**20071230000120
memutil.lisp use the functions sb-kernel::copy-*-from-system-area, which
aren't exported in the latest version of sbcl.
Fix it adding the :: when appropriate
]
[db-bdb bugfix: when bdb key comparison compared only the first half of utf16 strings
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[db-postmodern: optimized map-index for -by-value case
[EMAIL PROTECTED]
[db-postmodern: optimized form-slot-key for persistent-slot-reader
[EMAIL PROTECTED]
it uses SBCL internal function now, for other implementation it's less optimized.
]
[db-postmodern: small example update
[EMAIL PROTECTED]
[added sh script for flushing logs sample
[EMAIL PROTECTED]
[db-postmodern removed possiblity of using NIL as a key in btrees
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163828]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163701]
[removed a little compiler warning (typo)
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151929]
[remove kind-hints parameter from add-index
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151046
Probably a coming feature from Ian, but
right now it breaks the generic function add-index
and thus postmodern, so I removed it for now.
]
[Ensure set-db-synch is defined before pset is loaded
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[Fix to from-end traversal of new map-index
[EMAIL PROTECTED]
[New map-index implementation
[EMAIL PROTECTED]
[Cheaper get-instance-by-value
[EMAIL PROTECTED]
[TAG ELEPHANT-0-9-1
[EMAIL PROTECTED]
Patch bundle hash:
f7207a0af150f822a5b3f2cb3461ddeebbbcacbb_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
