> On 21 Dec 2015, at 23:48, Richard Uttner <[email protected]> wrote: > > This is my first contribution to the DevList and I hope you forgive me that > it sounds more like "I see something we dont need" than the other way round, > but there will be more positive posts of mine concering the Unicode debate > (which I will postpone to the New Year in order to be able to respond > quickly).
Everybody's opinion count, thank you for yours. > From a developer's point of view that is dealing with more than one > environment (3 Smalltalk environments including Pharo, where I am a newby) > and a handful of other languages, I dont like to reuse interfaces that are > hard to remember and understand though the task done by them is so easy that > I can write it down in one simple #inject:into-line 10 times faster than > looking it up in the implementors browser. Furthermore, something like > #inject:into: will work forever and never be deprecated! Fullstops. That is my fallback position: if we can't agree on an implementation, just throw them all out, it is very easy to write yourself. But I would not prefer it. > Next thing that is easily overlooked is the semantics. I am not that good in > mathematics, but I would suppose that summing up does by definition not need > a neutral element - only an implementation that can be written down in 2 > lines needs it. A mathematically correct implementation of summing up > anything (including nil ... see reason below) that implements #+ would be: > > Collection:>>#sum > | undef result | > undef := result := Object new. > self do: [:each | result := result == undef ifTrue: [each] ifFalse: [result > + each]]. Interesting, clever, but I guess you forgot a return of result at the end ? In that case, you would return some new object for an empty collection, not 0 like I would prefer. For me the whole discussion centers around what happens with an empty collection. We know, and currently have, an implementation that can work on non-empty collection without using a neutral element to start with. That is cool. However, if you check the actual senders, more than half of them are doing silly empty checks before invoking the summing. Which to me defeats what you gain by not writing an #inject:into: on the spot. > 2015-12-21 18:00 GMT+01:00 <[email protected]>: > Send Pharo-dev mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.pharo.org/mailman/listinfo/pharo-dev_lists.pharo.org > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Pharo-dev digest..." > > > Today's Topics: > > 1. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Dimitris Chloupis) > 2. Re: #sum:, #detectSum:, #sumNumbers: (Max Leske) > 3. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Mariano Martinez Peck) > 4. Re: #sum:, #detectSum:, #sumNumbers: (Max Leske) > 5. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Dimitris Chloupis) > 6. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Thierry Goubier) > 7. Re: #sum:, #detectSum:, #sumNumbers: (Tudor Girba) > 8. [pharo-project/pharo-core] (GitHub) > 9. [pharo-project/pharo-core] e8216b: 50508 (GitHub) > 10. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Denis Kudriashov) > 11. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Ben Coman) > 12. Re: [squeak-dev] [Unicode] Summary (Re: Unicode Support // e > acute example --> decomposition in Pharo?) (Sven Van Caekenberghe) > 13. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Thierry Goubier) > 14. Re: [squeak-dev] [Unicode] Summary (Re: Unicode Support // e > acute example --> decomposition in Pharo?) (Sven Van Caekenberghe) > 15. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Robert Withers) > 16. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort ([email protected]) > 17. Re: [ANN] Multiple Desktop support for Pharo 5 (H. Hirzel) > 18. No refactoring for equal/hash generation (Denis Kudriashov) > 19. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Denis Kudriashov) > 20. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Mariano Martinez Peck) > 21. Re: [Pharo-users] [ANN] Pharo Consortium Sponsored > Development Effort (Robert Withers) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 20 Dec 2015 17:17:42 +0000 > From: Dimitris Chloupis <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAN88a2F3CNMTZRbdHttwmHzzsxKgs5qZUj=ube4uam4tavz...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > how the communication is happening via Fuel , does it save to a fuel file > or it does it through memory some way ? > > On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis <[email protected]> wrote: > > > On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote: > > > On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: > > > > Fuel I assume enters here the equation as a data exchange format , the > > > > problem I have with fuel is that its not backward compatible which for > > me > > > > > > > Fuel works well in this case. Version compatibility is not an issue, > > because > > when you fork an image/VM the two cooperating images are essentially > > identical. > > > > > Once you have a set of OS processes running Pharo how they talk to each > > other > > > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, > > JSONRPC, > > > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. > > > > Any of those would work, but Fuel is a good way to enable all kinds of > > objects to be copied from one image to another through a stream. In the > > case of RemoteTask, all of the results of a remote computation can be put > > into a single result object (an array or a dictionary or whatever), and > > that result can be passed back to the parent image by reading the single > > object. So - fork a copy of the image, have it do some work, and read the > > result object from a stream. You know you are done when you have read > > exactly one object. > > > > Dave > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151220/b09bbb4e/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Sun, 20 Dec 2015 18:19:43 +0100 > From: Max Leske <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] #sum:, #detectSum:, #sumNumbers: > Message-ID: <[email protected]> > Content-Type: text/plain; charset="utf-8" > > > > On 20 Dec 2015, at 15:26, Ben Coman <[email protected]> wrote: > > > > On Mon, Dec 21, 2015 at 12:43 AM, Sven Van Caekenberghe <[email protected] > > <mailto:[email protected]>> wrote: > >> Doru, > >> > >> For me this whole discussion started because you (the standpoint that you > >> take) hijacked the best selector (#sum) for a use case that is much less > >> common than adding a collection of numbers which should give 0 when empty > >> (you hijack it by not wanting to return 0, which I totally understand, but > >> doing so you redefine the meaning/semantics). > >> > >> Max's point is to make everything more consistent and remove some API. I > >> support that too. > >> > >> Now, I like Max's proposal. > >> > >> But, you know, my conclusion when the thread ended was that it might be > >> better to throw all these selectors away, since #inject:into: covers most > >> use cases at least as well and at least as clear. > >> > >> Sven > >> > >>> On 20 Dec 2015, at 14:10, Tudor Girba <[email protected]> wrote: > >>> > >>> Hi, > >>> > >>> Could we not have sum, but sumNumbers instead? So, we would end up with: > >>> > >>> sum:ifEmpty: > >>> sum: (with error) > >>> sumNumbers (without error) > >>> > >>> From the outside, #sum: looks like it should parameterize #sum, but the > >>> implementation is actually different. So, given that in this > >>> implementation #sum is not a special case of #sum: the two should be > >>> named differently to reflect that difference. Hence my proposal is to > >>> keep #sumNumbers instead of #sum. > > > > > > I agree somewhat with Duro to avoid #sum and #sum: having different > > semantics, > > but I agree more with Sven about keeping #sum for numbers only and > > returning zero, > > so why not turn Doru's suggestion around... > > > > #sum (empty --> 0) > > #sumBy: (empty --> error) > > #sumBy: ifEmpty: (empty --> no error) > > > > alternatively could be #sumUsing: or #sumWith: or something similar. > > Also a good idea. However, looking at the methods on Collection, the pattern > is usually #message and #message:, e.g. #sorted and #sorted:. There used to > be #sortBy: but it was removed because the view was that the protocol > semantics should be the same across all methods. > The current case is a bit different of course since we?re dealing with the > problem of the zero element which doesn?t arise in most other cases. Still, I > think from a users point of view it would be strange to have to use #sumBy: > when every other message pair uses the #message / #message: pattern. > > I feel that this argument (thanks Ben :) ) is also a valid point against > Doru?s argument for #sumNumbers. While #sumNumbers may *technically* be the > best name (which we can?t seem to agree on?), #sum, #sum: and #sum:ifEmpty: > form a triple that naturally fits into the naming protocol applied elsewhere. > > > Cheers, > Max > > > > > > > > >>> > >>> Cheers, > >>> Doru > >>> > >>> > >>>> On Dec 20, 2015, at 1:47 PM, Max Leske <[email protected]> wrote: > >>>> > >>>>> > >>>>> On 20 Dec 2015, at 13:43, Gabriel Cotelli <[email protected]> wrote: > >>>>> > >>>>> Max, > >>>>> > >>>>> sum: aBlock ifEmpty: emptyBlock needs to obtain the sample evaluating > >>>>> the block. > >>>>> > >>>>> sum: aBlock ifEmpty: emptyBlock > >>>>> | sum sample | > >>>>> self isEmpty ifTrue: [ ^ emptyBlock value ]. > >>>>> sample := aBlock value: self anyOne. > >>>>> sum := self > >>>>> inject: sample > >>>>> into: [ :accum :each | accum + (aBlock value: each) ]. > >>>>> ^ sum - sample > >>>> > >>>> > >>>> Thanks! Missed that. > >>>> > >>>>> > >>>>> On Sun, Dec 20, 2015 at 8:59 AM, Max Leske <[email protected]> wrote: > >>>>> I would like to wrap up this discussion. > >>>>> > >>>>> > >>>>>> On 05 Dec 2015, at 18:14, stepharo <[email protected]> wrote: > >>>>>> > >>>>>> So what is the conclusion? > >>>>>> I like the idea of Esteban M to have iterator because it moves some > >>>>>> behavior out of core classes. > >>>>>> > >>>>>> [[[ > >>>>>> > >>>>>> aCollection arithmetic sum: [...] or.... aCollection > >>>>>> arithmetic avg. > >>>>>> > >>>>>> ]]] > >>>>>> > >>>>> > >>>>> While I think that iterators are an intriguing idea I also believe that > >>>>> they are beyond the scope of this issue. If anybody wants to follow up > >>>>> on iterators (or unit types for that matter) please start a new thread > >>>>> / issue. > >>>>> > >>>>> > >>>>> I propose to use Sven?s version for #sum:ifEmpty:. The result would be > >>>>> these three methods: > >>>>> > >>>>> sum > >>>>> ^ self > >>>>> sum: [ :each | each ] > >>>>> ifEmpty: [ 0 ] > >>>>> > >>>>> sum: aBlock > >>>>> ^ self > >>>>> sum: aBlock > >>>>> ifEmpty: [ self errorEmptyCollection ] > >>>>> > >>>>> sum: aBlock ifEmpty: emptyBlock > >>>>> | sum sample | > >>>>> self isEmpty ifTrue: [ ^ emptyBlock value ]. > >>>>> sample := self anyOne. > >>>>> sum := self > >>>>> inject: sample > >>>>> into: [ :accum :each | accum + (aBlock value: each) ]. > >>>>> ^ sum - sample > >>>>> > >>>>> > >>>>> I?ve attached a couple of benchmark results below. To me they show that > >>>>> 1. the new implementation is maybe a tiny bit slower but > >>>>> insignificantly so (if you?re going for performance you?ll probably > >>>>> write your own optimised version anyway) > >>>>> 2. there is no need to duplicate the code (like #sum and #sum: > >>>>> currently do). It?s perfectly fine to delegate to #sum:ifEmpty: > >>>>> > >>>>> > >>>>> > >>>>> In addition to the above changes I would like to remove #detectSum: (-> > >>>>> #sum:) and #sumNumbers (-> #sum). > >>>>> > >>>>> > >>>>> Note that once we agree on changing this API, we will need to also > >>>>> change #detectMin:, #detectMax:, #min, #max as well as all overrides > >>>>> (e.g. RunArray, Interval) of these and of #sum et. al. to stay > >>>>> consistent. The changes would of course be in line with this change, > >>>>> such that every operation has a unary selector with a sensible default, > >>>>> one that takes a block and throws an error for empty collections and a > >>>>> third that takes a block for the iteration and one for the empty case. > >>>>> > >>>>> > >>>>> Please cast your vote for these changes: > >>>>> > >>>>> 1. Do you agree to changing #sum and #sum: in the suggested way? > > yes > > > >>>>> 2. Do you agree to the removal of #detectSum:? > > ?? (not familiar) > > > >>>>> 3. Do you agree to the removal of #sumNumbers? > > yes > > > >>>>> 4. Do you agree that the #max and #min selectors also need to be > >>>>> adapted? > > ?? (not familiar) > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151220/6e60a165/attachment-0001.html> > > ------------------------------ > > Message: 3 > Date: Sun, 20 Dec 2015 15:54:23 -0300 > From: Mariano Martinez Peck <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAA+-=mukav9utpeqgjpry-_yfavoomp2dxgankoqhvddw1k...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi Dimitri, Dave et all. > > First let me do a little disclaimer. Regarding the developments itself of > this sponsorship, we discussed about the topics I said in the first email. > However, those are not written in stone and that's why I also asked for > feedback. As you know, Pharo has relatively little resources so those > should be well spent. If we decide to take a spin into a project X, it > doesn't mean other projects are not worth. It simply means that we think X > will be the most useful for the time being. > > One of the final goals of the discusses topics, is to improve a bit the > infrastructure for using Pharo as scripting (imagine Coral). For such a > thing, we believe that improving FileSystem and OSProcess-like is a good > first step to improve the underlying infrastructure. > > That being said, I of course like the idea of distributing processing > across multiple images. No wonder, my PhD tool (Marea) and related > sub-projects (Fuel, Ghost proxies, etc) have something to do with the > mentioned topic. Dave already mentioned about RemoteTask, which was > something I was about to mention. There is also Seamless project (now used > by Remote Debugger stuff) and quite other tools. Finally, I think this > project may have a big percentage that could be considered as "research". > So having a PhD / master student working on this would also be nice. In > fact, maybe Denis Kudriashov might work on that topic??? (he is now in RMOD > team). > > Also, not that forking the image is not my main short-term goal. As said, I > will experiment first with a simple tool to execute OS commands that would > be FFI based, trying to make async operations, and that would help in most > cases. Maybe for those that are not supported, then fallback in OSProcess. > Note that Dave (OSProcess author) is very helpful with me and we are > discussing together about this. > > Torsten, thanks about OS. I will check out that project. I started with > OSProcess and Limbo. So it's good to have another to check. > > I will try to make the survey starting next week. > > > > > > On Sun, Dec 20, 2015 at 2:17 PM, Dimitris Chloupis <[email protected]> > wrote: > > > how the communication is happening via Fuel , does it save to a fuel file > > or it does it through memory some way ? > > > > On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis <[email protected]> > > wrote: > > > >> On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote: > >> > On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: > >> > > Fuel I assume enters here the equation as a data exchange format , the > >> > > problem I have with fuel is that its not backward compatible which > >> for me > >> > > >> > >> Fuel works well in this case. Version compatibility is not an issue, > >> because > >> when you fork an image/VM the two cooperating images are essentially > >> identical. > >> > >> > Once you have a set of OS processes running Pharo how they talk to each > >> other > >> > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, > >> JSONRPC, > >> > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. > >> > >> Any of those would work, but Fuel is a good way to enable all kinds of > >> objects to be copied from one image to another through a stream. In the > >> case of RemoteTask, all of the results of a remote computation can be put > >> into a single result object (an array or a dictionary or whatever), and > >> that result can be passed back to the parent image by reading the single > >> object. So - fork a copy of the image, have it do some work, and read the > >> result object from a stream. You know you are done when you have read > >> exactly one object. > >> > >> Dave > >> > >> > > > -- > Mariano > http://marianopeck.wordpress.com > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151220/408c1ea1/attachment-0001.html> > > ------------------------------ > > Message: 4 > Date: Sun, 20 Dec 2015 20:23:24 +0100 > From: Max Leske <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] #sum:, #detectSum:, #sumNumbers: > Message-ID: <[email protected]> > Content-Type: text/plain; charset="utf-8" > > I did a pass on all the changes that would be required (whatever the outcome > of this discussion). Looks easy enough. One interesting point: > FloatArray>>sum is implemented as a primitive in the FloatArrayPlugin and the > zero element is explicitly defined as 0.0. The primitive will not fail for an > empty collection but return 0.0. That would be in line with our new > definition of #sum. > > > One other thing: should the old selectors be marked as deprecated rather than > being removed? I think that?s the general policy, right? > > Cheers, > Max > > > > On 20 Dec 2015, at 18:19, Max Leske <[email protected]> wrote: > > > > > >> On 20 Dec 2015, at 15:26, Ben Coman <[email protected] > >> <mailto:[email protected]>> wrote: > >> > >> On Mon, Dec 21, 2015 at 12:43 AM, Sven Van Caekenberghe <[email protected] > >> <mailto:[email protected]>> wrote: > >>> Doru, > >>> > >>> For me this whole discussion started because you (the standpoint that you > >>> take) hijacked the best selector (#sum) for a use case that is much less > >>> common than adding a collection of numbers which should give 0 when empty > >>> (you hijack it by not wanting to return 0, which I totally understand, > >>> but doing so you redefine the meaning/semantics). > >>> > >>> Max's point is to make everything more consistent and remove some API. I > >>> support that too. > >>> > >>> Now, I like Max's proposal. > >>> > >>> But, you know, my conclusion when the thread ended was that it might be > >>> better to throw all these selectors away, since #inject:into: covers most > >>> use cases at least as well and at least as clear. > >>> > >>> Sven > >>> > >>>> On 20 Dec 2015, at 14:10, Tudor Girba <[email protected] > >>>> <mailto:[email protected]>> wrote: > >>>> > >>>> Hi, > >>>> > >>>> Could we not have sum, but sumNumbers instead? So, we would end up with: > >>>> > >>>> sum:ifEmpty: > >>>> sum: (with error) > >>>> sumNumbers (without error) > >>>> > >>>> From the outside, #sum: looks like it should parameterize #sum, but the > >>>> implementation is actually different. So, given that in this > >>>> implementation #sum is not a special case of #sum: the two should be > >>>> named differently to reflect that difference. Hence my proposal is to > >>>> keep #sumNumbers instead of #sum. > >> > >> > >> I agree somewhat with Duro to avoid #sum and #sum: having different > >> semantics, > >> but I agree more with Sven about keeping #sum for numbers only and > >> returning zero, > >> so why not turn Doru's suggestion around... > >> > >> #sum (empty --> 0) > >> #sumBy: (empty --> error) > >> #sumBy: ifEmpty: (empty --> no error) > >> > >> alternatively could be #sumUsing: or #sumWith: or something similar. > > > > Also a good idea. However, looking at the methods on Collection, the > > pattern is usually #message and #message:, e.g. #sorted and #sorted:. There > > used to be #sortBy: but it was removed because the view was that the > > protocol semantics should be the same across all methods. > > The current case is a bit different of course since we?re dealing with the > > problem of the zero element which doesn?t arise in most other cases. Still, > > I think from a users point of view it would be strange to have to use > > #sumBy: when every other message pair uses the #message / #message: pattern. > > > > I feel that this argument (thanks Ben :) ) is also a valid point against > > Doru?s argument for #sumNumbers. While #sumNumbers may *technically* be the > > best name (which we can?t seem to agree on?), #sum, #sum: and #sum:ifEmpty: > > form a triple that naturally fits into the naming protocol applied > > elsewhere. > > > > > > Cheers, > > Max > > > >> > >> > >> > >>>> > >>>> Cheers, > >>>> Doru > >>>> > >>>> > >>>>> On Dec 20, 2015, at 1:47 PM, Max Leske <[email protected] > >>>>> <mailto:[email protected]>> wrote: > >>>>> > >>>>>> > >>>>>> On 20 Dec 2015, at 13:43, Gabriel Cotelli <[email protected] > >>>>>> <mailto:[email protected]>> wrote: > >>>>>> > >>>>>> Max, > >>>>>> > >>>>>> sum: aBlock ifEmpty: emptyBlock needs to obtain the sample evaluating > >>>>>> the block. > >>>>>> > >>>>>> sum: aBlock ifEmpty: emptyBlock > >>>>>> | sum sample | > >>>>>> self isEmpty ifTrue: [ ^ emptyBlock value ]. > >>>>>> sample := aBlock value: self anyOne. > >>>>>> sum := self > >>>>>> inject: sample > >>>>>> into: [ :accum :each | accum + (aBlock value: each) ]. > >>>>>> ^ sum - sample > >>>>> > >>>>> > >>>>> Thanks! Missed that. > >>>>> > >>>>>> > >>>>>> On Sun, Dec 20, 2015 at 8:59 AM, Max Leske <[email protected] > >>>>>> <mailto:[email protected]>> wrote: > >>>>>> I would like to wrap up this discussion. > >>>>>> > >>>>>> > >>>>>>> On 05 Dec 2015, at 18:14, stepharo <[email protected] > >>>>>>> <mailto:[email protected]>> wrote: > >>>>>>> > >>>>>>> So what is the conclusion? > >>>>>>> I like the idea of Esteban M to have iterator because it moves some > >>>>>>> behavior out of core classes. > >>>>>>> > >>>>>>> [[[ > >>>>>>> > >>>>>>> aCollection arithmetic sum: [...] or.... aCollection > >>>>>>> arithmetic avg. > >>>>>>> > >>>>>>> ]]] > >>>>>>> > >>>>>> > >>>>>> While I think that iterators are an intriguing idea I also believe > >>>>>> that they are beyond the scope of this issue. If anybody wants to > >>>>>> follow up on iterators (or unit types for that matter) please start a > >>>>>> new thread / issue. > >>>>>> > >>>>>> > >>>>>> I propose to use Sven?s version for #sum:ifEmpty:. The result would be > >>>>>> these three methods: > >>>>>> > >>>>>> sum > >>>>>> ^ self > >>>>>> sum: [ :each | each ] > >>>>>> ifEmpty: [ 0 ] > >>>>>> > >>>>>> sum: aBlock > >>>>>> ^ self > >>>>>> sum: aBlock > >>>>>> ifEmpty: [ self errorEmptyCollection ] > >>>>>> > >>>>>> sum: aBlock ifEmpty: emptyBlock > >>>>>> | sum sample | > >>>>>> self isEmpty ifTrue: [ ^ emptyBlock value ]. > >>>>>> sample := self anyOne. > >>>>>> sum := self > >>>>>> inject: sample > >>>>>> into: [ :accum :each | accum + (aBlock value: each) ]. > >>>>>> ^ sum - sample > >>>>>> > >>>>>> > >>>>>> I?ve attached a couple of benchmark results below. To me they show that > >>>>>> 1. the new implementation is maybe a tiny bit slower but > >>>>>> insignificantly so (if you?re going for performance you?ll probably > >>>>>> write your own optimised version anyway) > >>>>>> 2. there is no need to duplicate the code (like #sum and #sum: > >>>>>> currently do). It?s perfectly fine to delegate to #sum:ifEmpty: > >>>>>> > >>>>>> > >>>>>> > >>>>>> In addition to the above changes I would like to remove #detectSum: > >>>>>> (-> #sum:) and #sumNumbers (-> #sum). > >>>>>> > >>>>>> > >>>>>> Note that once we agree on changing this API, we will need to also > >>>>>> change #detectMin:, #detectMax:, #min, #max as well as all overrides > >>>>>> (e.g. RunArray, Interval) of these and of #sum et. al. to stay > >>>>>> consistent. The changes would of course be in line with this change, > >>>>>> such that every operation has a unary selector with a sensible > >>>>>> default, one that takes a block and throws an error for empty > >>>>>> collections and a third that takes a block for the iteration and one > >>>>>> for the empty case. > >>>>>> > >>>>>> > >>>>>> Please cast your vote for these changes: > >>>>>> > >>>>>> 1. Do you agree to changing #sum and #sum: in the suggested way? > >> yes > >> > >>>>>> 2. Do you agree to the removal of #detectSum:? > >> ?? (not familiar) > >> > >>>>>> 3. Do you agree to the removal of #sumNumbers? > >> yes > >> > >>>>>> 4. Do you agree that the #max and #min selectors also need to be > >>>>>> adapted? > >> ?? (not familiar) > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151220/7c9fdd77/attachment-0001.html> > > ------------------------------ > > Message: 5 > Date: Sun, 20 Dec 2015 19:38:58 +0000 > From: Dimitris Chloupis <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAN88a2FmuoJj4DJ998wbN2KwrA=mtkt_hbd3r6grw-_xnnt...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > yeah I completely agree that any improvement is more than welcomed. Thanks > I will check out RemoteTask and Seamless :) > > On Sun, Dec 20, 2015 at 8:55 PM Mariano Martinez Peck <[email protected]> > wrote: > > > Hi Dimitri, Dave et all. > > > > First let me do a little disclaimer. Regarding the developments itself of > > this sponsorship, we discussed about the topics I said in the first email. > > However, those are not written in stone and that's why I also asked for > > feedback. As you know, Pharo has relatively little resources so those > > should be well spent. If we decide to take a spin into a project X, it > > doesn't mean other projects are not worth. It simply means that we think X > > will be the most useful for the time being. > > > > One of the final goals of the discusses topics, is to improve a bit the > > infrastructure for using Pharo as scripting (imagine Coral). For such a > > thing, we believe that improving FileSystem and OSProcess-like is a good > > first step to improve the underlying infrastructure. > > > > That being said, I of course like the idea of distributing processing > > across multiple images. No wonder, my PhD tool (Marea) and related > > sub-projects (Fuel, Ghost proxies, etc) have something to do with the > > mentioned topic. Dave already mentioned about RemoteTask, which was > > something I was about to mention. There is also Seamless project (now used > > by Remote Debugger stuff) and quite other tools. Finally, I think this > > project may have a big percentage that could be considered as "research". > > So having a PhD / master student working on this would also be nice. In > > fact, maybe Denis Kudriashov might work on that topic??? (he is now in > > RMOD team). > > > > Also, not that forking the image is not my main short-term goal. As said, > > I will experiment first with a simple tool to execute OS commands that > > would be FFI based, trying to make async operations, and that would help in > > most cases. Maybe for those that are not supported, then fallback in > > OSProcess. Note that Dave (OSProcess author) is very helpful with me and we > > are discussing together about this. > > > > Torsten, thanks about OS. I will check out that project. I started with > > OSProcess and Limbo. So it's good to have another to check. > > > > I will try to make the survey starting next week. > > > > > > > > > > > > On Sun, Dec 20, 2015 at 2:17 PM, Dimitris Chloupis <[email protected]> > > wrote: > > > >> how the communication is happening via Fuel , does it save to a fuel file > >> or it does it through memory some way ? > >> > >> On Sun, Dec 20, 2015 at 6:04 PM David T. Lewis <[email protected]> > >> wrote: > >> > >>> On Sun, Dec 20, 2015 at 05:07:47PM +0800, Pierce Ng wrote: > >>> > On Sun, Dec 20, 2015 at 08:56:43AM +0000, Dimitris Chloupis wrote: > >>> > > Fuel I assume enters here the equation as a data exchange format , > >>> the > >>> > > problem I have with fuel is that its not backward compatible which > >>> for me > >>> > > >>> > >>> Fuel works well in this case. Version compatibility is not an issue, > >>> because > >>> when you fork an image/VM the two cooperating images are essentially > >>> identical. > >>> > >>> > Once you have a set of OS processes running Pharo how they talk to > >>> each other > >>> > is up to you no? E.g. the Pharo instances could use ZeroMQ, XML-RPC, > >>> JSONRPC, > >>> > msgpack, coordinate via Redis/SQLite/PostgreSQL, etc. > >>> > >>> Any of those would work, but Fuel is a good way to enable all kinds of > >>> objects to be copied from one image to another through a stream. In the > >>> case of RemoteTask, all of the results of a remote computation can be put > >>> into a single result object (an array or a dictionary or whatever), and > >>> that result can be passed back to the parent image by reading the single > >>> object. So - fork a copy of the image, have it do some work, and read the > >>> result object from a stream. You know you are done when you have read > >>> exactly one object. > >>> > >>> Dave > >>> > >>> > > > > > > -- > > Mariano > > http://marianopeck.wordpress.com > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151220/4619633c/attachment-0001.html> > > ------------------------------ > > Message: 6 > Date: Sun, 20 Dec 2015 21:21:31 +0100 > From: Thierry Goubier <[email protected]> > To: [email protected] > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Mariano, > > Le 20/12/2015 19:54, Mariano Martinez Peck a ?crit : > > Hi Dimitri, Dave et all. > > > > First let me do a little disclaimer. Regarding the developments itself > > of this sponsorship, we discussed about the topics I said in the first > > email. However, those are not written in stone and that's why I also > > asked for feedback. As you know, Pharo has relatively little resources > > so those should be well spent. If we decide to take a spin into a > > project X, it doesn't mean other projects are not worth. It simply means > > that we think X will be the most useful for the time being. > > > > One of the final goals of the discusses topics, is to improve a bit the > > infrastructure for using Pharo as scripting (imagine Coral). For such a > > thing, we believe that improving FileSystem and OSProcess-like is a good > > first step to improve the underlying infrastructure. > > Would we have access then to a command-line oriented minimal Pharo image? > > > That being said, I of course like the idea of distributing processing > > across multiple images. No wonder, my PhD tool (Marea) and related > > sub-projects (Fuel, Ghost proxies, etc) have something to do with the > > mentioned topic. Dave already mentioned about RemoteTask, which was > > something I was about to mention. There is also Seamless project (now > > used by Remote Debugger stuff) and quite other tools. Finally, I think > > this project may have a big percentage that could be considered as > > "research". So having a PhD / master student working on this would also > > be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he > > is now in RMOD team). > > > > Also, not that forking the image is not my main short-term goal. As > > said, I will experiment first with a simple tool to execute OS commands > > that would be FFI based, trying to make async operations, and that would > > help in most cases. Maybe for those that are not supported, then > > fallback in OSProcess. Note that Dave (OSProcess author) is very helpful > > with me and we are discussing together about this. > > My guess is that, at least on Linux/Unix, that FFI will be exactly the > same as the one used/expressed by the OSProcess plugin, that is > popen/system. > > Now, given some of the difficulties seen when dealing with external > processes, I'd be wary of the "async" side of things. Random failures > seemingly linked to signal loss are certainly not something I'd like to > debug. > > Something interesting and powerful would be pseudo-tty support. It would > allow a Pharo image to maintain an open connection to a shell, for > example, reducing the cost and latency of launching external commands > and maintaining a context. > > Regards, > > Thierry > > > > ------------------------------ > > Message: 7 > Date: Sun, 20 Dec 2015 22:09:18 +0100 > From: Tudor Girba <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] #sum:, #detectSum:, #sumNumbers: > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8 > > Hi, > > I cannot seem to agree with the current line of reasoning, so I am > withdrawing from this debate. It?s late in the year and I know I need a > break, so it is likely that I am missing something obvious or that I am just > persisting in some sort of bike-shedding point of view :). > > Thanks for keeping up with this. > > I wish you all a lovely holiday and see you next year! > > Doru > > > > On Dec 20, 2015, at 8:23 PM, Max Leske <[email protected]> wrote: > > > > I did a pass on all the changes that would be required (whatever the > > outcome of this discussion). Looks easy enough. One interesting point: > > FloatArray>>sum is implemented as a primitive in the FloatArrayPlugin and > > the zero element is explicitly defined as 0.0. The primitive will not fail > > for an empty collection but return 0.0. That would be in line with our new > > definition of #sum. > > > > > > One other thing: should the old selectors be marked as deprecated rather > > than being removed? I think that?s the general policy, right? > > > > Cheers, > > Max > > > > > >> On 20 Dec 2015, at 18:19, Max Leske <[email protected]> wrote: > >> > >> > >>> On 20 Dec 2015, at 15:26, Ben Coman <[email protected]> wrote: > >>> > >>> On Mon, Dec 21, 2015 at 12:43 AM, Sven Van Caekenberghe <[email protected]> > >>> wrote: > >>>> Doru, > >>>> > >>>> For me this whole discussion started because you (the standpoint that > >>>> you take) hijacked the best selector (#sum) for a use case that is much > >>>> less common than adding a collection of numbers which should give 0 when > >>>> empty (you hijack it by not wanting to return 0, which I totally > >>>> understand, but doing so you redefine the meaning/semantics). > >>>> > >>>> Max's point is to make everything more consistent and remove some API. I > >>>> support that too. > >>>> > >>>> Now, I like Max's proposal. > >>>> > >>>> But, you know, my conclusion when the thread ended was that it might be > >>>> better to throw all these selectors away, since #inject:into: covers > >>>> most use cases at least as well and at least as clear. > >>>> > >>>> Sven > >>>> > >>>>> On 20 Dec 2015, at 14:10, Tudor Girba <[email protected]> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> Could we not have sum, but sumNumbers instead? So, we would end up with: > >>>>> > >>>>> sum:ifEmpty: > >>>>> sum: (with error) > >>>>> sumNumbers (without error) > >>>>> > >>>>> From the outside, #sum: looks like it should parameterize #sum, but the > >>>>> implementation is actually different. So, given that in this > >>>>> implementation #sum is not a special case of #sum: the two should be > >>>>> named differently to reflect that difference. Hence my proposal is to > >>>>> keep #sumNumbers instead of #sum. > >>> > >>> > >>> I agree somewhat with Duro to avoid #sum and #sum: having different > >>> semantics, > >>> but I agree more with Sven about keeping #sum for numbers only and > >>> returning zero, > >>> so why not turn Doru's suggestion around... > >>> > >>> #sum (empty --> 0) > >>> #sumBy: (empty --> error) > >>> #sumBy: ifEmpty: (empty --> no error) > >>> > >>> alternatively could be #sumUsing: or #sumWith: or something similar. > >> > >> Also a good idea. However, looking at the methods on Collection, the > >> pattern is usually #message and #message:, e.g. #sorted and #sorted:. > >> There used to be #sortBy: but it was removed because the view was that the > >> protocol semantics should be the same across all methods. > >> The current case is a bit different of course since we?re dealing with the > >> problem of the zero element which doesn?t arise in most other cases. > >> Still, I think from a users point of view it would be strange to have to > >> use #sumBy: when every other message pair uses the #message / #message: > >> pattern. > >> > >> I feel that this argument (thanks Ben :) ) is also a valid point against > >> Doru?s argument for #sumNumbers. While #sumNumbers may *technically* be > >> the best name (which we can?t seem to agree on?), #sum, #sum: and > >> #sum:ifEmpty: form a triple that naturally fits into the naming protocol > >> applied elsewhere. > >> > >> > >> Cheers, > >> Max > >> > >>> > >>> > >>> > >>>>> > >>>>> Cheers, > >>>>> Doru > >>>>> > >>>>> > >>>>>> On Dec 20, 2015, at 1:47 PM, Max Leske <[email protected]> wrote: > >>>>>> > >>>>>>> > >>>>>>> On 20 Dec 2015, at 13:43, Gabriel Cotelli <[email protected]> wrote: > >>>>>>> > >>>>>>> Max, > >>>>>>> > >>>>>>> sum: aBlock ifEmpty: emptyBlock needs to obtain the sample evaluating > >>>>>>> the block. > >>>>>>> > >>>>>>> sum: aBlock ifEmpty: emptyBlock > >>>>>>> | sum sample | > >>>>>>> self isEmpty ifTrue: [ ^ emptyBlock value ]. > >>>>>>> sample := aBlock value: self anyOne. > >>>>>>> sum := self > >>>>>>> inject: sample > >>>>>>> into: [ :accum :each | accum + (aBlock value: each) ]. > >>>>>>> ^ sum - sample > >>>>>> > >>>>>> > >>>>>> Thanks! Missed that. > >>>>>> > >>>>>>> > >>>>>>> On Sun, Dec 20, 2015 at 8:59 AM, Max Leske <[email protected]> wrote: > >>>>>>> I would like to wrap up this discussion. > >>>>>>> > >>>>>>> > >>>>>>>> On 05 Dec 2015, at 18:14, stepharo <[email protected]> wrote: > >>>>>>>> > >>>>>>>> So what is the conclusion? > >>>>>>>> I like the idea of Esteban M to have iterator because it moves some > >>>>>>>> behavior out of core classes. > >>>>>>>> > >>>>>>>> [[[ > >>>>>>>> > >>>>>>>> aCollection arithmetic sum: [...] or.... aCollection > >>>>>>>> arithmetic avg. > >>>>>>>> > >>>>>>>> ]]] > >>>>>>>> > >>>>>>> > >>>>>>> While I think that iterators are an intriguing idea I also believe > >>>>>>> that they are beyond the scope of this issue. If anybody wants to > >>>>>>> follow up on iterators (or unit types for that matter) please start a > >>>>>>> new thread / issue. > >>>>>>> > >>>>>>> > >>>>>>> I propose to use Sven?s version for #sum:ifEmpty:. The result would > >>>>>>> be these three methods: > >>>>>>> > >>>>>>> sum > >>>>>>> ^ self > >>>>>>> sum: [ :each | each ] > >>>>>>> ifEmpty: [ 0 ] > >>>>>>> > >>>>>>> sum: aBlock > >>>>>>> ^ self > >>>>>>> sum: aBlock > >>>>>>> ifEmpty: [ self errorEmptyCollection ] > >>>>>>> > >>>>>>> sum: aBlock ifEmpty: emptyBlock > >>>>>>> | sum sample | > >>>>>>> self isEmpty ifTrue: [ ^ emptyBlock value ]. > >>>>>>> sample := self anyOne. > >>>>>>> sum := self > >>>>>>> inject: sample > >>>>>>> into: [ :accum :each | accum + (aBlock value: each) ]. > >>>>>>> ^ sum - sample > >>>>>>> > >>>>>>> > >>>>>>> I?ve attached a couple of benchmark results below. To me they show > >>>>>>> that > >>>>>>> 1. the new implementation is maybe a tiny bit slower but > >>>>>>> insignificantly so (if you?re going for performance you?ll probably > >>>>>>> write your own optimised version anyway) > >>>>>>> 2. there is no need to duplicate the code (like #sum and #sum: > >>>>>>> currently do). It?s perfectly fine to delegate to #sum:ifEmpty: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> In addition to the above changes I would like to remove #detectSum: > >>>>>>> (-> #sum:) and #sumNumbers (-> #sum). > >>>>>>> > >>>>>>> > >>>>>>> Note that once we agree on changing this API, we will need to also > >>>>>>> change #detectMin:, #detectMax:, #min, #max as well as all overrides > >>>>>>> (e.g. RunArray, Interval) of these and of #sum et. al. to stay > >>>>>>> consistent. The changes would of course be in line with this change, > >>>>>>> such that every operation has a unary selector with a sensible > >>>>>>> default, one that takes a block and throws an error for empty > >>>>>>> collections and a third that takes a block for the iteration and one > >>>>>>> for the empty case. > >>>>>>> > >>>>>>> > >>>>>>> Please cast your vote for these changes: > >>>>>>> > >>>>>>> 1. Do you agree to changing #sum and #sum: in the suggested way? > >>> yes > >>> > >>>>>>> 2. Do you agree to the removal of #detectSum:? > >>> ?? (not familiar) > >>> > >>>>>>> 3. Do you agree to the removal of #sumNumbers? > >>> yes > >>> > >>>>>>> 4. Do you agree that the #max and #min selectors also need to be > >>>>>>> adapted? > >>> ?? (not familiar) > >> > > > > -- > www.tudorgirba.com > www.feenk.com > > "Speaking louder won't make the point worthier." > > > > > ------------------------------ > > Message: 8 > Date: Mon, 21 Dec 2015 01:15:32 -0800 > From: GitHub <[email protected]> > To: [email protected] > Subject: [Pharo-dev] [pharo-project/pharo-core] > Message-ID: > > <5677c334bc72_b793fea7b9332bc129...@hookshot-fe3-cp1-prd.iad.github.net.mail> > > Content-Type: text/plain; charset="utf-8" > > Branch: refs/tags/50508 > Home: https://github.com/pharo-project/pharo-core > > ------------------------------ > > Message: 9 > Date: Mon, 21 Dec 2015 01:15:31 -0800 > From: GitHub <[email protected]> > To: [email protected] > Subject: [Pharo-dev] [pharo-project/pharo-core] e8216b: 50508 > Message-ID: > > <5677c33347719_5f103f8ed07cb2a0116...@hookshot-fe2-cp1-prd.iad.github.net.mail> > > Content-Type: text/plain; charset="utf-8" > > Branch: refs/heads/5.0 > Home: https://github.com/pharo-project/pharo-core > Commit: e8216bc7a0ab9fb6f8289d8aabea9fcd1174f878 > > https://github.com/pharo-project/pharo-core/commit/e8216bc7a0ab9fb6f8289d8aabea9fcd1174f878 > Author: Jenkins Build Server <[email protected]> > Date: 2015-12-21 (Mon, 21 Dec 2015) > > Changed paths: > A Fuel.package/extension/Behavior/instance/largeIdentityHash.st > R Kernel.package/Behavior.class/instance/as yet > unclassified/largeIdentityHash.st > M Kernel.package/ProtoObject.class/instance/pointing > to/pointersToExcept_.st > R ScriptLoader50.package/ScriptLoader.class/instance/pharo - > scripts/script50507.st > A ScriptLoader50.package/ScriptLoader.class/instance/pharo - > scripts/script50508.st > R ScriptLoader50.package/ScriptLoader.class/instance/pharo - > updates/update50507.st > A ScriptLoader50.package/ScriptLoader.class/instance/pharo - > updates/update50508.st > M > ScriptLoader50.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st > M System-Support.package/SmalltalkImage.class/instance/special > objects/newSpecialObjectsArray.st > M > System-Support.package/SystemNavigation.class/instance/query/allObjectsDo_.st > M Tools-Test.package/MethodFinderTest.class/instance/test > examples/testPointY.st > > Log Message: > ----------- > 50508 > 17277 newSpecialObjectsArray should return a process list instead of a linked > list at item 41 > https://pharo.fogbugz.com/f/cases/17277 > > 17268 Spur Migration - #pointersTo not working > https://pharo.fogbugz.com/f/cases/17268 > > 17271 Spur Migration - MethodFinderTest is relaying in the order of the > returned selectors for the test. > https://pharo.fogbugz.com/f/cases/17271 > > 17272 Uncategorized spur-related methods should be classified > https://pharo.fogbugz.com/f/cases/17272 > > http://files.pharo.org/image/50/50508.zip > > > > ------------------------------ > > Message: 10 > Date: Mon, 21 Dec 2015 10:30:12 +0100 > From: Denis Kudriashov <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAG0zXM7VrRus+_NMvhUj3iMd=y77i3j74j6kxfv5ske+v6t...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello > > 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck <[email protected]>: > > > That being said, I of course like the idea of distributing processing > > across multiple images. No wonder, my PhD tool (Marea) and related > > sub-projects (Fuel, Ghost proxies, etc) have something to do with the > > mentioned topic. Dave already mentioned about RemoteTask, which was > > something I was about to mention. There is also Seamless project (now used > > by Remote Debugger stuff) and quite other tools. Finally, I think this > > project may have a big percentage that could be considered as "research". > > So having a PhD / master student working on this would also be nice. In > > fact, maybe Denis Kudriashov might work on that topic??? (he is now in > > RMOD team). > > > Yes. This is what I'm doing now. At last weeks of this year I plan to > finish my improvements to Seamless. Then I will adapt system and tools for > remote usage. Main goal is allow work with remote images same way we work > locally: remote debugging, browsing, any kind of system navigation, editing > code. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151221/cfa69ddf/attachment-0001.html> > > ------------------------------ > > Message: 11 > Date: Mon, 21 Dec 2015 21:05:55 +1100 > From: Ben Coman <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAProBT=SbeLUHK2SX6=CPHJATJF-coqRhjuVD=7f5iwspvg...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Mon, Dec 21, 2015 at 8:30 PM, Denis Kudriashov <[email protected]> > wrote: > > Hello > > > > 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck <[email protected]>: > >> > >> That being said, I of course like the idea of distributing processing > >> across multiple images. No wonder, my PhD tool (Marea) and related > >> sub-projects (Fuel, Ghost proxies, etc) have something to do with the > >> mentioned topic. Dave already mentioned about RemoteTask, which was > >> something I was about to mention. There is also Seamless project (now used > >> by Remote Debugger stuff) and quite other tools. Finally, I think this > >> project may have a big percentage that could be considered as "research". > >> So > >> having a PhD / master student working on this would also be nice. In fact, > >> maybe Denis Kudriashov might work on that topic??? (he is now in RMOD > >> team). > > > > > > Yes. This is what I'm doing now. At last weeks of this year I plan to finish > > my improvements to Seamless. Then I will adapt system and tools for remote > > usage. Main goal is allow work with remote images same way we work locally: > > remote debugging, browsing, any kind of system navigation, editing code. > > Cool. I'm looking forward to maybe one day working like that with a > minimal image on embedded hardware. > > cheers -ben > > > > ------------------------------ > > Message: 12 > Date: Mon, 21 Dec 2015 11:17:37 +0100 > From: Sven Van Caekenberghe <[email protected]> > To: Pharo Development List <[email protected]> > Cc: [email protected] > Subject: Re: [Pharo-dev] [squeak-dev] [Unicode] Summary (Re: Unicode > Support // e acute example --> decomposition in Pharo?) > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8 > > Andres, > > There are no plans at all to drop any of the existing character encodings > from Pharo. UTF-16 LE & BE will remain part of the standard image, as are all > single byte encodings. No need to worry. > > Sven > > > On 19 Dec 2015, at 03:04, Andres Valloud > > <[email protected]> wrote: > > > > So a lot of Windows APIs require UTF-16. What's up with UTF-8 being the > > only choice mentioned for external communication? > > > > Unicode string encodings like UTF-* and strings of "characters" (that is, > > sequences of Unicode code points) should be clearly distinguished. Do you > > really mean "UTF-32", or do you mean "UCS-4"? Even those two are not > > exactly the same. > > > > On 12/18/15 5:47 , H. Hirzel wrote: > >> Hello Sven > >> > >> Thank you for your report about about your experimental, proof of > >> concept, prototype project, that aims to improve Unicode support. > >> Please include me in the loop. > >> > >> Below is is my attempt at summarizing the Unicode discussion of the last > >> weeks. > >> Corrections /comments / additions are welcome. > >> > >> Kind regards > >> > >> Hannes > >> > >> > >> 1) There is a need for improved Unicode support implemented _within_ > >> the image , probably as a library. > >> > >> 1a) This follows the example of the the Twitter CLDR library (i.e. > >> re-implementation of ICU components for Ruby). > >> https://github.com/twitter/twitter-cldr-rb > >> > >> Other languages/libraries have similar approaches > >> - dotNet, > >> https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeInfo%28v=vs.110%29.aspx) > >> - Python https://docs.python.org/3/howto/unicode.html > >> - Go http://blog.golang.org/strings > >> - Swift, > >> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html > >> - Perl http://blog.golang.org/strings > >> > >> 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This > >> is because of security and portability reasons (Eliot Miranda) and > >> because of the Smalltalk approach that wants to expose basic > >> algorithms in Smalltalk code. In addition the 16bit based ICU library > >> does not fit well with the Squeak/Pharo UTF32 model. > >> > >> 2) The Unicode infrastructure (21(32) bit wide Characters as immediate > >> objects, use of UTF-32 internally, indexable strings, UTF8 for outside > >> communication, support of code converters) is a very valuable > >> foundation which makes algorithms more straightforward at the expense > >> of a more memory usage. It not used to its full potential at all > >> currently though a lot of hard work has been done. > >> > >> 3) The Unicode algorithms are mostly table / database driven. This > >> means that dictionary lookup is a prominent part of the algorithms. > >> The essential building block for this is that the Unicode character > >> database UCD (http://www.unicode.org/ucd/) is made available > >> _within_ the image with the full content as needed by the target > >> languages / scripts one wants to deal with. The process of loading the > >> UCD should be made configurable. > >> > >> 3a) a lot of people are interested in the Latin script (and scripts of > >> similar complexity) only. > >> 3b) The UCD data in XML form > >> http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with > >> and without the CJK characters. > >> > >> 4) The next step is to implement normalization > >> (http://www.unicode.org/reports/tr15/#Norm_Forms). Glad to read that > >> you have reached results here with the test data: > >> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. > >> > >> 5) Pharo offers nice inspectors to view dictionaries and ordered > >> collections (table view, drill down) which facilitates the development > >> to table driven algorithms. The data structures and algorithm are do > >> not depend on a particular dialect though and may be ported to Squeak > >> or Cuis. > >> > >> 6) After having implemented normalization, comparison may be > >> implemented. This needs CLDR access (collation, Unicode Common Locale > >> Data Repository, http://cldr.unicode.org/ ). > >> > >> > >> 7) An architecture has the following subsystems > >> > >> 7a) Basic character handling (21(32)bit characters in indexable > >> strings, point 2) > >> 7b) Runtime access to the Unicode Character Database (point 3) > >> 7c) Converters > >> 7d) Normalization (point 4) > >> 7e) CLDR access (point 6) > >> > >> > >> 8) The implementation should be driven by the current needs. > >> > >> An attainable next goal is to release > >> > >> 8a) a StringBuilder utility class for easier construction of test strings > >> i.e. instead of > >> > >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >>> String). > >> > >> do > >> normalizer composeString: > >> (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') > >> > >> and construct some test cases with it which illustrate some basic > >> Unicode issues. > >> > >> 8b) identity testing for major languages (e.g. French, German, > >> Spanish) and scripts of similar complexity. I > >> > >> 8c) to provide some more documentation of past and concurrent efforts. > >> > >> Note: This summary has only covered string manipulation, not rendering > >> on the screen which is a different issue. > >> > >> > >> On 12/16/15, Sven Van Caekenberghe <[email protected]> wrote: > >>> Hi Hannes, > >>> > >>> My detailed comments/answers below, after quoting 2 of your emails: > >>> > >>>> On 10 Dec 2015, at 22:17, H. Hirzel <[email protected]> wrote: > >>>> > >>>> Hello Sven > >>>> > >>>> On 12/9/15, Sven Van Caekenberghe <[email protected]> wrote: > >>>> > >>>>> The simplest example in a common language is (the French letter ?) is > >>>>> > >>>>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] > >>>>> > >>>>> which can also be written as > >>>>> > >>>>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT > >>>>> [U+0301] > >>>>> > >>>>> The former being a composed normal form, the latter a decomposed normal > >>>>> form. (And yes, it is even much more complicated than that, it goes on > >>>>> for > >>>>> 1000s of pages). > >>>>> > >>>>> In the above example, the concept of character/string is indeed fuzzy. > >>>>> > >>>>> HTH, > >>>>> > >>>>> Sven > >>>> > >>>> Thanks for this example. I have created a wiki page with it > >>>> > >>>> I wonder what the Pharo equivalent is of the following Squeak expression > >>>> > >>>> $? asString asDecomposedUnicode > >>>> > >>>> Regards > >>>> > >>>> Hannes > >>> > >>> You also wrote: > >>> > >>>> The text below shows how to deal with the Unicode e acute example > >>>> brought up by Sven in terms of comparing strings. Currently Pharo and > >>>> Cuis do not do Normalization of strings. Limited support is in Squeak. > >>>> It will be shown how NFD normalization may be implemented. > >>>> > >>>> > >>>> Swift programming language > >>>> ----------------------------------------- > >>>> > >>>> How does the Swift programming language [1] deal with Unicode strings? > >>>> > >>>> // "Voulez-vous un caf??" using LATIN SMALL LETTER E WITH ACUTE > >>>> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" > >>>> > >>>> // "Voulez-vous un café?" using LATIN SMALL LETTER E and > >>>> COMBINING ACUTE ACCENT > >>>> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" > >>>> > >>>> if eAcuteQuestion == combinedEAcuteQuestion { > >>>> print("These two strings are considered equal") > >>>> } > >>>> // prints "These two strings are considered equal" > >>>> > >>>> The equality operator uses the NFD (Normalization Form Decomposed)[2] > >>>> form for the comparison appyling a method > >>>> #decomposedStringWithCanonicalMapping[3] > >>>> > >>>> > >>>> Squeak / Pharo > >>>> ----------------------- > >>>> > >>>> Comparison without NFD [3] > >>>> > >>>> > >>>> "Voulez-vous un caf??" > >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>>> asString, '?'. > >>>> > >>>> > >>>> eAcuteQuestion = combinedEAcuteQuestion > >>>> false > >>>> > >>>> eAcuteQuestion == combinedEAcuteQuestion > >>>> false > >>>> > >>>> The result is false. A Unicode conformant application however should > >>>> return *true*. > >>>> > >>>> Reason for this is that Squeak / Pharo strings are not put into NFD > >>>> before testing for equality = > >>>> > >>>> > >>>> Squeak Unicode strings may be tested for Unicode conformant equality > >>>> by converting them to NFD before testing. > >>>> > >>>> > >>>> > >>>> Squeak using NFD > >>>> > >>>> asDecomposedUnicode[4] transforms a string into NFD for cases where a > >>>> Unicode code point if decomposed, is decomposed only to two code > >>>> points [5]. This is so because when initializing [6] the Unicode > >>>> Character Database in Squeak this is a limitation imposed by the code > >>>> which reads UnicodeData.txt [7][8]. This is not a necessary > >>>> limitation. The code may be rewritten at the price of a more complex > >>>> implementation of #asDecomposedUnicode. > >>>> > >>>> "Voulez-vous un caf??" > >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>>> asString, '?'. > >>>> > >>>> > >>>> eAcuteQuestion asDecomposedUnicode = > >>>> combinedEAcuteQuestion asDecomposedUnicode > >>>> > >>>> true > >>>> > >>>> > >>>> > >>>> Conclusion > >>>> ------------------ > >>>> > >>>> Implementing a method like #decomposedStringWithCanonicalMapping > >>>> (swift) which puts a string into NFD (Normalization Form D) is an > >>>> important building block towards better Unicode compliance. A Squeak > >>>> proposal is given by [4]. It needs to be reviewed.extended. > >>>> > >>>> It should probably be extended for cases where there are more than > >>>> two code points in the decomposed form (3 or more?) > >>>> > >>>> The implementing of NFD comparison gives us an equality test for a > >>>> comparatively small effort for simple cases covering a large number of > >>>> use cases (Languages using the Latin script). > >>>> > >>>> The algorithm is table driven by the UCD [8]. From this follows an > >>>> simple but important fact for conformant implementations need runtime > >>>> access to information from the Unicode Character Database [UCD][9]. > >>>> > >>>> > >>>> [1] > >>>> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html#//apple_ref/doc/uid/TP40014097-CH7-ID285 > >>>> [2] http://www.unicode.org/glossary/#normalization_form_d > >>>> [3] > >>>> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/index.html#//apple_ref/occ/instm/NSString/decomposedStringWithCanonicalMapping > >>>> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 > >>>> [5] http://www.unicode.org/glossary/#code_point > >>>> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 > >>>> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt > >>>> [8] Unicode Character Database documentation > >>>> http://unicode.org/reports/tr44/ > >>>> [9] http://www.unicode.org/reports/tr23/ > >>> > >>> > >>> Today, we have a Unicode and CombinedCharacter class in Pharo, and there > >>> is > >>> different but similar Unicode code in Squeak. These are too simple (even > >>> though they might work, partially). > >>> > >>> The scope of the original threads is way too wide: a new string type, > >>> normalisation, collation, being cross dialect, mixing all kinds of > >>> character > >>> and encoding definitions. All interesting, but not much will come out of > >>> it. > >>> But the point that we cannot leave proper text string handling to an > >>> outside > >>> library is indeed key. > >>> > >>> That is why a couple of people in the Pharo community (myself included) > >>> started an experimental, proof of concept, prototype project, that aims to > >>> improve Unicode support. We will announce it to a wider public when we > >>> feel > >>> we have something to show for. The goal is in the first place to > >>> understand > >>> and implement the fundamental algorithms, starting with the 4 forms of > >>> Normalisation. But we're working on collation/sorting too. > >>> > >>> This work is of course being done for/in Pharo, using some of the > >>> facilities > >>> only available there. It probably won't be difficult to port, but we can't > >>> be bothered with probability right now. > >>> > >>> What we started with is loading UCD data and making it available as a nice > >>> objects (30.000 of them). > >>> > >>> So now you can do things like > >>> > >>> $? unicodeCharacterData. > >>> > >>> => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" > >>> > >>> $? unicodeCharacterData uppercase asCharacter. > >>> > >>> => "$?" > >>> > >>> $? unicodeCharacterData decompositionMapping. > >>> > >>> => "#(101 769)" > >>> > >>> There is also a cool GT Inspector view: > >>> > >>> > >>> > >>> Next we started implementing a normaliser. It was rather easy to get > >>> support > >>> for simpler languages going. The next code snippets use explicit code > >>> arrays, because copying decomposed diacritics to my mail client does not > >>> work (they get automatically composed), in a Pharo Workspace this does > >>> work > >>> nicely with plain strings. The higher numbers are the diacritics. > >>> > >>> (normalizer decomposeString: 'les ?l?ves Fran?ais') collect: #codePoint > >>> as: > >>> Array. > >>> > >>> => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 > >>> 807 97 105 115)" > >>> > >>> (normalizer decomposeString: 'D?sseldorf K?nigsallee') collect: #codePoint > >>> as: Array. > >>> > >>> => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 > >>> 115 97 108 108 101 101)" > >>> > >>> normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 > >>> 115 > >>> 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). > >>> > >>> => "'les ?l?ves Fran?ais'" > >>> > >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >>> String). > >>> > >>> => "'D?sseldorf K?nigsallee'" > >>> > >>> However, the real algorithm following the official specification (and > >>> other > >>> elements of Unicode that interact with it) is way more complicated (think > >>> about all those special languages/scripts out there). We're focused on > >>> understanding/implementing that now. > >>> > >>> Next, unit tests were added (of course). As well as a test that uses > >>> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > >>> 75.000 individual test cases to check conformance to the official Unicode > >>> Normalization specification. > >>> > >>> Right now (with super cool hangul / jamo code by Henrik), we hit the > >>> following stats: > >>> > >>> #testNFC 16998/18593 (91.42%) > >>> #testNFD 16797/18593 (90.34%) > >>> #testNFKC 13321/18593 (71.65%) > >>> #testNFKD 16564/18593 (89.09%) > >>> > >>> Way better than the naive implementations, but not yet there. > >>> > >>> We are also experimenting and thinking a lot about how to best implement > >>> all > >>> this, trying out different models/ideas/apis/representations. > >>> > >>> It will move slowly, but you will hear from us again in the coming > >>> weeks/months. > >>> > >>> Sven > >>> > >>> PS: Pharo developers with a good understanding of this subject area that > >>> want to help, let me know and we'll put you in the loop. Hacking and > >>> specification reading are required ;-) > >>> > >>> > >> > >> . > >> > > > On 19 Dec 2015, at 03:04, Andres Valloud > > <[email protected]> wrote: > > > > So a lot of Windows APIs require UTF-16. What's up with UTF-8 being the > > only choice mentioned for external communication? > > > > Unicode string encodings like UTF-* and strings of "characters" (that is, > > sequences of Unicode code points) should be clearly distinguished. Do you > > really mean "UTF-32", or do you mean "UCS-4"? Even those two are not > > exactly the same. > > > > On 12/18/15 5:47 , H. Hirzel wrote: > >> Hello Sven > >> > >> Thank you for your report about about your experimental, proof of > >> concept, prototype project, that aims to improve Unicode support. > >> Please include me in the loop. > >> > >> Below is is my attempt at summarizing the Unicode discussion of the last > >> weeks. > >> Corrections /comments / additions are welcome. > >> > >> Kind regards > >> > >> Hannes > >> > >> > >> 1) There is a need for improved Unicode support implemented _within_ > >> the image , probably as a library. > >> > >> 1a) This follows the example of the the Twitter CLDR library (i.e. > >> re-implementation of ICU components for Ruby). > >> https://github.com/twitter/twitter-cldr-rb > >> > >> Other languages/libraries have similar approaches > >> - dotNet, > >> https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeInfo%28v=vs.110%29.aspx) > >> - Python https://docs.python.org/3/howto/unicode.html > >> - Go http://blog.golang.org/strings > >> - Swift, > >> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html > >> - Perl http://blog.golang.org/strings > >> > >> 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This > >> is because of security and portability reasons (Eliot Miranda) and > >> because of the Smalltalk approach that wants to expose basic > >> algorithms in Smalltalk code. In addition the 16bit based ICU library > >> does not fit well with the Squeak/Pharo UTF32 model. > >> > >> 2) The Unicode infrastructure (21(32) bit wide Characters as immediate > >> objects, use of UTF-32 internally, indexable strings, UTF8 for outside > >> communication, support of code converters) is a very valuable > >> foundation which makes algorithms more straightforward at the expense > >> of a more memory usage. It not used to its full potential at all > >> currently though a lot of hard work has been done. > >> > >> 3) The Unicode algorithms are mostly table / database driven. This > >> means that dictionary lookup is a prominent part of the algorithms. > >> The essential building block for this is that the Unicode character > >> database UCD (http://www.unicode.org/ucd/) is made available > >> _within_ the image with the full content as needed by the target > >> languages / scripts one wants to deal with. The process of loading the > >> UCD should be made configurable. > >> > >> 3a) a lot of people are interested in the Latin script (and scripts of > >> similar complexity) only. > >> 3b) The UCD data in XML form > >> http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with > >> and without the CJK characters. > >> > >> 4) The next step is to implement normalization > >> (http://www.unicode.org/reports/tr15/#Norm_Forms). Glad to read that > >> you have reached results here with the test data: > >> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. > >> > >> 5) Pharo offers nice inspectors to view dictionaries and ordered > >> collections (table view, drill down) which facilitates the development > >> to table driven algorithms. The data structures and algorithm are do > >> not depend on a particular dialect though and may be ported to Squeak > >> or Cuis. > >> > >> 6) After having implemented normalization, comparison may be > >> implemented. This needs CLDR access (collation, Unicode Common Locale > >> Data Repository, http://cldr.unicode.org/ ). > >> > >> > >> 7) An architecture has the following subsystems > >> > >> 7a) Basic character handling (21(32)bit characters in indexable > >> strings, point 2) > >> 7b) Runtime access to the Unicode Character Database (point 3) > >> 7c) Converters > >> 7d) Normalization (point 4) > >> 7e) CLDR access (point 6) > >> > >> > >> 8) The implementation should be driven by the current needs. > >> > >> An attainable next goal is to release > >> > >> 8a) a StringBuilder utility class for easier construction of test strings > >> i.e. instead of > >> > >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >>> String). > >> > >> do > >> normalizer composeString: > >> (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') > >> > >> and construct some test cases with it which illustrate some basic > >> Unicode issues. > >> > >> 8b) identity testing for major languages (e.g. French, German, > >> Spanish) and scripts of similar complexity. I > >> > >> 8c) to provide some more documentation of past and concurrent efforts. > >> > >> Note: This summary has only covered string manipulation, not rendering > >> on the screen which is a different issue. > >> > >> > >> On 12/16/15, Sven Van Caekenberghe <[email protected]> wrote: > >>> Hi Hannes, > >>> > >>> My detailed comments/answers below, after quoting 2 of your emails: > >>> > >>>> On 10 Dec 2015, at 22:17, H. Hirzel <[email protected]> wrote: > >>>> > >>>> Hello Sven > >>>> > >>>> On 12/9/15, Sven Van Caekenberghe <[email protected]> wrote: > >>>> > >>>>> The simplest example in a common language is (the French letter ?) is > >>>>> > >>>>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] > >>>>> > >>>>> which can also be written as > >>>>> > >>>>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT > >>>>> [U+0301] > >>>>> > >>>>> The former being a composed normal form, the latter a decomposed normal > >>>>> form. (And yes, it is even much more complicated than that, it goes on > >>>>> for > >>>>> 1000s of pages). > >>>>> > >>>>> In the above example, the concept of character/string is indeed fuzzy. > >>>>> > >>>>> HTH, > >>>>> > >>>>> Sven > >>>> > >>>> Thanks for this example. I have created a wiki page with it > >>>> > >>>> I wonder what the Pharo equivalent is of the following Squeak expression > >>>> > >>>> $? asString asDecomposedUnicode > >>>> > >>>> Regards > >>>> > >>>> Hannes > >>> > >>> You also wrote: > >>> > >>>> The text below shows how to deal with the Unicode e acute example > >>>> brought up by Sven in terms of comparing strings. Currently Pharo and > >>>> Cuis do not do Normalization of strings. Limited support is in Squeak. > >>>> It will be shown how NFD normalization may be implemented. > >>>> > >>>> > >>>> Swift programming language > >>>> ----------------------------------------- > >>>> > >>>> How does the Swift programming language [1] deal with Unicode strings? > >>>> > >>>> // "Voulez-vous un caf??" using LATIN SMALL LETTER E WITH ACUTE > >>>> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" > >>>> > >>>> // "Voulez-vous un café?" using LATIN SMALL LETTER E and > >>>> COMBINING ACUTE ACCENT > >>>> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" > >>>> > >>>> if eAcuteQuestion == combinedEAcuteQuestion { > >>>> print("These two strings are considered equal") > >>>> } > >>>> // prints "These two strings are considered equal" > >>>> > >>>> The equality operator uses the NFD (Normalization Form Decomposed)[2] > >>>> form for the comparison appyling a method > >>>> #decomposedStringWithCanonicalMapping[3] > >>>> > >>>> > >>>> Squeak / Pharo > >>>> ----------------------- > >>>> > >>>> Comparison without NFD [3] > >>>> > >>>> > >>>> "Voulez-vous un caf??" > >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>>> asString, '?'. > >>>> > >>>> > >>>> eAcuteQuestion = combinedEAcuteQuestion > >>>> false > >>>> > >>>> eAcuteQuestion == combinedEAcuteQuestion > >>>> false > >>>> > >>>> The result is false. A Unicode conformant application however should > >>>> return *true*. > >>>> > >>>> Reason for this is that Squeak / Pharo strings are not put into NFD > >>>> before testing for equality = > >>>> > >>>> > >>>> Squeak Unicode strings may be tested for Unicode conformant equality > >>>> by converting them to NFD before testing. > >>>> > >>>> > >>>> > >>>> Squeak using NFD > >>>> > >>>> asDecomposedUnicode[4] transforms a string into NFD for cases where a > >>>> Unicode code point if decomposed, is decomposed only to two code > >>>> points [5]. This is so because when initializing [6] the Unicode > >>>> Character Database in Squeak this is a limitation imposed by the code > >>>> which reads UnicodeData.txt [7][8]. This is not a necessary > >>>> limitation. The code may be rewritten at the price of a more complex > >>>> implementation of #asDecomposedUnicode. > >>>> > >>>> "Voulez-vous un caf??" > >>>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>>> asString, '?'. > >>>> > >>>> > >>>> eAcuteQuestion asDecomposedUnicode = > >>>> combinedEAcuteQuestion asDecomposedUnicode > >>>> > >>>> true > >>>> > >>>> > >>>> > >>>> Conclusion > >>>> ------------------ > >>>> > >>>> Implementing a method like #decomposedStringWithCanonicalMapping > >>>> (swift) which puts a string into NFD (Normalization Form D) is an > >>>> important building block towards better Unicode compliance. A Squeak > >>>> proposal is given by [4]. It needs to be reviewed.extended. > >>>> > >>>> It should probably be extended for cases where there are more than > >>>> two code points in the decomposed form (3 or more?) > >>>> > >>>> The implementing of NFD comparison gives us an equality test for a > >>>> comparatively small effort for simple cases covering a large number of > >>>> use cases (Languages using the Latin script). > >>>> > >>>> The algorithm is table driven by the UCD [8]. From this follows an > >>>> simple but important fact for conformant implementations need runtime > >>>> access to information from the Unicode Character Database [UCD][9]. > >>>> > >>>> > >>>> [1] > >>>> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html#//apple_ref/doc/uid/TP40014097-CH7-ID285 > >>>> [2] http://www.unicode.org/glossary/#normalization_form_d > >>>> [3] > >>>> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/index.html#//apple_ref/occ/instm/NSString/decomposedStringWithCanonicalMapping > >>>> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 > >>>> [5] http://www.unicode.org/glossary/#code_point > >>>> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 > >>>> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt > >>>> [8] Unicode Character Database documentation > >>>> http://unicode.org/reports/tr44/ > >>>> [9] http://www.unicode.org/reports/tr23/ > >>> > >>> > >>> Today, we have a Unicode and CombinedCharacter class in Pharo, and there > >>> is > >>> different but similar Unicode code in Squeak. These are too simple (even > >>> though they might work, partially). > >>> > >>> The scope of the original threads is way too wide: a new string type, > >>> normalisation, collation, being cross dialect, mixing all kinds of > >>> character > >>> and encoding definitions. All interesting, but not much will come out of > >>> it. > >>> But the point that we cannot leave proper text string handling to an > >>> outside > >>> library is indeed key. > >>> > >>> That is why a couple of people in the Pharo community (myself included) > >>> started an experimental, proof of concept, prototype project, that aims to > >>> improve Unicode support. We will announce it to a wider public when we > >>> feel > >>> we have something to show for. The goal is in the first place to > >>> understand > >>> and implement the fundamental algorithms, starting with the 4 forms of > >>> Normalisation. But we're working on collation/sorting too. > >>> > >>> This work is of course being done for/in Pharo, using some of the > >>> facilities > >>> only available there. It probably won't be difficult to port, but we can't > >>> be bothered with probability right now. > >>> > >>> What we started with is loading UCD data and making it available as a nice > >>> objects (30.000 of them). > >>> > >>> So now you can do things like > >>> > >>> $? unicodeCharacterData. > >>> > >>> => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" > >>> > >>> $? unicodeCharacterData uppercase asCharacter. > >>> > >>> => "$?" > >>> > >>> $? unicodeCharacterData decompositionMapping. > >>> > >>> => "#(101 769)" > >>> > >>> There is also a cool GT Inspector view: > >>> > >>> > >>> > >>> Next we started implementing a normaliser. It was rather easy to get > >>> support > >>> for simpler languages going. The next code snippets use explicit code > >>> arrays, because copying decomposed diacritics to my mail client does not > >>> work (they get automatically composed), in a Pharo Workspace this does > >>> work > >>> nicely with plain strings. The higher numbers are the diacritics. > >>> > >>> (normalizer decomposeString: 'les ?l?ves Fran?ais') collect: #codePoint > >>> as: > >>> Array. > >>> > >>> => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 > >>> 807 97 105 115)" > >>> > >>> (normalizer decomposeString: 'D?sseldorf K?nigsallee') collect: #codePoint > >>> as: Array. > >>> > >>> => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 > >>> 103 > >>> 115 97 108 108 101 101)" > >>> > >>> normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 > >>> 115 > >>> 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). > >>> > >>> => "'les ?l?ves Fran?ais'" > >>> > >>> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >>> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >>> String). > >>> > >>> => "'D?sseldorf K?nigsallee'" > >>> > >>> However, the real algorithm following the official specification (and > >>> other > >>> elements of Unicode that interact with it) is way more complicated (think > >>> about all those special languages/scripts out there). We're focused on > >>> understanding/implementing that now. > >>> > >>> Next, unit tests were added (of course). As well as a test that uses > >>> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > >>> 75.000 individual test cases to check conformance to the official Unicode > >>> Normalization specification. > >>> > >>> Right now (with super cool hangul / jamo code by Henrik), we hit the > >>> following stats: > >>> > >>> #testNFC 16998/18593 (91.42%) > >>> #testNFD 16797/18593 (90.34%) > >>> #testNFKC 13321/18593 (71.65%) > >>> #testNFKD 16564/18593 (89.09%) > >>> > >>> Way better than the naive implementations, but not yet there. > >>> > >>> We are also experimenting and thinking a lot about how to best implement > >>> all > >>> this, trying out different models/ideas/apis/representations. > >>> > >>> It will move slowly, but you will hear from us again in the coming > >>> weeks/months. > >>> > >>> Sven > >>> > >>> PS: Pharo developers with a good understanding of this subject area that > >>> want to help, let me know and we'll put you in the loop. Hacking and > >>> specification reading are required ;-) > >>> > >>> > >> > >> . > >> > > > > > ------------------------------ > > Message: 13 > Date: Mon, 21 Dec 2015 11:18:33 +0100 > From: Thierry Goubier <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8; format=flowed > > Le 21/12/2015 11:05, Ben Coman a ?crit : > > On Mon, Dec 21, 2015 at 8:30 PM, Denis Kudriashov <[email protected]> > > wrote: > >> Hello > >> > >> 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck <[email protected]>: > >>> > >>> That being said, I of course like the idea of distributing processing > >>> across multiple images. No wonder, my PhD tool (Marea) and related > >>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the > >>> mentioned topic. Dave already mentioned about RemoteTask, which was > >>> something I was about to mention. There is also Seamless project (now used > >>> by Remote Debugger stuff) and quite other tools. Finally, I think this > >>> project may have a big percentage that could be considered as "research". > >>> So > >>> having a PhD / master student working on this would also be nice. In fact, > >>> maybe Denis Kudriashov might work on that topic??? (he is now in RMOD > >>> team). > >> > >> > >> Yes. This is what I'm doing now. At last weeks of this year I plan to > >> finish > >> my improvements to Seamless. Then I will adapt system and tools for remote > >> usage. Main goal is allow work with remote images same way we work locally: > >> remote debugging, browsing, any kind of system navigation, editing code. > > > > Cool. I'm looking forward to maybe one day working like that with a > > minimal image on embedded hardware. > > With ARM and MIPS support coming, this looks closer than ever :) > > Thierry > > > > ------------------------------ > > Message: 14 > Date: Mon, 21 Dec 2015 11:19:25 +0100 > From: Sven Van Caekenberghe <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [squeak-dev] [Unicode] Summary (Re: Unicode > Support // e acute example --> decomposition in Pharo?) > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8 > > Hi Hannes, > > > On 18 Dec 2015, at 14:47, H. Hirzel <[email protected]> wrote: > > > > Hello Sven > > > > Thank you for your report about about your experimental, proof of > > concept, prototype project, that aims to improve Unicode support. > > Please include me in the loop. > > > > Below is is my attempt at summarizing the Unicode discussion of the last > > weeks. > > Excellent summary. > > > Corrections /comments / additions are welcome. > > Not really, it is pretty accurate. > > We are working on all parts of your architecture (point 7 below), with a > focus on correct algorithms, not new representations of strings (the current > ones are mostly fine). > > I can tell you that right now, we are already at 100% of the tests that use > http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > 75.000 individual test cases to check conformance to the official Unicode > Normalization specification. So, NFD, NFKD, NFC and NFKC are done now, in > principle. > > I am reasonably sure that within a week or two, after we clean up and > document everything a bit, we'll be able to announce and show the first > results. > > Happy Holidays ! > > Sven > > > Kind regards > > > > Hannes > > > > > > 1) There is a need for improved Unicode support implemented _within_ > > the image , probably as a library. > > > > 1a) This follows the example of the the Twitter CLDR library (i.e. > > re-implementation of ICU components for Ruby). > > https://github.com/twitter/twitter-cldr-rb > > > > Other languages/libraries have similar approaches > > - dotNet, > > https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeInfo%28v=vs.110%29.aspx) > > - Python https://docs.python.org/3/howto/unicode.html > > - Go http://blog.golang.org/strings > > - Swift, > > https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html > > - Perl http://blog.golang.org/strings > > > > 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This > > is because of security and portability reasons (Eliot Miranda) and > > because of the Smalltalk approach that wants to expose basic > > algorithms in Smalltalk code. In addition the 16bit based ICU library > > does not fit well with the Squeak/Pharo UTF32 model. > > > > 2) The Unicode infrastructure (21(32) bit wide Characters as immediate > > objects, use of UTF-32 internally, indexable strings, UTF8 for outside > > communication, support of code converters) is a very valuable > > foundation which makes algorithms more straightforward at the expense > > of a more memory usage. It not used to its full potential at all > > currently though a lot of hard work has been done. > > > > 3) The Unicode algorithms are mostly table / database driven. This > > means that dictionary lookup is a prominent part of the algorithms. > > The essential building block for this is that the Unicode character > > database UCD (http://www.unicode.org/ucd/) is made available > > _within_ the image with the full content as needed by the target > > languages / scripts one wants to deal with. The process of loading the > > UCD should be made configurable. > > > > 3a) a lot of people are interested in the Latin script (and scripts of > > similar complexity) only. > > 3b) The UCD data in XML form > > http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with > > and without the CJK characters. > > > > 4) The next step is to implement normalization > > (http://www.unicode.org/reports/tr15/#Norm_Forms). Glad to read that > > you have reached results here with the test data: > > http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. > > > > 5) Pharo offers nice inspectors to view dictionaries and ordered > > collections (table view, drill down) which facilitates the development > > to table driven algorithms. The data structures and algorithm are do > > not depend on a particular dialect though and may be ported to Squeak > > or Cuis. > > > > 6) After having implemented normalization, comparison may be > > implemented. This needs CLDR access (collation, Unicode Common Locale > > Data Repository, http://cldr.unicode.org/ ). > > > > > > 7) An architecture has the following subsystems > > > > 7a) Basic character handling (21(32)bit characters in indexable > > strings, point 2) > > 7b) Runtime access to the Unicode Character Database (point 3) > > 7c) Converters > > 7d) Normalization (point 4) > > 7e) CLDR access (point 6) > > > > > > 8) The implementation should be driven by the current needs. > > > > An attainable next goal is to release > > > > 8a) a StringBuilder utility class for easier construction of test strings > > i.e. instead of > > > >> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >> String). > > > > do > > normalizer composeString: > > (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') > > > > and construct some test cases with it which illustrate some basic > > Unicode issues. > > > > 8b) identity testing for major languages (e.g. French, German, > > Spanish) and scripts of similar complexity. I > > > > 8c) to provide some more documentation of past and concurrent efforts. > > > > Note: This summary has only covered string manipulation, not rendering > > on the screen which is a different issue. > > > > > > On 12/16/15, Sven Van Caekenberghe <[email protected]> wrote: > >> Hi Hannes, > >> > >> My detailed comments/answers below, after quoting 2 of your emails: > >> > >>> On 10 Dec 2015, at 22:17, H. Hirzel <[email protected]> wrote: > >>> > >>> Hello Sven > >>> > >>> On 12/9/15, Sven Van Caekenberghe <[email protected]> wrote: > >>> > >>>> The simplest example in a common language is (the French letter ?) is > >>>> > >>>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] > >>>> > >>>> which can also be written as > >>>> > >>>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT > >>>> [U+0301] > >>>> > >>>> The former being a composed normal form, the latter a decomposed normal > >>>> form. (And yes, it is even much more complicated than that, it goes on > >>>> for > >>>> 1000s of pages). > >>>> > >>>> In the above example, the concept of character/string is indeed fuzzy. > >>>> > >>>> HTH, > >>>> > >>>> Sven > >>> > >>> Thanks for this example. I have created a wiki page with it > >>> > >>> I wonder what the Pharo equivalent is of the following Squeak expression > >>> > >>> $? asString asDecomposedUnicode > >>> > >>> Regards > >>> > >>> Hannes > >> > >> You also wrote: > >> > >>> The text below shows how to deal with the Unicode e acute example > >>> brought up by Sven in terms of comparing strings. Currently Pharo and > >>> Cuis do not do Normalization of strings. Limited support is in Squeak. > >>> It will be shown how NFD normalization may be implemented. > >>> > >>> > >>> Swift programming language > >>> ----------------------------------------- > >>> > >>> How does the Swift programming language [1] deal with Unicode strings? > >>> > >>> // "Voulez-vous un caf??" using LATIN SMALL LETTER E WITH ACUTE > >>> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" > >>> > >>> // "Voulez-vous un café?" using LATIN SMALL LETTER E and > >>> COMBINING ACUTE ACCENT > >>> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" > >>> > >>> if eAcuteQuestion == combinedEAcuteQuestion { > >>> print("These two strings are considered equal") > >>> } > >>> // prints "These two strings are considered equal" > >>> > >>> The equality operator uses the NFD (Normalization Form Decomposed)[2] > >>> form for the comparison appyling a method > >>> #decomposedStringWithCanonicalMapping[3] > >>> > >>> > >>> Squeak / Pharo > >>> ----------------------- > >>> > >>> Comparison without NFD [3] > >>> > >>> > >>> "Voulez-vous un caf??" > >>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>> asString, '?'. > >>> > >>> > >>> eAcuteQuestion = combinedEAcuteQuestion > >>> false > >>> > >>> eAcuteQuestion == combinedEAcuteQuestion > >>> false > >>> > >>> The result is false. A Unicode conformant application however should > >>> return *true*. > >>> > >>> Reason for this is that Squeak / Pharo strings are not put into NFD > >>> before testing for equality = > >>> > >>> > >>> Squeak Unicode strings may be tested for Unicode conformant equality > >>> by converting them to NFD before testing. > >>> > >>> > >>> > >>> Squeak using NFD > >>> > >>> asDecomposedUnicode[4] transforms a string into NFD for cases where a > >>> Unicode code point if decomposed, is decomposed only to two code > >>> points [5]. This is so because when initializing [6] the Unicode > >>> Character Database in Squeak this is a limitation imposed by the code > >>> which reads UnicodeData.txt [7][8]. This is not a necessary > >>> limitation. The code may be rewritten at the price of a more complex > >>> implementation of #asDecomposedUnicode. > >>> > >>> "Voulez-vous un caf??" > >>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>> asString, '?'. > >>> > >>> > >>> eAcuteQuestion asDecomposedUnicode = > >>> combinedEAcuteQuestion asDecomposedUnicode > >>> > >>> true > >>> > >>> > >>> > >>> Conclusion > >>> ------------------ > >>> > >>> Implementing a method like #decomposedStringWithCanonicalMapping > >>> (swift) which puts a string into NFD (Normalization Form D) is an > >>> important building block towards better Unicode compliance. A Squeak > >>> proposal is given by [4]. It needs to be reviewed.extended. > >>> > >>> It should probably be extended for cases where there are more than > >>> two code points in the decomposed form (3 or more?) > >>> > >>> The implementing of NFD comparison gives us an equality test for a > >>> comparatively small effort for simple cases covering a large number of > >>> use cases (Languages using the Latin script). > >>> > >>> The algorithm is table driven by the UCD [8]. From this follows an > >>> simple but important fact for conformant implementations need runtime > >>> access to information from the Unicode Character Database [UCD][9]. > >>> > >>> > >>> [1] > >>> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html#//apple_ref/doc/uid/TP40014097-CH7-ID285 > >>> [2] http://www.unicode.org/glossary/#normalization_form_d > >>> [3] > >>> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/index.html#//apple_ref/occ/instm/NSString/decomposedStringWithCanonicalMapping > >>> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 > >>> [5] http://www.unicode.org/glossary/#code_point > >>> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 > >>> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt > >>> [8] Unicode Character Database documentation > >>> http://unicode.org/reports/tr44/ > >>> [9] http://www.unicode.org/reports/tr23/ > >> > >> > >> Today, we have a Unicode and CombinedCharacter class in Pharo, and there is > >> different but similar Unicode code in Squeak. These are too simple (even > >> though they might work, partially). > >> > >> The scope of the original threads is way too wide: a new string type, > >> normalisation, collation, being cross dialect, mixing all kinds of > >> character > >> and encoding definitions. All interesting, but not much will come out of > >> it. > >> But the point that we cannot leave proper text string handling to an > >> outside > >> library is indeed key. > >> > >> That is why a couple of people in the Pharo community (myself included) > >> started an experimental, proof of concept, prototype project, that aims to > >> improve Unicode support. We will announce it to a wider public when we feel > >> we have something to show for. The goal is in the first place to understand > >> and implement the fundamental algorithms, starting with the 4 forms of > >> Normalisation. But we're working on collation/sorting too. > >> > >> This work is of course being done for/in Pharo, using some of the > >> facilities > >> only available there. It probably won't be difficult to port, but we can't > >> be bothered with probability right now. > >> > >> What we started with is loading UCD data and making it available as a nice > >> objects (30.000 of them). > >> > >> So now you can do things like > >> > >> $? unicodeCharacterData. > >> > >> => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" > >> > >> $? unicodeCharacterData uppercase asCharacter. > >> > >> => "$?" > >> > >> $? unicodeCharacterData decompositionMapping. > >> > >> => "#(101 769)" > >> > >> There is also a cool GT Inspector view: > >> > >> > >> > >> Next we started implementing a normaliser. It was rather easy to get > >> support > >> for simpler languages going. The next code snippets use explicit code > >> arrays, because copying decomposed diacritics to my mail client does not > >> work (they get automatically composed), in a Pharo Workspace this does work > >> nicely with plain strings. The higher numbers are the diacritics. > >> > >> (normalizer decomposeString: 'les ?l?ves Fran?ais') collect: #codePoint as: > >> Array. > >> > >> => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 > >> 807 97 105 115)" > >> > >> (normalizer decomposeString: 'D?sseldorf K?nigsallee') collect: #codePoint > >> as: Array. > >> > >> => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 > >> 115 97 108 108 101 101)" > >> > >> normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 > >> 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). > >> > >> => "'les ?l?ves Fran?ais'" > >> > >> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >> String). > >> > >> => "'D?sseldorf K?nigsallee'" > >> > >> However, the real algorithm following the official specification (and other > >> elements of Unicode that interact with it) is way more complicated (think > >> about all those special languages/scripts out there). We're focused on > >> understanding/implementing that now. > >> > >> Next, unit tests were added (of course). As well as a test that uses > >> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > >> 75.000 individual test cases to check conformance to the official Unicode > >> Normalization specification. > >> > >> Right now (with super cool hangul / jamo code by Henrik), we hit the > >> following stats: > >> > >> #testNFC 16998/18593 (91.42%) > >> #testNFD 16797/18593 (90.34%) > >> #testNFKC 13321/18593 (71.65%) > >> #testNFKD 16564/18593 (89.09%) > >> > >> Way better than the naive implementations, but not yet there. > >> > >> We are also experimenting and thinking a lot about how to best implement > >> all > >> this, trying out different models/ideas/apis/representations. > >> > >> It will move slowly, but you will hear from us again in the coming > >> weeks/months. > >> > >> Sven > >> > >> PS: Pharo developers with a good understanding of this subject area that > >> want to help, let me know and we'll put you in the loop. Hacking and > >> specification reading are required ;-) > >> > >> > > > Kind regards > > > > Hannes > > > > > > 1) There is a need for improved Unicode support implemented _within_ > > the image , probably as a library. > > > > 1a) This follows the example of the the Twitter CLDR library (i.e. > > re-implementation of ICU components for Ruby). > > https://github.com/twitter/twitter-cldr-rb > > > > Other languages/libraries have similar approaches > > - dotNet, > > https://msdn.microsoft.com/en-us/library/System.Globalization.CharUnicodeInfo%28v=vs.110%29.aspx) > > - Python https://docs.python.org/3/howto/unicode.html > > - Go http://blog.golang.org/strings > > - Swift, > > https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html > > - Perl http://blog.golang.org/strings > > > > 1b) ICU is _not_ the way to go (http://site.icu-project.org/) . This > > is because of security and portability reasons (Eliot Miranda) and > > because of the Smalltalk approach that wants to expose basic > > algorithms in Smalltalk code. In addition the 16bit based ICU library > > does not fit well with the Squeak/Pharo UTF32 model. > > > > 2) The Unicode infrastructure (21(32) bit wide Characters as immediate > > objects, use of UTF-32 internally, indexable strings, UTF8 for outside > > communication, support of code converters) is a very valuable > > foundation which makes algorithms more straightforward at the expense > > of a more memory usage. It not used to its full potential at all > > currently though a lot of hard work has been done. > > > > 3) The Unicode algorithms are mostly table / database driven. This > > means that dictionary lookup is a prominent part of the algorithms. > > The essential building block for this is that the Unicode character > > database UCD (http://www.unicode.org/ucd/) is made available > > _within_ the image with the full content as needed by the target > > languages / scripts one wants to deal with. The process of loading the > > UCD should be made configurable. > > > > 3a) a lot of people are interested in the Latin script (and scripts of > > similar complexity) only. > > 3b) The UCD data in XML form > > http://www.unicode.org/Public/8.0.0/ucdxml/ offers a download with > > and without the CJK characters. > > > > 4) The next step is to implement normalization > > (http://www.unicode.org/reports/tr15/#Norm_Forms). Glad to read that > > you have reached results here with the test data: > > http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt. > > > > 5) Pharo offers nice inspectors to view dictionaries and ordered > > collections (table view, drill down) which facilitates the development > > to table driven algorithms. The data structures and algorithm are do > > not depend on a particular dialect though and may be ported to Squeak > > or Cuis. > > > > 6) After having implemented normalization, comparison may be > > implemented. This needs CLDR access (collation, Unicode Common Locale > > Data Repository, http://cldr.unicode.org/ ). > > > > > > 7) An architecture has the following subsystems > > > > 7a) Basic character handling (21(32)bit characters in indexable > > strings, point 2) > > 7b) Runtime access to the Unicode Character Database (point 3) > > 7c) Converters > > 7d) Normalization (point 4) > > 7e) CLDR access (point 6) > > > > > > 8) The implementation should be driven by the current needs. > > > > An attainable next goal is to release > > > > 8a) a StringBuilder utility class for easier construction of test strings > > i.e. instead of > > > >> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >> String). > > > > do > > normalizer composeString: > > (StringBuilder construct: 'Du\u0308sseldorf Ko\u0308nigsallee') > > > > and construct some test cases with it which illustrate some basic > > Unicode issues. > > > > 8b) identity testing for major languages (e.g. French, German, > > Spanish) and scripts of similar complexity. I > > > > 8c) to provide some more documentation of past and concurrent efforts. > > > > Note: This summary has only covered string manipulation, not rendering > > on the screen which is a different issue. > > > > > > On 12/16/15, Sven Van Caekenberghe <[email protected]> wrote: > >> Hi Hannes, > >> > >> My detailed comments/answers below, after quoting 2 of your emails: > >> > >>> On 10 Dec 2015, at 22:17, H. Hirzel <[email protected]> wrote: > >>> > >>> Hello Sven > >>> > >>> On 12/9/15, Sven Van Caekenberghe <[email protected]> wrote: > >>> > >>>> The simplest example in a common language is (the French letter ?) is > >>>> > >>>> LATIN SMALL LETTER E WITH ACUTE [U+00E9] > >>>> > >>>> which can also be written as > >>>> > >>>> LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT > >>>> [U+0301] > >>>> > >>>> The former being a composed normal form, the latter a decomposed normal > >>>> form. (And yes, it is even much more complicated than that, it goes on > >>>> for > >>>> 1000s of pages). > >>>> > >>>> In the above example, the concept of character/string is indeed fuzzy. > >>>> > >>>> HTH, > >>>> > >>>> Sven > >>> > >>> Thanks for this example. I have created a wiki page with it > >>> > >>> I wonder what the Pharo equivalent is of the following Squeak expression > >>> > >>> $? asString asDecomposedUnicode > >>> > >>> Regards > >>> > >>> Hannes > >> > >> You also wrote: > >> > >>> The text below shows how to deal with the Unicode e acute example > >>> brought up by Sven in terms of comparing strings. Currently Pharo and > >>> Cuis do not do Normalization of strings. Limited support is in Squeak. > >>> It will be shown how NFD normalization may be implemented. > >>> > >>> > >>> Swift programming language > >>> ----------------------------------------- > >>> > >>> How does the Swift programming language [1] deal with Unicode strings? > >>> > >>> // "Voulez-vous un caf??" using LATIN SMALL LETTER E WITH ACUTE > >>> let eAcuteQuestion = "Voulez-vous un caf\u{E9}?" > >>> > >>> // "Voulez-vous un café?" using LATIN SMALL LETTER E and > >>> COMBINING ACUTE ACCENT > >>> let combinedEAcuteQuestion = "Voulez-vous un caf\u{65}\u{301}?" > >>> > >>> if eAcuteQuestion == combinedEAcuteQuestion { > >>> print("These two strings are considered equal") > >>> } > >>> // prints "These two strings are considered equal" > >>> > >>> The equality operator uses the NFD (Normalization Form Decomposed)[2] > >>> form for the comparison appyling a method > >>> #decomposedStringWithCanonicalMapping[3] > >>> > >>> > >>> Squeak / Pharo > >>> ----------------------- > >>> > >>> Comparison without NFD [3] > >>> > >>> > >>> "Voulez-vous un caf??" > >>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>> asString, '?'. > >>> > >>> > >>> eAcuteQuestion = combinedEAcuteQuestion > >>> false > >>> > >>> eAcuteQuestion == combinedEAcuteQuestion > >>> false > >>> > >>> The result is false. A Unicode conformant application however should > >>> return *true*. > >>> > >>> Reason for this is that Squeak / Pharo strings are not put into NFD > >>> before testing for equality = > >>> > >>> > >>> Squeak Unicode strings may be tested for Unicode conformant equality > >>> by converting them to NFD before testing. > >>> > >>> > >>> > >>> Squeak using NFD > >>> > >>> asDecomposedUnicode[4] transforms a string into NFD for cases where a > >>> Unicode code point if decomposed, is decomposed only to two code > >>> points [5]. This is so because when initializing [6] the Unicode > >>> Character Database in Squeak this is a limitation imposed by the code > >>> which reads UnicodeData.txt [7][8]. This is not a necessary > >>> limitation. The code may be rewritten at the price of a more complex > >>> implementation of #asDecomposedUnicode. > >>> > >>> "Voulez-vous un caf??" > >>> eAcuteQuestion := 'Voulez-vous un caf', 16rE9 asCharacter asString, '?'. > >>> combinedEAcuteQuestion := 'Voulez-vous un cafe', 16r301 asCharacter > >>> asString, '?'. > >>> > >>> > >>> eAcuteQuestion asDecomposedUnicode = > >>> combinedEAcuteQuestion asDecomposedUnicode > >>> > >>> true > >>> > >>> > >>> > >>> Conclusion > >>> ------------------ > >>> > >>> Implementing a method like #decomposedStringWithCanonicalMapping > >>> (swift) which puts a string into NFD (Normalization Form D) is an > >>> important building block towards better Unicode compliance. A Squeak > >>> proposal is given by [4]. It needs to be reviewed.extended. > >>> > >>> It should probably be extended for cases where there are more than > >>> two code points in the decomposed form (3 or more?) > >>> > >>> The implementing of NFD comparison gives us an equality test for a > >>> comparatively small effort for simple cases covering a large number of > >>> use cases (Languages using the Latin script). > >>> > >>> The algorithm is table driven by the UCD [8]. From this follows an > >>> simple but important fact for conformant implementations need runtime > >>> access to information from the Unicode Character Database [UCD][9]. > >>> > >>> > >>> [1] > >>> https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html#//apple_ref/doc/uid/TP40014097-CH7-ID285 > >>> [2] http://www.unicode.org/glossary/#normalization_form_d > >>> [3] > >>> https://developer.apple.com/library/ios/documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/index.html#//apple_ref/occ/instm/NSString/decomposedStringWithCanonicalMapping > >>> [4] String asDecomposedUnicode http://wiki.squeak.org/squeak/6250 > >>> [5] http://www.unicode.org/glossary/#code_point > >>> [6] Unicode initialize http://wiki.squeak.org/squeak/6248 > >>> [7] http://www.unicode.org/Public/UNIDATA/UnicodeData.txt > >>> [8] Unicode Character Database documentation > >>> http://unicode.org/reports/tr44/ > >>> [9] http://www.unicode.org/reports/tr23/ > >> > >> > >> Today, we have a Unicode and CombinedCharacter class in Pharo, and there is > >> different but similar Unicode code in Squeak. These are too simple (even > >> though they might work, partially). > >> > >> The scope of the original threads is way too wide: a new string type, > >> normalisation, collation, being cross dialect, mixing all kinds of > >> character > >> and encoding definitions. All interesting, but not much will come out of > >> it. > >> But the point that we cannot leave proper text string handling to an > >> outside > >> library is indeed key. > >> > >> That is why a couple of people in the Pharo community (myself included) > >> started an experimental, proof of concept, prototype project, that aims to > >> improve Unicode support. We will announce it to a wider public when we feel > >> we have something to show for. The goal is in the first place to understand > >> and implement the fundamental algorithms, starting with the 4 forms of > >> Normalisation. But we're working on collation/sorting too. > >> > >> This work is of course being done for/in Pharo, using some of the > >> facilities > >> only available there. It probably won't be difficult to port, but we can't > >> be bothered with probability right now. > >> > >> What we started with is loading UCD data and making it available as a nice > >> objects (30.000 of them). > >> > >> So now you can do things like > >> > >> $? unicodeCharacterData. > >> > >> => "U+00E9 LATIN SMALL LETTER E WITH ACUTE (LATIN SMALL LETTER E ACUTE)" > >> > >> $? unicodeCharacterData uppercase asCharacter. > >> > >> => "$?" > >> > >> $? unicodeCharacterData decompositionMapping. > >> > >> => "#(101 769)" > >> > >> There is also a cool GT Inspector view: > >> > >> > >> > >> Next we started implementing a normaliser. It was rather easy to get > >> support > >> for simpler languages going. The next code snippets use explicit code > >> arrays, because copying decomposed diacritics to my mail client does not > >> work (they get automatically composed), in a Pharo Workspace this does work > >> nicely with plain strings. The higher numbers are the diacritics. > >> > >> (normalizer decomposeString: 'les ?l?ves Fran?ais') collect: #codePoint as: > >> Array. > >> > >> => "#(108 101 115 32 101 769 108 101 768 118 101 115 32 70 114 97 110 99 > >> 807 97 105 115)" > >> > >> (normalizer decomposeString: 'D?sseldorf K?nigsallee') collect: #codePoint > >> as: Array. > >> > >> => "#(68 117 776 115 115 101 108 100 111 114 102 32 75 111 776 110 105 103 > >> 115 97 108 108 101 101)" > >> > >> normalizer composeString: (#(108 101 115 32 101 769 108 101 768 118 101 115 > >> 32 70 114 97 110 99 807 97 105 115) collect: #asCharacter as: String). > >> > >> => "'les ?l?ves Fran?ais'" > >> > >> normalizer composeString: (#(68 117 776 115 115 101 108 100 111 114 102 32 > >> 75 111 776 110 105 103 115 97 108 108 101 101) collect: #asCharacter as: > >> String). > >> > >> => "'D?sseldorf K?nigsallee'" > >> > >> However, the real algorithm following the official specification (and other > >> elements of Unicode that interact with it) is way more complicated (think > >> about all those special languages/scripts out there). We're focused on > >> understanding/implementing that now. > >> > >> Next, unit tests were added (of course). As well as a test that uses > >> http://www.unicode.org/Public/UNIDATA/NormalizationTest.txt to run about > >> 75.000 individual test cases to check conformance to the official Unicode > >> Normalization specification. > >> > >> Right now (with super cool hangul / jamo code by Henrik), we hit the > >> following stats: > >> > >> #testNFC 16998/18593 (91.42%) > >> #testNFD 16797/18593 (90.34%) > >> #testNFKC 13321/18593 (71.65%) > >> #testNFKD 16564/18593 (89.09%) > >> > >> Way better than the naive implementations, but not yet there. > >> > >> We are also experimenting and thinking a lot about how to best implement > >> all > >> this, trying out different models/ideas/apis/representations. > >> > >> It will move slowly, but you will hear from us again in the coming > >> weeks/months. > >> > >> Sven > >> > >> PS: Pharo developers with a good understanding of this subject area that > >> want to help, let me know and we'll put you in the loop. Hacking and > >> specification reading are required ;-) > >> > >> > > > > > ------------------------------ > > Message: 15 > Date: Mon, 21 Dec 2015 05:19:37 -0500 > From: Robert Withers <[email protected]> > To: [email protected] > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8; format=flowed > > > On 12/21/2015 05:05 AM, Ben Coman wrote: > > On Mon, Dec 21, 2015 at 8:30 PM, Denis Kudriashov <[email protected]> > > wrote: > >> Hello > >> > >> 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck <[email protected]>: > >>> That being said, I of course like the idea of distributing processing > >>> across multiple images. No wonder, my PhD tool (Marea) and related > >>> sub-projects (Fuel, Ghost proxies, etc) have something to do with the > >>> mentioned topic. Dave already mentioned about RemoteTask, which was > >>> something I was about to mention. There is also Seamless project (now used > >>> by Remote Debugger stuff) and quite other tools. Finally, I think this > >>> project may have a big percentage that could be considered as "research". > >>> So > >>> having a PhD / master student working on this would also be nice. In fact, > >>> maybe Denis Kudriashov might work on that topic??? (he is now in RMOD > >>> team). > >> > >> Yes. This is what I'm doing now. At last weeks of this year I plan to > >> finish > >> my improvements to Seamless. Then I will adapt system and tools for remote > >> usage. Main goal is allow work with remote images same way we work locally: > >> remote debugging, browsing, any kind of system navigation, editing code. > > Cool. I'm looking forward to maybe one day working like that with a > > minimal image on embedded hardware. > > I look forward to it. There's some set of capabilities you can't get > with your daily java or any non-image based dev environment, which > leaves Pharo ruling the cloud. Give it a few years. > > Have you thought about SecureSession as the session solution? It does > have advantages and disadvantages, if I may briefly proselytize, > advertise and pitch you on this good news of yours: > > advantages: > a) no certificates used > b) no third parties involved > c) highly secure > d) performant with crypto plugins > e) code visibility: entirely in squeak, no platform code requirements > f) community support of native code > g) forward error correction and compression on the horizon, hopefully. > > disadvantages: > a) not performant without plugins > b) in further development > c) protocol likely will change in a coordinated fashion > d) non-standard protocol > e) non-validated by appropriate federal validators, as good crypto > f) governments cannot crack it. > g) may be slower than existing solution. > > there 7 for and 7 against for a neutral honest appraisal from the > author. Fair dinkum. > > -- > . .. ... ^,^ robert > Go Panthers! > > > > > ------------------------------ > > Message: 16 > Date: Mon, 21 Dec 2015 11:21:55 +0100 > From: "[email protected]" <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAMynPtzipb7MGouWR+Nf=m889D=jbjt0-8nwmca1tzstneg...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Love to read that! > > On Mon, Dec 21, 2015 at 10:30 AM, Denis Kudriashov <[email protected]> > wrote: > > > Hello > > > > 2015-12-20 19:54 GMT+01:00 Mariano Martinez Peck <[email protected]>: > > > >> That being said, I of course like the idea of distributing processing > >> across multiple images. No wonder, my PhD tool (Marea) and related > >> sub-projects (Fuel, Ghost proxies, etc) have something to do with the > >> mentioned topic. Dave already mentioned about RemoteTask, which was > >> something I was about to mention. There is also Seamless project (now used > >> by Remote Debugger stuff) and quite other tools. Finally, I think this > >> project may have a big percentage that could be considered as "research". > >> So having a PhD / master student working on this would also be nice. In > >> fact, maybe Denis Kudriashov might work on that topic??? (he is now in > >> RMOD team). > > > > > > Yes. This is what I'm doing now. At last weeks of this year I plan to > > finish my improvements to Seamless. Then I will adapt system and tools for > > remote usage. Main goal is allow work with remote images same way we work > > locally: remote debugging, browsing, any kind of system navigation, editing > > code. > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151221/a7edeb53/attachment-0001.html> > > ------------------------------ > > Message: 17 > Date: Mon, 21 Dec 2015 15:49:18 +0100 > From: "H. Hirzel" <[email protected]> > To: Pharo Development List <[email protected]> > Cc: Any question about pharo is welcome <[email protected]>, > [email protected] > Subject: Re: [Pharo-dev] [ANN] Multiple Desktop support for Pharo 5 > Message-ID: > <CAGQxfVj0wcQVpFy6AwpBXAjwNYa-zG9h=1jxugwnrecrqdy...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Hello Torsten > > the desktops are named > > 'Desktop 1' > 'Desktop 2' > 'Desktop 3'. > > Is it possible to change these names? > > Regards > Hannes > > On 8/27/15, Torsten Bergmann <[email protected]> wrote: > > Julien Delplanque provided this week a goodie to switch between > > "desktops" - but his initial solution was more or less hiding windows > > and not really switching between real Pharo worlds/desktops. > > > > I gave him some tips what could be done on the pharo-user list. Havent > > heard > > from him afterwards. > > > > Now I was able to spend a few hours on this topic myself and implemented > > a full multiple desktop solution myself. > > > > This works in Pharo 5 only (currently) and requires latest VM (at least on > > Windows) > > from files.pharo.org to get the keyboard shortcuts right. > > > > To try: > > > > Gofer new > > smalltalkhubUser: 'TorstenBergmann' project: 'DesktopManager'; > > configuration; > > loadDevelopment. > > > > The goodie has some nice features like keyboard navigation, world menu > > integration and even a custom spotter with preview of the desktops. > > > > Quick start: > > =========== > > - evaluate the above expression in a Pharo 5 image > > - check the world menu "Desktop" > > - you can press CTRL + D and then CTRL + A (= Desktop Add) to add a new > > desktop > > - you can press CTRL + D and then CTRL + D (= Desktop Desktop) to open the > > overview > > - you can press CTRL + D and then CTRL + P (= Desktop Previous) to navigate > > to the previous desktop > > - you can press CTRL + D and then CTRL + N (= Desktop Next) to navigate to > > the next desktop > > > > The code is hopefully a good example on how to build a custom spotter, > > shortcuts, inspector extensions, ... > > Additionally all this is described in a new article including screenshots > > and a guide on how to use this new goodie: > > > > https://medium.com/@astares/multiple-desktops-for-pharo-5cbc46f3179f > > > > Actually the article took more time to write than coding but I hope > > it helps explaining Pharo and why it is a power tool. > > > > Project is on > > http://www.smalltalkhub.com/#!/~TorstenBergmann/DesktopManager > > Article and code is still subject to change. > > > > Have fun > > T. > > > > ------------------------------ > > Message: 18 > Date: Mon, 21 Dec 2015 15:55:29 +0100 > From: Denis Kudriashov <[email protected]> > To: Discusses Development of Pharo <[email protected]> > Subject: [Pharo-dev] No refactoring for equal/hash generation > Message-ID: > <cag0zxm4esnvgwluy1y4bepuz--uqyaophy_10fg3hkz+3fq...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi. > > There is no menu item in Nautilus to generate equal/hash methods for class. > But there is RBGenerateEqualHashRefactoring class which can do that. And I > remember that in past we had such feature from the browser. > > Should we restore it? I can submit issue for this. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151221/28914ead/attachment-0001.html> > > ------------------------------ > > Message: 19 > Date: Mon, 21 Dec 2015 16:17:23 +0100 > From: Denis Kudriashov <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAG0zXM7f9oVxoyB1M0ZwsM0nfM0A6WJyHLHgoxdYxqfXi-T=g...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > 2015-12-21 11:19 GMT+01:00 Robert Withers <[email protected]>: > > > governments cannot crack it. > > > Why it is disadvantage? > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151221/c8d52013/attachment-0001.html> > > ------------------------------ > > Message: 20 > Date: Mon, 21 Dec 2015 13:20:30 -0300 > From: Mariano Martinez Peck <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: > <CAA+-=mWnV=fDLFTmgK9-R=qonjs9orat6dkv-wwr+snwknk...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Sun, Dec 20, 2015 at 5:21 PM, Thierry Goubier <[email protected]> > wrote: > > > Hi Mariano, > > > > Le 20/12/2015 19:54, Mariano Martinez Peck a ?crit : > > > >> Hi Dimitri, Dave et all. > >> > >> First let me do a little disclaimer. Regarding the developments itself > >> of this sponsorship, we discussed about the topics I said in the first > >> email. However, those are not written in stone and that's why I also > >> asked for feedback. As you know, Pharo has relatively little resources > >> so those should be well spent. If we decide to take a spin into a > >> project X, it doesn't mean other projects are not worth. It simply means > >> that we think X will be the most useful for the time being. > >> > >> One of the final goals of the discusses topics, is to improve a bit the > >> infrastructure for using Pharo as scripting (imagine Coral). For such a > >> thing, we believe that improving FileSystem and OSProcess-like is a good > >> first step to improve the underlying infrastructure. > >> > > > > Would we have access then to a command-line oriented minimal Pharo image? > > > > > Well, just for putting perspective my sponsorship is about 3 full months of > work effort. It doesn't mean I won't work beyond that. But to give you an > idea. I think we all want minimal images , command line etc. And Pharo has > already done a huge effort in that direction. > > > > > That being said, I of course like the idea of distributing processing > >> across multiple images. No wonder, my PhD tool (Marea) and related > >> sub-projects (Fuel, Ghost proxies, etc) have something to do with the > >> mentioned topic. Dave already mentioned about RemoteTask, which was > >> something I was about to mention. There is also Seamless project (now > >> used by Remote Debugger stuff) and quite other tools. Finally, I think > >> this project may have a big percentage that could be considered as > >> "research". So having a PhD / master student working on this would also > >> be nice. In fact, maybe Denis Kudriashov might work on that topic??? (he > >> is now in RMOD team). > >> > >> Also, not that forking the image is not my main short-term goal. As > >> said, I will experiment first with a simple tool to execute OS commands > >> that would be FFI based, trying to make async operations, and that would > >> help in most cases. Maybe for those that are not supported, then > >> fallback in OSProcess. Note that Dave (OSProcess author) is very helpful > >> with me and we are discussing together about this. > >> > > > > My guess is that, at least on Linux/Unix, that FFI will be exactly the > > same as the one used/expressed by the OSProcess plugin, that is > > popen/system. > > > > No, wait. You are getting down to the details so I want to discuss :) > > OSProcess does not use popen()/system() but rather fork()+exec(). That > means that OSProcess has implementing the very very low building blocks > that would allow you do to almost everything. But then it has it's > complicated matters. fork() + exec() cannot be implemented via FFI (this > was already discussed in another thread). However, there is a lot of other > stuff that we believe it could. So... what we wanted to see if it is > possible is to have one only (or very few) primitives (like forkAndExec) > and all the rest via FFI. And then all the features I listed in the first > email of this thread. > > With that in mind, we believe that: > > 1) Before starting dealing with forkAndExec, we could try a first prototype > using popen(). That would allow us test a 100% FFI based approeach, deal > with writing and reading from pipes via stream, face the blocking issue, > see a prototype on how to read async from pipes etc etc etc. If we see this > works nicely, on a second step we can build another approach that instead > of popen() uses forkAndExec primitive. > > 2) We believe that there is a large number of OSProcess users that use > OSProcess only for executing OS commands AND popen() approach would be more > than enough (we will do a survey about this). So, a very simply solution > via FFI for these users rather than having all OSProcess, Command Shell, > the plugins etc, may be worth. > > What do you think? > > > > Now, given some of the difficulties seen when dealing with external > > processes, I'd be wary of the "async" side of things. Random failures > > seemingly linked to signal loss are certainly not something I'd like to > > debug. > > > > > Indeed, it won't be easy I guess. But sometimes the process you execute may > take quite some time and blocking the VM for that period is not a good > choice. So... what would you prefer to solve that? Do you prefer > image-based pooling as OSProcess does when IOPlugin is not used? > > > > Something interesting and powerful would be pseudo-tty support. It would > > allow a Pharo image to maintain an open connection to a shell, for example, > > reducing the cost and latency of launching external commands and > > maintaining a context. > > > > Interesting! > > -- > Mariano > http://marianopeck.wordpress.com > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151221/550d5b37/attachment-0001.html> > > ------------------------------ > > Message: 21 > Date: Mon, 21 Dec 2015 11:20:32 -0500 > From: Robert Withers <[email protected]> > To: Pharo Development List <[email protected]> > Subject: Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Consortium > Sponsored Development Effort > Message-ID: <[email protected]> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > > > On 12/21/2015 10:17 AM, Denis Kudriashov wrote: > > > > 2015-12-21 11:19 GMT+01:00 Robert Withers <[email protected] > > <mailto:[email protected]>>: > > > > governments cannot crack it. > > > > > > Why it is disadvantage? > > Governments stay in power, or exercise collective sovereignty, through > dominion, agency and possession. Loss of control over information, and > by extension currency, affects possession, or having control. They still > have dominion and agency, but without three legs you can build no stool > to sit upon. If loss of control over information and currency undermines > possession, then the concept of the modern nation-state is undermined. I > view this as a negative. > > -- > . .. ... ^,^ robert > Go Panthers! > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/attachments/20151221/cacb3d4f/attachment-0001.html> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Pharo-dev mailing list > [email protected] > http://lists.pharo.org/mailman/listinfo/pharo-dev_lists.pharo.org > > > ------------------------------ > > End of Pharo-dev Digest, Vol 32, Issue 31 > ***************************************** >
