Re: [Pharo-dev] [Pharo-users] [ANN] Success story Mobility Map
> On 25 Sep 2018, at 14:39, Norbert Hartl wrote: > > > >> Am 25.09.2018 um 12:52 schrieb Sven Van Caekenberghe : >> >> Wow. Very nice, well done. >> >> Any chance on some more technical details, as in what 'connected by a >> message queue for the communication' exactly means ? How did you approach >> micro services exactly ? >> > Sure :) Thanks, this is very interesting ! > The installation spawns multiple physical machines. All the machines are > joined to a docker swarm. The installation is reified as either task or > service from the view on the docker swarm. Meaning you instantiate an > arbitrary amount of services and docker swarm distributes them among the > physical machines. Usually you don’t take control which is running where but > you can. At this point you have spread dozens of pharo images among multiple > machines and each of them has an IP address. Furthermore in docker swarm you > have a reification of a network meaning that every instance in a network can > see all other instances on this network. Each service can be reached by its > service name in that network. Docker swarm does all the iptables/firewall and > DNS setup for you. Are you happy with docker swarm's availability/fail-over behaviour ? In other words: does it work when one image/instance goes bad, does it detect and restore the missing functionality ? > In order to have communication between those runtimes we use rabbitmq because > you were so nice writing a driver for it ;) The rabbitmq does have a support > for cluster setup, meaning each of the physical machines has a rabbitmq > installation and they know each other. So it does not matter to which > instance you send messages to and on which you register for receiving > messages. So every pharo image connects to the service rabbitmq and opens a > queue for interaction. Same question: does RabbitMQ's clustering work well under stress/problems ? Syncing all queues between all machines sounds quite heavy (I never tried it, but maybe it just works). > Each service like the car sharing opens a queue e.g. /queue/carSharing and > listens on it. The broker images are stateful so they open queues like > /queue/mobility-map-afdeg32 where afdeg32 is the container id of the instance > (hostname in docker). In each request the queue name to reply is sent as a > header. So we can make sure that the right image gets the message back. This > way we can have sticky sessions keeping volatile data in memory for the > lifecycle of a session. There is one worker image which opens a queue > /queue/mobility-map where session independent requests can be processed. I think I understand ;-) > In order to ease development we are sharing code between the broker and the > micro service. Each micro service has a -Common package where the classes are > in that build the interface. The classes in here are a kind of data entity > facades. They use NeoJSON to map to and from a stream. The class name is send > with the message as a header so the remote side knows what to materialize. > The handling is unified for the four cases > > - Request as inquiry to another micro service > - Response returns values to a Request > - Error is transferred like a Response but is then signalled on the receiving > side > - Notification connects the announcers on the broker and the micro service > side. Yes, makes total sense. > Asynchronous calls we solved using Promises and Futures. Each async call to > the Q becomes a promise (that blocks on #value) and is combined to a future > value containing all promises with support to generate a delta of all > resolved promises. This we need because you issue a search that takes longer > and you want to display results as soon as they are resolved not after all > haven been resolved. Which Promise/Future framework/library are you using in Pharo ? You did not go for single threaded worker images ? > And a lot more. This is a coarse grained overview over the architecture. I’m > happy to answer further questions about this. > > Norbert > >>> On 25 Sep 2018, at 12:20, Norbert Hartl wrote: >>> >>> As presented on ESUG here is the brief description of one of our current >>> projects. >>> >>> Mobility Map >>> —— >>> >>> Mobility Map is a broker for mobility services. It offers multi-modal >>> routing search enabling users to find the best travel options between >>> locations. Travel options include car sharing, bikes, trains, busses etc. >>> Rented cars can be offered for ride sharing on booking time letting other >>> people find it to participate in the ride. Single travel options are >>> combined in travel plans that can be booked and managed in a very easy way. >>> >>> For this project main requirements were scalability to serve a large user >>> base and flexibility to add more additional providers to the broker. The >>> application has been realized using web technologies for the frontend and >>>
[Pharo-dev] [Pharo 7.0-dev] Build #1265: 22479-Imporove-kernel-testing
There is a new Pharo build available! The status of the build #1265 was: SUCCESS. The Pull Request #1821 was integrated: "22479-Imporove-kernel-testing" Pull request url: https://github.com/pharo-project/pharo/pull/1821 Issue Url: https://pharo.fogbugz.com/f/cases/22479 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development/1265/
Re: [Pharo-dev] [ANN] Success story Mobility Map
Thanks for sharing this Norbert. I'm happy for you as a company, and also for Pharo, this should help promoting it to bystanders, the whole set of tools and libraries helps demonstrating the convenience and applicability of Pharo in "modern" designs (docker, swarms, MQ, microservices, and whatnot). Regards Esteban A. Maringolo El mar., 25 sept. 2018 a las 9:40, Norbert Hartl () escribió: > > > > Am 25.09.2018 um 12:52 schrieb Sven Van Caekenberghe : > > > > Wow. Very nice, well done. > > > > Any chance on some more technical details, as in what 'connected by a > message queue for the communication' exactly means ? How did you approach > micro services exactly ? > > > Sure :) > > The installation spawns multiple physical machines. All the machines are > joined to a docker swarm. The installation is reified as either task or > service from the view on the docker swarm. Meaning you instantiate an > arbitrary amount of services and docker swarm distributes them among the > physical machines. Usually you don’t take control which is running where > but you can. At this point you have spread dozens of pharo images among > multiple machines and each of them has an IP address. Furthermore in docker > swarm you have a reification of a network meaning that every instance in a > network can see all other instances on this network. Each service can be > reached by its service name in that network. Docker swarm does all the > iptables/firewall and DNS setup for you. > > In order to have communication between those runtimes we use rabbitmq > because you were so nice writing a driver for it ;) The rabbitmq does have > a support for cluster setup, meaning each of the physical machines has a > rabbitmq installation and they know each other. So it does not matter to > which instance you send messages to and on which you register for receiving > messages. So every pharo image connects to the service rabbitmq and opens a > queue for interaction. > > Each service like the car sharing opens a queue e.g. /queue/carSharing and > listens on it. The broker images are stateful so they open queues like > /queue/mobility-map-afdeg32 where afdeg32 is the container id of the > instance (hostname in docker). In each request the queue name to reply is > sent as a header. So we can make sure that the right image gets the message > back. This way we can have sticky sessions keeping volatile data in memory > for the lifecycle of a session. There is one worker image which opens a > queue /queue/mobility-map where session independent requests can be > processed. > > In order to ease development we are sharing code between the broker and > the micro service. Each micro service has a -Common package where the > classes are in that build the interface. The classes in here are a kind of > data entity facades. They use NeoJSON to map to and from a stream. The > class name is send with the message as a header so the remote side knows > what to materialize. The handling is unified for the four cases > > - Request as inquiry to another micro service > - Response returns values to a Request > - Error is transferred like a Response but is then signalled on the > receiving side > - Notification connects the announcers on the broker and the micro service > side. > > Asynchronous calls we solved using Promises and Futures. Each async call > to the Q becomes a promise (that blocks on #value) and is combined to a > future value containing all promises with support to generate a delta of > all resolved promises. This we need because you issue a search that takes > longer and you want to display results as soon as they are resolved not > after all haven been resolved. > > And a lot more. This is a coarse grained overview over the architecture. > I’m happy to answer further questions about this. > > Norbert > > >> On 25 Sep 2018, at 12:20, Norbert Hartl wrote: > >> > >> As presented on ESUG here is the brief description of one of our > current projects. > >> > >> Mobility Map > >> —— > >> > >> Mobility Map is a broker for mobility services. It offers multi-modal > routing search enabling users to find the best travel options between > locations. Travel options include car sharing, bikes, trains, busses etc. > Rented cars can be offered for ride sharing on booking time letting other > people find it to participate in the ride. Single travel options are > combined in travel plans that can be booked and managed in a very easy way. > >> > >> For this project main requirements were scalability to serve a large > user base and flexibility to add more additional providers to the broker. > The application has been realized using web technologies for the frontend > and pharo for the backend. Using a microservice architecture combined with > a broker it is easy to extend the platform with additional brokers. > Deployment is done using docker swarm for distributing dozens of pharo > images among multiple server machines connected by a message queue for the >
Re: [Pharo-dev] [ANN] Pharo Consortium upgrade to Platinum: ZWEIDENKER
On Tue, 25 Sep 2018 at 20:03, Marcus Denker wrote: > The Pharo Consortium is very happy to announce that ZWEIDENKER has ugraded > to be a Platinum member. > Its great to see your support and commitment to Pharo - in addition to the many direct code contributions. cheers -ben
Re: [Pharo-dev] [ANN] Success story Mobility Map
> Am 25.09.2018 um 12:52 schrieb Sven Van Caekenberghe : > > Wow. Very nice, well done. > > Any chance on some more technical details, as in what 'connected by a message > queue for the communication' exactly means ? How did you approach micro > services exactly ? > Sure :) The installation spawns multiple physical machines. All the machines are joined to a docker swarm. The installation is reified as either task or service from the view on the docker swarm. Meaning you instantiate an arbitrary amount of services and docker swarm distributes them among the physical machines. Usually you don’t take control which is running where but you can. At this point you have spread dozens of pharo images among multiple machines and each of them has an IP address. Furthermore in docker swarm you have a reification of a network meaning that every instance in a network can see all other instances on this network. Each service can be reached by its service name in that network. Docker swarm does all the iptables/firewall and DNS setup for you. In order to have communication between those runtimes we use rabbitmq because you were so nice writing a driver for it ;) The rabbitmq does have a support for cluster setup, meaning each of the physical machines has a rabbitmq installation and they know each other. So it does not matter to which instance you send messages to and on which you register for receiving messages. So every pharo image connects to the service rabbitmq and opens a queue for interaction. Each service like the car sharing opens a queue e.g. /queue/carSharing and listens on it. The broker images are stateful so they open queues like /queue/mobility-map-afdeg32 where afdeg32 is the container id of the instance (hostname in docker). In each request the queue name to reply is sent as a header. So we can make sure that the right image gets the message back. This way we can have sticky sessions keeping volatile data in memory for the lifecycle of a session. There is one worker image which opens a queue /queue/mobility-map where session independent requests can be processed. In order to ease development we are sharing code between the broker and the micro service. Each micro service has a -Common package where the classes are in that build the interface. The classes in here are a kind of data entity facades. They use NeoJSON to map to and from a stream. The class name is send with the message as a header so the remote side knows what to materialize. The handling is unified for the four cases - Request as inquiry to another micro service - Response returns values to a Request - Error is transferred like a Response but is then signalled on the receiving side - Notification connects the announcers on the broker and the micro service side. Asynchronous calls we solved using Promises and Futures. Each async call to the Q becomes a promise (that blocks on #value) and is combined to a future value containing all promises with support to generate a delta of all resolved promises. This we need because you issue a search that takes longer and you want to display results as soon as they are resolved not after all haven been resolved. And a lot more. This is a coarse grained overview over the architecture. I’m happy to answer further questions about this. Norbert >> On 25 Sep 2018, at 12:20, Norbert Hartl wrote: >> >> As presented on ESUG here is the brief description of one of our current >> projects. >> >> Mobility Map >> —— >> >> Mobility Map is a broker for mobility services. It offers multi-modal >> routing search enabling users to find the best travel options between >> locations. Travel options include car sharing, bikes, trains, busses etc. >> Rented cars can be offered for ride sharing on booking time letting other >> people find it to participate in the ride. Single travel options are >> combined in travel plans that can be booked and managed in a very easy way. >> >> For this project main requirements were scalability to serve a large user >> base and flexibility to add more additional providers to the broker. The >> application has been realized using web technologies for the frontend and >> pharo for the backend. Using a microservice architecture combined with a >> broker it is easy to extend the platform with additional brokers. Deployment >> is done using docker swarm for distributing dozens of pharo images among >> multiple server machines connected by a message queue for the communication. >> Pharo supported that scenario very well enabling us the meet the >> requirements with less effort. >> >> Pharo turned out to be a perfect fit to develop the application in a agile >> way. Small development cycles with continuous integration and continuous >> delivery enables fast turnarounds for the customers to validate progress. >> >> This is a screenshot of the search page for multi-modal results: >> >> >> > >
Re: [Pharo-dev] Fraction and ScaledDecimal should be not be serialized as Float in STON format
> On 18 Sep 2018, at 16:45, Sven Van Caekenberghe wrote: > > Of course, we could also extend the basic number parser and allow the native > (stored) notation > > 1/3 > 3/2s2 > > Note how that last one should use the more exact #storeOn: notation, not the > #printOn: > > I'll have to think about this a bit more, as I can really not choose right > now. I am still not sure if we should do this or not, but here is a POC implementation of the above. It implements the following syntax extension (see the class comment of STON for the rest). number int int denominator int denominator scale int frac int exp int frac exp denominator / digits scale s digits STONWriter>>#writeFraction: fraction jsonMode ifTrue: [ self writeFloat: fraction asFloat ] ifFalse: [ writeStream print: fraction numerator; nextPut: $/; print: fraction denominator ] STONWriter>>#writeScaledDecimal: scaledDecimal jsonMode ifTrue: [ self writeFloat: scaledDecimal asFloat ] ifFalse: [ writeStream print: scaledDecimal numerator; nextPut: $/; print: scaledDecimal denominator; nextPut: $s; print: scaledDecimal scale ] Fraction>>#stonOn: stonWriter stonWriter writeFraction: self ScaledDecimal>>#stonOn: stonWriter stonWriter writeScaledDecimal: self STONReader>>parseNumber | negated number | negated := readStream peekFor: $-. number := self parseNumberInteger. (readStream peekFor: $/) ifTrue: [ number := Fraction numerator: number denominator: self parseNumberInteger. (readStream peekFor: $s) ifTrue: [ number := ScaledDecimal newFromNumber: number scale: self parseNumberInteger ] ] ifFalse: [ (readStream peekFor: $.) ifTrue: [ number := number + self parseNumberFraction ]. ((readStream peekFor: $e) or: [ readStream peekFor: $E ]) ifTrue: [ number := number * self parseNumberExponent ] ]. negated ifTrue: [ number := number negated ]. self consumeWhitespace. ^ number Sven
[Pharo-dev] [ANN] Pharo Consortium upgrade to Platinum: ZWEIDENKER
The Pharo Consortium is very happy to announce that ZWEIDENKER has ugraded to be a Platinum member. About - ZWEIDENKER: http://zweidenker.de - Pharo Consortium: http://consortium.pharo.org The goal of the Pharo Consortium is to allow companies and institutions to support the ongoing development and future of Pharo. Individuals can support Pharo via the Pharo Association: - http://association.pharo.org
[Pharo-dev] [Pharo 7.0-dev] Build #1264: 22471 prepare pharo7.0.0 rc1 release
There is a new Pharo build available! The status of the build #1264 was: SUCCESS. The Pull Request #1817 was integrated: "22471 prepare pharo7.0.0 rc1 release" Pull request url: https://github.com/pharo-project/pharo/pull/1817 Issue Url: https://pharo.fogbugz.com/f/cases/22471 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development/1264/
Re: [Pharo-dev] [ANN] Success story Mobility Map
Wow. Very nice, well done. Any chance on some more technical details, as in what 'connected by a message queue for the communication' exactly means ? How did you approach micro services exactly ? > On 25 Sep 2018, at 12:20, Norbert Hartl wrote: > > As presented on ESUG here is the brief description of one of our current > projects. > > Mobility Map > —— > > Mobility Map is a broker for mobility services. It offers multi-modal routing > search enabling users to find the best travel options between locations. > Travel options include car sharing, bikes, trains, busses etc. Rented cars > can be offered for ride sharing on booking time letting other people find it > to participate in the ride. Single travel options are combined in travel > plans that can be booked and managed in a very easy way. > > For this project main requirements were scalability to serve a large user > base and flexibility to add more additional providers to the broker. The > application has been realized using web technologies for the frontend and > pharo for the backend. Using a microservice architecture combined with a > broker it is easy to extend the platform with additional brokers. Deployment > is done using docker swarm for distributing dozens of pharo images among > multiple server machines connected by a message queue for the communication. > Pharo supported that scenario very well enabling us the meet the requirements > with less effort. > > Pharo turned out to be a perfect fit to develop the application in a agile > way. Small development cycles with continuous integration and continuous > delivery enables fast turnarounds for the customers to validate progress. > > This is a screenshot of the search page for multi-modal results: > > >
[Pharo-dev] [Pharo 7.0-dev] Build #1263: Avoid race conditions between tests by removing parallel runs
There is a new Pharo build available! The status of the build #1263 was: SUCCESS. The Pull Request #1824 was integrated: "Avoid race conditions between tests by removing parallel runs" Pull request url: https://github.com/pharo-project/pharo/pull/1824 Issue Url: https://pharo.fogbugz.com/f/cases/22485 Build Url: https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development/1263/
Re: [Pharo-dev] Files-Core and Zinc-Character-Encoding-Core package load sequence
Hi Sven & Guille, On Tue, 25 Sep 2018 at 10:49, Guillermo Polito wrote: > > > > On Mon, Sep 24, 2018 at 11:23 PM Sven Van Caekenberghe wrote: >> >> >> >> > On 24 Sep 2018, at 21:55, Alistair Grant wrote: >> > >> > Hi Guille, Sven, Esteban and Everyone, >> > >> > Can someone (dis)confirm that the Zinc-Character-Encoding-Core package is >> > considered part of the pharo "core" image? >> >> Yes, for me anyway, it is (or should be) part of core. > > > Yes, they are part of the Core image. Simply because an image without any I/O > is an image you cannot interact with, so it's kind of useless :) > And an image doing I/O without managing encoding is pretty useless too... Thanks for the confirmation. I suspected this was the case, but didn't want to be wrong later on. > Also, keep in mind that we try to maintain Zn packages in sync with Sven's > repository, and we try to keep them loadable/compatible with other pharo > versions. > Usually Zn encoders should be stable enough in terms of API and features, but > in case you need to extend them in some way we should keep Sven in the loop > so we can sync both repositories ^^. Sure, at the moment I can't envisage any required changes. Thanks again, Alistair
Re: [Pharo-dev] Files-Core and Zinc-Character-Encoding-Core package load sequence
On Mon, Sep 24, 2018 at 11:23 PM Sven Van Caekenberghe wrote: > > > > On 24 Sep 2018, at 21:55, Alistair Grant wrote: > > > > Hi Guille, Sven, Esteban and Everyone, > > > > Can someone (dis)confirm that the Zinc-Character-Encoding-Core package is > > considered part of the pharo "core" image? > > Yes, for me anyway, it is (or should be) part of core. > Yes, they are part of the Core image. Simply because an image without any I/O is an image you cannot interact with, so it's kind of useless :) And an image doing I/O without managing encoding is pretty useless too... Also, keep in mind that we try to maintain Zn packages in sync with Sven's repository, and we try to keep them loadable/compatible with other pharo versions. Usually Zn encoders should be stable enough in terms of API and features, but in case you need to extend them in some way we should keep Sven in the loop so we can sync both repositories ^^. Guille
Re: [Pharo-dev] update linux info page again, please [CentOS/RHEL]
Hello, I have added it. https://pharo.org/gnu-linux-installation-64 > On 23 Sep 2018, at 18:32, Petr Fischer via Pharo-dev > wrote: > > > From: Petr Fischer > Subject: update linux info page again, please [CentOS/RHEL] > Date: 23 September 2018 at 18:32:00 CEST > To: pharo-dev@lists.pharo.org > > > Hello, > > please, edit CentOS7 installation page: > https://pharo.org/gnu-linux-installation-64 > > and add workaround for CentOS7 section, wihout this symlink, iceberg is > broken: > > (do as root) > cd /usr/lib64 > ln -s libcurl.so.4 libcurl-gnutls.so.4 > > Best solution is to put this in the RPM post installation step (I mean RPM > from official Pharo CentOS repository) - but we will wait for this fix for > months, maybe years, so, workaround info on that page is necessary and easy > to manage. > > More details here: > http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/2018-July/272624.html > > Thanks! pf > > >
[Pharo-dev] [ANN] Pharo Consortium New Bronze Member: Spesenfuchs
The Pharo Consortium is very happy to announce that Spesenfuchs has joined the Consortium as a Bronze Member. About - Spesenfuchs: http://spesenfuchs.de - Pharo Consortium: http://consortium.pharo.org The goal of the Pharo Consortium is to allow companies and institutions to support the ongoing development and future of Pharo. Individuals can support Pharo via the Pharo Association: - http://association.pharo.org
