[Pharo-dev] Re: [ANN] Pharo 12 Release !
Great work! > On 26 Apr 2024, at 12:19, Esteban Lorenzano via Pharo-dev > wrote: > > Dear Pharo users and dynamic language lovers: > We have released Pharo version 12! > What is Pharo? > Pharo is a pure object-oriented programming language and a powerful > environment focused on simplicity and immediate feedback. > • Simple & powerful language: No constructors, no types declaration, no > interfaces, no primitive types. Yet a powerful and elegant language with a > full syntax fitting in one postcard! Pharo is objects and messages all the > way down. > • Live, immersive environment: Immediate feedback at any moment of your > development: Developing, testing, debugging. Even in production environments, > you will never be stuck in compiling and deploying steps again! > • Amazing debugging experience: Pharo environment includes a debugger > unlike anything you've seen before. It allows you to step through code, > restart the execution of methods, create methods on the fly, and much more! > • Pharo is yours: Pharo is made by an incredible community, with more > than 100 contributors for the last revision of the platform and hundreds of > people constantly contributing with frameworks and libraries. > • Fully open-source: Pharo full stack is released under MIT License and > available on GitHub > ... more on the Pharo Features page. > In this iteration of Pharo, we continue working on our objectives of > improvement, clean-up and modularization. Also, we included a number of > usability and speed improvements. A complete list of changes and improvements > is available in our Changelog > Some highlights of this amazing version: > Highlights New breakpoint system > The debug point system is a breakpoint model that supersedes the previous > implementation of breakpoints and watchpoints. They are configurable, > composable, and extensible. The traditional breakpoints remain available, > including conditional breakpoints, one-time breakpoints, and object-centric > breakpoints. Additionally, there are new types of breakpoints, such as > chained-breakpoints, which condition the activation of certain breakpoints on > the triggering of others (e.g., breakpoint B only activates if breakpoint A > is hit first). Debug points also feature a dedicated browser and integration > options. > Tools > • Scalable fluid class syntax is now the default one > • Preparing the introduction of the Bloc graphic system by migrating more > tools to Spec2 widgets > • Spec2 UI framework enhancements to support GTK 4 > • Leaner version of the Metacello package manager > • More robust and strict mode for FFI > System > • New architecture for refactorings and domain specific transformations > • Code loading speed improvement > • Fast browsing via fully optimized package tags > • Optmized memory usage via optimized method protocols > • Compiler simplifications and improvements > Virtual machine > • Massive image support with permanent space > • String/ByteArray comparison speed up > Development Effort > This new version is the result of 1895 Pull Requests integrated just in the > Pharo repository. We have closed 865 issues and received contributions from > more than 70 different contributors. We have also a lot of work in the > separate projects that are included in each Pharo release: > • http://github.com/pharo-spec/NewTools > • http://github.com/pharo-spec/NewTools-DocumentBrowser > • http://github.com/pharo-spec/Spec > • http://github.com/pharo-vcs/Iceberg > • http://github.com/ObjectProfile/Roassal3 > • http://github.com/pillar-markup/Microdown > • http://github.com/pillar-markup/BeautifulComments > • http://github.com/pharo-project/pharo-vm > Contributors > We always say Pharo is yours. It is yours because we made it for you, but > most importantly because it is made by the invaluable contributions of our > great community (yourself). A large community of people from all around the > world contributed to Pharo 12.0 by making pull requests, reporting bugs, > participating in discussion threads, providing feedback, and a lot of helpful > tasks in all our community channels. Thank you all for your contributions. > The Pharo Team > Discover Pharo: https://pharo.org/features > Try Pharo: http://pharo.org/download > Learn Pharo: http://pharo.org/documentation > >
[Pharo-dev] Re: [ANN] success story: ApptiveGrid
Yes, this is a really nice project, the front end as well. It reminds me a bit of FileMaker, but for the web. I am hoping you have commercial success with ApptiveGrid ! > On 12 Dec 2023, at 13:02, Norbert Hartl wrote: > > I wanted to write this for a very long time now…so finally…I’m very proud to > announce a new success story: ApptiveGrid > > ApptiveGrid is a SaaS tool to digitalize and automatize business processes. > > On the one hand ApptiveGrid is visual database that enables you to model your > database via web frontend. At the same time this model is available via REST > API. > > > > On top of the data model a form creator turns your model into a form that you > can send e.g. via email to inquire data from other people… > > > > On the other hand ApptiveGrid is a workflow system where you can define your > work flow in the web frontend and connect to events. These events are either > internal (resulting from a change in your data model) or external where you > can use web hooks to kick of work flows. > > > > With the combination of both parts ApptiveGrid is able to solve many of > modern digital workflows. It enables to make it for low cost and in almost no > time which are two pretty good reasons to use it. If you want to use it just > visit: http://www.apptivegrid.de. ApptiveGrid provides a start plan at no > cost where most of the functionalities are available. If you have additional > needs just write me. For the people in the pharo community I’m sure we can > provide a bit more on top. > > About the tech stack: > > - The business backend of ApptiveGrid is 100% pharo > - It uses https://github.com/svenvc/zinc components for the HTTP frontend > - and https://github.com/ApptiveGrid/Soil as the persistence solution. > - Each user has its own database (an empty soil database is 24kb on disk, > could be made even approx. 5kb). There are over 3100 databases in the system > right now > - one pharo image holds multiple soil databases open and provides memory > caching for the objects > - routing of requests is done with haproxy connection persistence > - the web frontend is made with Vue.js and a pharo library that we transpile > to JS with PharoJS > > If you are interested you can also watch my ESUG videos at > https://www.youtube.com/@esugboard. All talks in the last 2 years were about > ApptiveGrid > > If you have question, don’t hesitate to ask, I’m happy to answer! > > Norbert >
[Pharo-dev] Re: Stack overflow support?
This is super important. I always thought that this was an implementation problem in the sense that the overflow check was too costly/slow. Anyway, in most Common Lisp implementations the stack is also limited. When you hit the limit, a resumable exception is raised, allowing you to make the stack larger and continue. And of course you can set a large initial stack size per process. > On 9 Nov 2023, at 10:13, Guillermo Polito wrote: > > Hi all, > > We started (with many interruptions over the last months) working a bit with > Stephane on understanding what is the (positive and negative) impact of > stack-overflow support in Pharo. > The key idea is that if a process consumes too much stack (potentially > because of an infinite recursion) then the process should stop with an > exception. > > ## Why we want better stack consumption control > > This idea comes up to solve issues that are pretty common and hit especially > newbies. > For example, imagine you accidentally write an accessor such as > > ``` > A >> foo >^ self foo > ``` > > Students do this all the time, and I’ve also seen it in experienced people > who go too fast :). > More importantly, such recursions could happen also with not-so-obvious > indirect recursions (a sends b, b sends c, c sends a), and these could hit > anybody. > > This is aggravated because the current execution model allows us to have > infinite stacks —meaning: limited by available memory only. > This is indeed a nice feature for many use cases but it has its own drawbacks > when one of these kind of recursions are hit: > - code just loops forever taking space in the stack > - when there is no more stack space, context objects are created and moved > to the heap > - but those contexts are strongly held, so they are never GCed and take up > extra space > - even worse! they are there adding more work to the GC every time and > making the GC run more often looking for space that is not there > > ## Why Ctrl-dot does not always work > > Of course, super users know there is this “Ctrl dot” hidden feature that > should help you recover from this. > First, let's take out of the equation that this is only known by super users. > Now, in this situation, when Ctrl-dot is hit it will trigger a handler that > suspends the problematic process and opens a debugger on it. > But it could happen that, > - the stack is so big that the debugger is very sluggish (best-case scenario) > - the VM is just flooded doing GCs so maybe the Ctrl dot event does not even > arrive at Pharo or the trigger > - if the recursion is hit when printing an object (which is more common than > you could imagine), opening the debugger could trigger a new recursion and > never give back the control to the user > > ## What are we working on > > The main idea here is: Can we have a simple and efficient way to prevent such > kinds of situations? > > After many discussions around detecting recursion, we kinda arrived at the > simple solution of just detecting a stack overflow. > The solution is easy to understand (because it’s like other languages work) > and easy to implement because there is already support for that. > But this leaves open two questions: > - what happens when people want to use the “infinite stack” feature? > - when should a process stack overflow? What is a sensitive default value? > > Our draft implementation here > https://github.com/pharo-project/pharo-vm/pull/710 does the following to cope > with this: > - we can now parametrize the size of the stack (of each stack page to be > more accurate) when the VM starts up > - the stack overflow check can be disabled per process > > We also are running experiments to see what could be a sensitive stack size > for our normal usages. Here, for example, we ran almost all test cases in > Pharo separately (one suite per line below), and we observed how many tests > broke (x-axis) with different stack sizes (y-axis). > Here we see that most test suites require at least 20-24k to run properly, > some go up to 36k of stack before converging (i.e., the number of broken > tests does not change). > > > You’ll notice in the graph that There are some scenarios that break all the > time. This is because exception handling itself is recursive and may produce > more stack overflows depending on the size of the stack between the exception > and the exception handler. > So some more work is still required, mostly changing Pharo libraries to > properly support this. For example: > - should tests run in a fresh process with a fresh stack? > - should the exception mechanism use less recursion? > - resumable exceptions add stack pressure because they do not “unstack” > until the exception is finally handled, meaning that the stack used by > exception handling just adds up to the stack of the original code, can we do > better here? > > Probably there are more interesting questions here, that’s the “why"
[Pharo-dev] Re: Sockets not working under heavy VM load?
I know this is very old, but is this still an issue today ? > On 29 Dec 2021, at 12:30, Pavel Krivanek wrote: > > Hi, > > we have some performance issues with Glorp over P3 and, after some > investigation, I have a theory that could explain it. And I would like to > know your opinion if it is correct. > > To reproduce this issue, you can try something easy. Just create a process > with normal or higher priority that will try to read something from a socket. > And, in parallel, do in a separate thread an operation that takes all the > processor time for several seconds. > > > [ > 5 seconds wait. > Time now traceCr. > ZnClient new > url: 'https://www.google.com'; > get. > Time now traceCr. > ] forkAt: 60. > > [1000 timesRepeat: [ 1 factorial ] ] timeToRun > > (you may try a different safe server address) > > The processes run in parallel until the time when the first process tries to > obtain data from the socket. Then it hangs and waits until finishing of the > second process. > > My theory is that the VM socket plugin is able to receive incoming data from > the connection and save it into the buffer. In the meantime, the reading > process is waiting on the socket semaphore. But it looks like the plugin is > not able to release the semaphore while the VM is actually doing something. > It needs to wait until the VM has nothing useful to do, and then it is > finally able to release the semaphores so Pharo processes are resumed and can > start to read. > > Is my theory right? > > Cheers, > -- Pavel > > >
[Pharo-dev] Re: [ANN] CBOR for Pharo
> On 23 Jun 2023, at 00:06, stephane ducasse wrote: > > Super nice > for a moment I thought that you forgot the final G of the name :) Haha, that would have been nice. It is a bit of a weird acronym. I must admit that I did not hear about it until recently. There are several binary JSON specifications, but this one has an elegance to it, I liked it immediately. And the best way to understand it or get a feel for it is to implement it. From time to time I need to do a project like this: small scope, clear spec, no drama. I would also like to iterate that doing binary coding in Pharo is really nice, elegant and powerful. >> On 21 Jun 2023, at 11:25, Sven Van Caekenberghe wrote: >> >> [ANN] CBOR for Pharo >> >> https://github.com/svenvc/CBOR >> >> Concise Binary Object Representation (CBOR) is a binary data serialization >> format. CBOR is based on the JSON data model: a number of primitive types >> (integers, floats, booleans, strings and null), lists and maps (which >> represent objects or structures). Being binary, CBOR is more efficient than >> JSON, which is text based. CBOR also supports binary data and optional >> extensions (called tags). This implementation has native support for string >> and epoch based timestamps (DateAndTime) and big integers (Integer). >> >> This project contains encoding/decoding support for the half-precision, IEEE >> 754 16-bit (binary16) floating-point format. >> >> Sven >> >> >> -- >> Sven Van Caekenberghe >> Proudly supporting Pharo >> http://pharo.org >> http://association.pharo.org >> http://consortium.pharo.org >> >> >>
[Pharo-dev] [ANN] CBOR for Pharo
[ANN] CBOR for Pharo https://github.com/svenvc/CBOR Concise Binary Object Representation (CBOR) is a binary data serialization format. CBOR is based on the JSON data model: a number of primitive types (integers, floats, booleans, strings and null), lists and maps (which represent objects or structures). Being binary, CBOR is more efficient than JSON, which is text based. CBOR also supports binary data and optional extensions (called tags). This implementation has native support for string and epoch based timestamps (DateAndTime) and big integers (Integer). This project contains encoding/decoding support for the half-precision, IEEE 754 16-bit (binary16) floating-point format. Sven -- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
[Pharo-dev] Re: Equality/Hash Modification and Image Rehashing
A use case in the system is #stonPostReferenceResolution for Dictionary and Set. You could have a look at that (it is an edge case, but still). > On 30 Aug 2022, at 19:00, s...@clipperadams.com wrote: > > From Pharo Discord: > > IIUC when I change the comparison behavior of a class (i.e. reimplement #= > and #hash), sets containing those objects start acting oddly e.g. happily > storing duplicates of equal objects and failing to find objects they contain. > A solution seems to be Set allSubInstances do: #rehash. > > Is my understanding correct? If so, I find it odd that I've been using and > researching Smalltalk for well over a decade and I never remember > reading/hearing about this. I wonder if it can be baked into the IDE somehow > e.g. via a critic... > > And stephan responded: > > The mailing list posts about that I remember were at the time where the > number of object bits used for the hash were increased. > > Following up on his clue, I found this old post, which seems to confirm my > suspicion. This seems like a serious trap for users to fall into (I’m having > flashbacks to my C++ days lol). So, can/should we somehow handle this in the > IDE? Something like a refactoring (that has a confirmation window) that gives > you the choice to rehash image collections when redefining these methods? > > In a 1GB image, `Set allSubInstances do: #rehash` is almost instantaneous and > `Dictionary allSubInstances do: #rehash` takes several seconds. >
[Pharo-dev] Re: about object serialization and deserialization
Hi Stef, > On 15 Aug 2022, at 20:25, stephane ducasse wrote: > > Hi guys > > It is interesting to think about the design and use of object deserializers. > > https://www.slideshare.net/frohoff1/deserialize-my-shorts-or-how-i-learned-to-start-worrying-and-hate-java-object-deserialization > > <2204.09388.pdf> Interesting, and indeed, a big problem (especially denial of service attacks). Step one is to be conscious of the problem (which exists across all language/formats). Step two could be to add some sanity checks (limits) to parsers. Would be a nice subject for a (student) project. Sven
[Pharo-dev] Re: About Image Size Pharo9->Pharo11
Great work, Marcus. > On 11 May 2022, at 09:00, Marcus Denker wrote: > > Pharo9: 65,8 MB > Pharo10: 57,4 MB > Pharo11: 56,5 MB > > The space saving come from three things: > > 1) Code cleanup Pharo9-Pharo10 > = > > e.g. removal of support in Kernel and Compiler of old style blocks and > bytecodes, > to mention one smaller (in term of image size) but deep cleanup. > > Of course, we can do more here. There is still a lot of dead code (and > duplicated > things) in the image! > > > 2) Global Literal Sharing > === > > Literals are now, when the method is compiled, set to be read-only, > recursively in the case of arrays. > > This allows us to unify (share one instance) not only in the same method, but > globally over all methods. > > As the compiler can not do a global literal table when compiling a single > method (too slow), this is implemented > as an additional pass, called as part of the build process. > > If you load a lot of code, it might be an idea to do that as part of your > build again: > > ImageCleaner shareLiterals > > > 3) Improvement of code structure objects > === > > e.g, in Pharo10 we found two empty OrderedCollections per class that where > not needed. > > Further work on this idea is where the improvement in Pharo11 comes from, we > merged these three changes: > > - Class>>#classPool: empty dictionary for all classes, but not many define > class vars #11172 > https://github.com/pharo-project/pharo/issues/11172 > > - [Memory] sharedPools OrderedCollections waste memory #10957 > https://github.com/pharo-project/pharo/issues/10957 > > - [Memory] AllProtocol uses a lot of memory #10963 > https://github.com/pharo-project/pharo/issues/10963 > > > The classPools/sharedPool improves the size of the bootstrapped image not to0 > much as the boostrap creates > classes with nil there, but for newly loaded code this will be more visible. > > This direction has some more fairly simple next steps: > > - [Memory] commentSourcePointer is not needed for MetaClasses > https://github.com/pharo-project/pharo/issues/10958 > > - [Memory] Every class and meta class has both a ProtocolOrganizer and a > ClassOrganization #10959 > https://github.com/pharo-project/pharo/issues/10959 > > > ... and of course there are many more things to find with just looking a bit. > > (And it is interesting starting point to think about better language support > for making long living but rarely changed > data structures more memory efficient) > > If you want to get an impression of which classes use image space, > "SpaceTally printSpaceAnalysis" > prints a per-class analysis. > > Marcus
[Pharo-dev] Re: [ANN] Pharo 10 released!
Great news. A big thank you to all those involved ! > On 5 Apr 2022, at 12:39, Esteban Lorenzano wrote: > > Dear Pharo users and dynamic language lovers: > > We have released Pharo version 10 ! > > Pharo is a pure object-oriented programming language and a powerful > environment, focused on simplicity and immediate feedback. > > > > Pharo 10 was a short iteration where we focused mainly on stability and > enhancement of the environment : > > • Massive system cleanup > • > • gained speed > • removed dead code > • removed old/deprecated frameworks (Glamour, GTTools, Spec1) > • All Remaining tools written using the deprecated frameworks have been > rewritten: Dependency Analyser, Critique Browser, and many other small > utilities. > • Modularisation has made a leap, creating correct baselines (project > descriptions) for many internal systems, making possible the work and > deployment of minimal images. > • Removing support for the old Bytecode sets and embedded blocks > simplified the compiler and language core. > • As a result, our image size has been reduced by 10% (from 66MB to > 58MB) > • The VM has also improved in several areas: better async I/O support, > socket handling, FFI ABI, > Even being a short iteration, we have closed a massive amount of issues: > around 600 issues and 700 pull requests. A more extended changelog can be > found at > https://github.com/pharo-project/pharo-changelogs/blob/master/Pharo100ChangeLogs.md. > > While the technical improvements are significant, still the most impressive > fact is that the new code that got in the main Pharo 10 image was contributed > by more than 80 people. > > Pharo is more than code. It is an exciting project involving a great > community. > > We thank all the contributors to this release: > > Aaron Bieber, Ackerley Tng, Alban Benmouffek, Alejandra Cossio, Aless Hosry, > Alexandre Bergel, Aliaksei Syrel, Alistair Grant, Arturo Zambrano, Asbathou > Biyalou-Sama, Axel Marlard, Bastien Degardins, Ben Coman, Bernardo Contreras, > Bernhard Pieber, Carlo Teixeira, Carlos Lopez, Carolina Hernandez, Christophe > Demarey, Clotilde Toullec, Connor Skennerton, Cyril Ferlicot, Dave Mason, > David Wickes, Denis Kudriashov, Eric Gade, Erik Stel, Esteban Lorenzano, > Evelyn Cusi Lopez, Ezequiel R. Aguerre, Gabriel Omar Cotelli, Geraldine > Galindo, Giovanni Corriga, Guille Polito, Himanshu, Jan Bliznicenko, Jaromir > Matas, Kasper Østerbye, Kausthub Thekke Madathil, Konrad Hinsen, Kurt > Kilpela, Luz Paz, Marco Rimoldi, Marcus Denker, Martín Dias, Massimo > Nocentini, Max Leske, Maximilian-ignacio Willembrinck Santander, Miguel > Campero, Milton Mamani Torres, Nahuel Palumbo, Norbert Hartl, Norm Green, > Nour Djihan, Noury Bouraqadi, Oleksandr Zaitsev, Pablo Sánchez Rodríguez, > Pablo Tesone, Pavel Krivanek, Pierre Misse-Chanabier, Quentin Ducasse, > Raffaello Giulietti, Rakshit, Renaud de Villemeur, Rob Sayers, Roland > Bernard, Ronie Salgado, Santiago Bragagnolo, Sean DeNigris, Sebastian Jordan > Montt, Soufyane Labsari, Stephan Eggermont, Steven Costiou, Stéphane Ducasse, > Sven Van Caekenberghe, Theo Rogliano, Thomas Dupriez, Théo Lanord, Torsten > Bergmann, Vincent Blondeau. > > > (If you contributed to Pharo 10 development in any way and we missed your > name, please send us an email and we will add you). > > Enjoy! > > The Pharo Team > > Discover Pharo: https://pharo.org/features > > Try Pharo: http://pharo.org/download > > Learn Pharo: http://pharo.org/documentation
[Pharo-dev] Re: This week (10/2022) on the Pharo Issue Tracker
> On 11 Mar 2022, at 09:28, Tim Mackinnon wrote: > > This is really good stuff - we don't say thank you to everyone enough. > > Its easy to add new cool stuff... to safely fix ongoing history, and learn > from old mistakes to create better tested solutions is really hard - and a > testament to the durability and hard work of this community. > > Wow. +100 The continued hard infrastructure work is indeed totally amazing. Thanks you all ! > Tim > > On Fri, 11 Mar 2022, at 7:15 AM, Marcus Denker wrote: >> We merged 3 PRs and fixed 4 Issue tracker entries. >> >> Especially #10972 is important: with this the bug that lead to an >> overly huge image size is fixed. >> >> Now one can actually see that Pharo10 is smaller than Pharo9 due to: >> >> - code cleanup: removal of dead code, simplifications, cleanups >> - sharing of literals across all methods as part of the image build >> - first small improvement of size of the data structures of the code >> (related to Protocol) >> (much more possible here, but it has to wait for Pharo11) >> >> >> There are more PRs ready for Pharo10 that just need a review to be merged: >> https://github.com/pharo-project/pharo/pulls >> >> >> >> Fixes >> >> >> 10667-Installed-breakpoints-not-shown-in-UI-after-AST-cache-reset-during-image-saving >> >> #10962 >> https://github.com/pharo-project/pharo/pull/10962 >> >> ensure no literal is kept pinned in memory when building an image #10972 >> https://github.com/pharo-project/pharo/pull/10972 >> >> Extend class comment of Dictionary #10973 >> https://github.com/pharo-project/pharo/pull/10973
[Pharo-dev] [ANN] UTF8String
Hi, I recently revised some code that I wrote in 2019 and prepared it for publication. UTF8String is a proof of concept / prototype alternative String implementation for Pharo using a variable length UTF8 encoded internal representation. There is a writeup with more details in the README on GitHub: https://github.com/svenvc/UTF8String The public methods are commented and there are some basic unit tests. Just like the experiment of a SubString class that is just a [from,to] pair on an existing (readonly) String (which I believe someone once did, but I can't remember the details), this is another idea that might maybe one day be useful. Sven
[Pharo-dev] [ANN] Pharo Oracle Call Interface
Hi, Pharo Oracle Call Interface (https://github.com/Ironirc/Pharo-Oracle-Call-Interface) is a new project that is currently under development. It was started and implemented by Ivo Roefs. This library allows you to connect to an Oracle SQL RDBMS from Pharo (versions 8 and up). It provides a binding with Oracle's OCI (also known as the Oracle Call Interface) using FFI. Usage Once correctly installed on your local computer, you use OciClient to interact with the Oracle database: | client rows | client := OciClient new environmentString: 'pharotst202112_medium'; username: 'SCOTT'; password: 'TIGER'; yourself. rows := client query: 'SELECT * FROM emp'. client close. rows. Installation To install this library you clone the repository and load BaselineOfPharoOracleCallInterface. This can be done using the Iceberg UI tools or with the following script: Metacello new baseline: 'PharoOracleCallInterface'; repository: 'github://Ironirc/Pharo-Oracle-Call-Interface:main/'; load. You have to make sure that the Pharo VM can access/load the necessary native library for your client side OS platform. The easiest way to get started is using Oracle Instant Client and an Oracle Cloud account. You also have to set up and configure access, contact your DBA for help. Status This project is new and currently under development. It should not be used for production applications. Basic queries are working on 32 and 64 bit Windows and 64 bit macOS. We are looking for collaborators, fellow developers who can help. Ivo & Sven
[Pharo-dev] Re: Problem with Pharo 10 in GitHub Action CI using SmalltalkCI
Yes it is working again. > On 22 Feb 2022, at 16:39, Martin Dias wrote: > > Hi, > we had this problem in Bloc but looks fixed now (after re-run job, it's > green). > > Thanks, > Martín > > On Tue, Feb 22, 2022 at 7:30 AM Sven Van Caekenberghe wrote: > Yes, could be. > > I just triggered a manual CI build for Zinc HTTP Components and it now also > fails, while it succeeded yesterday (there were no code changes). > > So there definitely is a problem. > > Anyone else seeing problems CI building their projects for Pharo 10 ? > > > On 22 Feb 2022, at 11:16, Marcus Denker wrote: > > > > > > > >> On 22 Feb 2022, at 11:03, Sven Van Caekenberghe wrote: > >> > >> Hi, > >> > >> I just tried to enable Pharo64-10 (not Pharo64-10.0 but that is probably > >> another issue) in a GitHub Action CI using SmalltalkCI. > >> > >> https://github.com/svenvc/ston/blob/master/.github/workflows/CI.yml > >> https://github.com/svenvc/ston/runs/5286121566?check_suite_focus=true > >> > >> This fails with > >> > >> NotFound: [ :cl | cl isPlatformCompatible ] not found in OrderedCollection > >> OrderedCollection(Collection)>>errorNotFound: > >> [self errorNotFound: aBlock] in OrderedCollection(Collection)>>detect: in > >> Block: [self errorNotFound: aBlock] > >> OrderedCollection(Collection)>>detect:ifFound:ifNone: > >> OrderedCollection(Collection)>>detect:ifNone: > >> OrderedCollection(Collection)>>detect: > >> SmalltalkCI class>>platformClass > >> ... > >> > >> Does anyone have any idea ? > >> I seem to remember there were some issue with Metacello attributes, but > >> maybe that is not related. > >> > > > > maybe isPlatformCompatible checks the attributes? These where updated to be > > pharo10, not pharo9: > > > > New try at removing Pharo 9 metacello attributes #10819 > > https://github.com/pharo-project/pharo/pull/10819 > > > > > > Marcus
[Pharo-dev] Re: Problem with Pharo 10 in GitHub Action CI using SmalltalkCI
Yes, could be. I just triggered a manual CI build for Zinc HTTP Components and it now also fails, while it succeeded yesterday (there were no code changes). So there definitely is a problem. Anyone else seeing problems CI building their projects for Pharo 10 ? > On 22 Feb 2022, at 11:16, Marcus Denker wrote: > > > >> On 22 Feb 2022, at 11:03, Sven Van Caekenberghe wrote: >> >> Hi, >> >> I just tried to enable Pharo64-10 (not Pharo64-10.0 but that is probably >> another issue) in a GitHub Action CI using SmalltalkCI. >> >> https://github.com/svenvc/ston/blob/master/.github/workflows/CI.yml >> https://github.com/svenvc/ston/runs/5286121566?check_suite_focus=true >> >> This fails with >> >> NotFound: [ :cl | cl isPlatformCompatible ] not found in OrderedCollection >> OrderedCollection(Collection)>>errorNotFound: >> [self errorNotFound: aBlock] in OrderedCollection(Collection)>>detect: in >> Block: [self errorNotFound: aBlock] >> OrderedCollection(Collection)>>detect:ifFound:ifNone: >> OrderedCollection(Collection)>>detect:ifNone: >> OrderedCollection(Collection)>>detect: >> SmalltalkCI class>>platformClass >> ... >> >> Does anyone have any idea ? >> I seem to remember there were some issue with Metacello attributes, but >> maybe that is not related. >> > > maybe isPlatformCompatible checks the attributes? These where updated to be > pharo10, not pharo9: > > New try at removing Pharo 9 metacello attributes #10819 > https://github.com/pharo-project/pharo/pull/10819 > > > Marcus
[Pharo-dev] Problem with Pharo 10 in GitHub Action CI using SmalltalkCI
Hi, I just tried to enable Pharo64-10 (not Pharo64-10.0 but that is probably another issue) in a GitHub Action CI using SmalltalkCI. https://github.com/svenvc/ston/blob/master/.github/workflows/CI.yml https://github.com/svenvc/ston/runs/5286121566?check_suite_focus=true This fails with NotFound: [ :cl | cl isPlatformCompatible ] not found in OrderedCollection OrderedCollection(Collection)>>errorNotFound: [self errorNotFound: aBlock] in OrderedCollection(Collection)>>detect: in Block: [self errorNotFound: aBlock] OrderedCollection(Collection)>>detect:ifFound:ifNone: OrderedCollection(Collection)>>detect:ifNone: OrderedCollection(Collection)>>detect: SmalltalkCI class>>platformClass ... Does anyone have any idea ? I seem to remember there were some issue with Metacello attributes, but maybe that is not related. Sven PS: I could try with other repositories but that will probably give the same result. Only Zinc does run with Pharo 10, last build a day ago. I can't spot the difference.
[Pharo-dev] Re: ZTimestamp now has nanosecond precision
Hi Stef, ZTimestamp is meant to be used as an alternative/replacement for DateAndTime. That means that if you need a timestamp representation for which you would normally use DateAndTime, you can use now ZTimestamp. It is more efficient as it is half as large in memory and most operations are faster. It is also simpler. However, it is not 100% the same. It does not hold timezone information and is always in UTC, not your local timezone. The idea is that you only convert to/from another timezone when you need to, when reading/writing or in a UI. There is full support for that, but it is a specific approach. I am not advocating that ZTimestamp should replace DateAndTime in the standard image. I just would like people to be aware that there are alternatives with nice properties (including example based formatting/parsing and full timezone support, both in separate classes). Sven > On 30 Jan 2022, at 13:36, stephane ducasse wrote: > > Hi sven > > Ideally I would like to see the core of Pharo to improve. > In the past I was thinking that we could have a better “time” package but it > would become too large. > So now I would like to have > > - possibility to have a nice and compact kernel for pharo internal > - nice loadable libraries such as aconcagua or others > > Now the point of my email. > I would like to see some Z component such as ZTimestamp or ZUrl replace what > we have. > > Would it make sense > to rename ZTimestamp as DateAndTime and put in pharo? > or to replace DateAndTime but Ztimestamp renamed Timestamp? > > Now to me when I think about Date and Time I do not think about timestamp but > may be they have the same API at the end. > > S > >> On 27 Jan 2022, at 10:28, Sven Van Caekenberghe wrote: >> >> Hi, >> >> Last year, I extended ZTimestamp with a default nanosecond precision (it >> previously had whole second precision). This makes it a more complete >> replacement for DateAndTime. >> >> https://github.com/svenvc/ztimestamp >> https://github.com/svenvc/ztimestamp/commit/de6c5a670ed0c6cb18894b4917383dd9d1dbe503 >> >> The goals of ZTimestamp are to be simpler and more efficient (both faster >> and more compact). A ZTimestamp instance occupies half the space of a >> DateAndTime instance, its two slots being SmallIntegers. Back in the 32-bit >> days, the seconds since midnight field was too small for nanoseconds. Now, >> 64-bit is the default and there is enough room and no speed penalty for >> working with nanoseconds since midnight. >> >> In casual use, you won't see much of a difference, apart from the added >> detail: >> >> ZTimestamp now. "2022-01-27T09:21:20.528695Z" >> >> There are ways to convert to the older precision: >> >> ZTimestamp now truncated. >> ZTimestamp now rounded. >> ZTimestamp nowTruncated. >> >> Obviously, other methods and classes now support this added precision (like >> ZTimestampFormat). >> >> If you load or depend on tag/release v25 you get the latest version before >> this major change. >> >> If you encounter any problems please let me know. >> >> Sven
[Pharo-dev] ZTimestamp now has nanosecond precision
Hi, Last year, I extended ZTimestamp with a default nanosecond precision (it previously had whole second precision). This makes it a more complete replacement for DateAndTime. https://github.com/svenvc/ztimestamp https://github.com/svenvc/ztimestamp/commit/de6c5a670ed0c6cb18894b4917383dd9d1dbe503 The goals of ZTimestamp are to be simpler and more efficient (both faster and more compact). A ZTimestamp instance occupies half the space of a DateAndTime instance, its two slots being SmallIntegers. Back in the 32-bit days, the seconds since midnight field was too small for nanoseconds. Now, 64-bit is the default and there is enough room and no speed penalty for working with nanoseconds since midnight. In casual use, you won't see much of a difference, apart from the added detail: ZTimestamp now. "2022-01-27T09:21:20.528695Z" There are ways to convert to the older precision: ZTimestamp now truncated. ZTimestamp now rounded. ZTimestamp nowTruncated. Obviously, other methods and classes now support this added precision (like ZTimestampFormat). If you load or depend on tag/release v25 you get the latest version before this major change. If you encounter any problems please let me know. Sven
[Pharo-dev] Re: Array sum. is very slow
Yes that would certainly be useful. But, AFAIU, FloatArray consists of 32-bit Float numbers, I think we also need a DoubleFloatArray since 64-bit Floats are the default nowadays. > On 12 Jan 2022, at 16:31, Henrik Sperre Johansen > wrote: > > We could also try modifying Pharo to use C by reintroducing the FloatArray > plugin ;) > > | fa r | > fa := FloatArray new: 28800. > r := Random new. > 1 to fa size do: [ :i | fa at: i put: r next ]. > [ 1 to: fa size do: [ :i | fa sum ] ] timeToRun > > Pharo 9, no plugin: > 0:00:01:14.777 > Pharo 5, with plugin: > 0:00:00:00.526 > > Cheers, > Henry > > >> On 11 Jan 2022, at 10:08, Andrei Chis wrote: >> >> Hi Jimmie, >> >> I was scanning through this thread and saw that the Python call uses >> the sum function. If I remember correctly, in Python the built-in sum >> function is directly implemented in C [1] (unless Python is compiled >> with SLOW_SUM set to true). In that case on large arrays the function >> can easily be several times faster than just iterating over the >> individual objects as the Pharo code does. The benchmark seems to >> compare summing numbers in C with summing numbers in Pharo. Would be >> interesting to modify the Python code to use a similar loop as in >> Pharo for doing the sum. >> >> Cheers, >> Andrei >> >> [1] >> https://github.com/python/cpython/blob/135cabd328504e1648d17242b42b675cdbd0193b/Python/bltinmodule.c#L2461 >> >>> On Mon, Jan 10, 2022 at 9:06 PM Jimmie Houchin wrote: >>> >>> Some experiments and discoveries. >>> >>> I am running my full language test every time. It is the only way I can >>> compare results. It is also what fully stresses the language. >>> >>> The reason I wrote the test as I did is because I wanted to know a couple >>> of things. Is the language sufficiently performant on basic maths. I am not >>> doing any high PolyMath level math. Simple things like moving averages over >>> portions of arrays. >>> >>> The other is efficiency of array iteration and access. This why #sum is the >>> best test of this attribute. #sum iterates and accesses every element of >>> the array. It will reveal if there are any problems. >>> >>> The default test Julia 1m15s, Python 24.5 minutes, Pharo 2hour 4minutes. >>> >>> When I comment out the #sum and #average calls, Pharo completes the test in >>> 3.5 seconds. So almost all the time is spent in those two calls. >>> >>> So most of this conversation has focused on why #sum is as slow as it is or >>> how to improve the performance of #sum with other implementations. >>> >>> >>> >>> So I decided to breakdown the #sum and try some things. >>> >>> Starting with the initial implementation and SequenceableCollection's >>> default #sum time of 02:04:03 >>> >>> >>> "This implementation does no work. Only iterates through the array. >>> It completed in 00:10:08" >>> sum >>> | sum | >>>sum := 1. >>> 1 to: self size do: [ :each | ]. >>> ^ sum >>> >>> >>> "This implementation does no work, but adds to iteration, accessing the >>> value of the array. >>> It completed in 00:32:32. >>> Quite a bit of time for simply iterating and accessing." >>> sum >>> | sum | >>> sum := 1. >>> 1 to: self size do: [ :each | self at: each ]. >>> ^ sum >>> >>> >>> "This implementation I had in my initial email as an experiment and also >>> several other did the same in theirs. >>> A naive simple implementation. >>> It completed in 01:00:53. Half the time of the original." >>> sum >>> | sum | >>> sum := 0. >>> 1 to: self size do: [ :each | >>> sum := sum + (self at: each) ]. >>> ^ sum >>> >>> >>> >>> "This implementation I also had in my initial email as an experiment I had >>> done. >>> It completed in 00:50:18. >>> It reduces the iterations and increases the accesses per iteration. >>> It is the fastest implementation so far." >>> sum >>> | sum | >>> sum := 0. >>> 1 to: ((self size quo: 10) * 10) by: 10 do: [ :i | >>> sum := sum + (self at: i) + (self at: (i + 1)) + (self at: (i + 2)) + >>> (self at: (i + 3)) + (self at: (i + 4)) + (self at: (i + 5)) + >>> (self at: (i + 6)) + (self at: (i + 7)) + (self at: (i + 8)) + (self at: (i >>> + 9))]. >>> >>> ((self size quo: 10) * 10 + 1) to: self size do: [ :i | >>> sum := sum + (self at: i)]. >>> ^ sum >>> >>> Summary >>> >>> For whatever reason iterating and accessing on an Array is expensive. That >>> alone took longer than Python to complete the entire test. >>> >>> I had allowed this knowledge of how much slower Pharo was to stop me from >>> using Pharo. Encouraged me to explore other options. >>> >>> I have the option to use any language I want. I like Pharo. I do not like >>> Python at all. Julia is unexciting to me. I don't like their anti-OO >>> approach. >>> >>> At one point I had a fairly complete Pharo implementation, which is where I >>> got frustrated with backtesting taking days. >>> >>> That implementation is gone. I had
[Pharo-dev] Re: Array sum. is very slow
> On 11 Jan 2022, at 11:17, Sven Van Caekenberghe wrote: > > which would seem to be 3 times faster ! And with my changes (faster #sum, message spy removed): [ (LanguageTest newSize: 60*24*5*4 iterations: 1) run ] timeToRun. "0:00:00:26.612" 6 times faster.
[Pharo-dev] Re: Array sum. is very slow
Hi Andrei, That is a good catch, indeed, that makes all the difference and is an unfair comparison. If I take Jimmie's code and add def sum2(l): sum = 0 for i in range(0,len(l)): sum = sum + l[i] return sum def average(l): return sum2(l)/len(l) and replace the other calls of sum to sum2 in loop1 and loop2, I get the following for 1 iteration: >>> doit(1) Tue Jan 11 10:34:24 2022 Creating list createList(n), na[-1]: 0.28803 reps: 1 inside at top loop1: start: Tue Jan 11 10:34:24 2022 Loop1 time: 1.5645889163017273 nsum: 11242.949400371168 navg: 0.3903801875128878 loop2: start: Tue Jan 11 10:35:58 2022 Loop2 time: -27364895.977849767 nsum: 10816.16871440453 navg: 0.3755614136946017 finished: Tue Jan 11 10:37:33 2022 start time: 1641893664.795651 end time: 1641893853.597397 total time: 1614528959.1841362 nsum: 10816.16871440453 navg: 0.3755614136946017 The total time is calculated wrongly, but doing the calculation in Pharo: (1641893853.597397 - 1641893664.795651) seconds. "0:00:03:08.80174613" so 3 minutes. Jimmie's unmodified Pharo code give for 1 iteration: [ (LanguageTest newSize: 60*24*5*4 iterations: 1) run ] timeToRun. "0:00:01:00.438" Starting test for array size: 28800 iterations: 1 Creating array of size: 28800 timeToRun: 0:00:00:00.035 Starting loop 1 at: 2022-01-11T10:53:53.423313+01:00 1: 2022-01-11T10:53:53 innerttr: 0:00:00:30.073 averageTime: 0:00:00:30.073 Loop 1 time: nil nsum: 11242.949400371168 navg: 0.3903801875128878 Starting loop 2 at: 2022-01-11T10:54:23.497281+01:00 1: 2022-01-11T10:54:23 innerttr: 0:00:00:30.306 averageTime: 0:00:00:30.306 Loop 2 time: 0:00:00:30.306 nsum: 10816.168714404532 navg: 0.3755614136946018 End of test. TotalTime: 0:00:01:00.416 which would seem to be 3 times faster ! Benchmarking is a black art. Sven > On 11 Jan 2022, at 10:07, Andrei Chis wrote: > > Hi Jimmie, > > I was scanning through this thread and saw that the Python call uses > the sum function. If I remember correctly, in Python the built-in sum > function is directly implemented in C [1] (unless Python is compiled > with SLOW_SUM set to true). In that case on large arrays the function > can easily be several times faster than just iterating over the > individual objects as the Pharo code does. The benchmark seems to > compare summing numbers in C with summing numbers in Pharo. Would be > interesting to modify the Python code to use a similar loop as in > Pharo for doing the sum. > > Cheers, > Andrei > > [1] > https://github.com/python/cpython/blob/135cabd328504e1648d17242b42b675cdbd0193b/Python/bltinmodule.c#L2461 > > On Mon, Jan 10, 2022 at 9:06 PM Jimmie Houchin wrote: >> >> Some experiments and discoveries. >> >> I am running my full language test every time. It is the only way I can >> compare results. It is also what fully stresses the language. >> >> The reason I wrote the test as I did is because I wanted to know a couple of >> things. Is the language sufficiently performant on basic maths. I am not >> doing any high PolyMath level math. Simple things like moving averages over >> portions of arrays. >> >> The other is efficiency of array iteration and access. This why #sum is the >> best test of this attribute. #sum iterates and accesses every element of the >> array. It will reveal if there are any problems. >> >> The default test Julia 1m15s, Python 24.5 minutes, Pharo 2hour 4minutes. >> >> When I comment out the #sum and #average calls, Pharo completes the test in >> 3.5 seconds. So almost all the time is spent in those two calls. >> >> So most of this conversation has focused on why #sum is as slow as it is or >> how to improve the performance of #sum with other implementations. >> >> >> >> So I decided to breakdown the #sum and try some things. >> >> Starting with the initial implementation and SequenceableCollection's >> default #sum time of 02:04:03 >> >> >> "This implementation does no work. Only iterates through the array. >> It completed in 00:10:08" >> sum >>| sum | >> sum := 1. >>1 to: self size do: [ :each | ]. >>^ sum >> >> >> "This implementation does no work, but adds to iteration, accessing the >> value of the array. >> It completed in 00:32:32. >> Quite a bit of time for simply iterating and accessing." >> sum >>| sum | >>sum := 1. >>1 to: self size do: [ :each | self at: each ]. >>^ sum >> >> >> "This implementation I had in my initial email as an experiment and also >> several other did the same in theirs. >> A naive simple implementation. >> It completed in 01:00:53. Half the time of the original." >> sum >> | sum | >>sum := 0. >>1 to: self size do: [ :each | >>sum := sum + (self at: each) ]. >>^ sum >> >> >> >> "This implementation I also had in my initial email as an experiment I had >> done. >> It completed in 00:50:18. >> It reduces the iterations and increases the accesses per iteration. >> It is the
[Pharo-dev] Re: Array sum. is very slow
Hi Jimmy, I made a couple more changes: - I added SequenceableCollection>>#sum | sum | sum := 0. 1 to: self size do: [ :each | sum := sum + (self at: each) ]. ^ sum as an extension method. It is not 100% semantically the same as the original, but it works for our case here. this also optimises #average BTW. This is the main one. - I tried to avoid a couple of integer -> float conversions in normalize: n | nn | nn := n = 0 ifTrue: [ 0.000123456789 ] ifFalse: [ n asFloat ]. [ nn <= 0.0001 ] whileTrue: [ nn := nn * 10.0 ]. [ nn >= 1.0 ] whileTrue: [ nn := nn * 0.1 ]. ^ nn - Avoided one assignment in loop1calc: i j: j n: n | v | v := n * (i+n) * (j-n) * 0.1234567. ^ self normalize: (v*v*v) the time for 10 iterations now is halved: === Starting test for array size: 28800 iterations: 10 Creating array of size: 28800 timeToRun: 0:00:00:00.002 Starting loop 1 at: 2022-01-07T19:28:52.109011+01:00 Loop 1 time: nil nsum: 11234.235001659386 navg: 0.3900776042242842 Starting loop 2 at: 2022-01-07T19:31:21.821784+01:00 Loop 2 time: 0:00:02:28.017 nsum: 11245.697629561537 navg: 0.3904756121375534 End of test. TotalTime: 0:00:04:57.733 === Sven > On 7 Jan 2022, at 16:30, Sven Van Caekenberghe wrote: > > > >> On 7 Jan 2022, at 16:05, Jimmie Houchin wrote: >> >> Hello Sven, >> >> I went and removed the Stdouts that you mention and other timing code from >> the loops. >> >> I am running the test now, to see if that makes much difference. I do not >> think it will. >> >> The reason I put that in there is because it take so long to run. It can be >> frustrating to wait and wait and not know if your test is doing anything or >> not. So I put the code in to let me know. >> >> One of your parameters is incorrect. It is 100 iterations not 10. > > Ah, I misread the Python code, on top it says, reps = 10, while at the bottom > it does indeed say, doit(100). > > So the time should be multiplied by 10. > > The logging, esp. the #flush will slow things down. But the removing the > message tally spy is important too. > > The general implementation of #sum is not optimal in the case of a fixed > array. Consider: > > data := Array new: 1e5 withAll: 0.5. > > [ data sum ] bench. "'494.503 per second'" > > [ | sum | sum := 0. data do: [ :each | sum := sum + each ]. sum ] bench. > "'680.128 per second'" > > [ | sum | sum := 0. 1 to: 1e5 do: [ :each | sum := sum + (data at: each) ]. > sum ] bench. "'1033.180 per second'" > > As others have remarked: doing #average right after #sum is doing the same > thing twice. But maybe that is not the point. > >> I learned early on in this experiment that I have to do a large number of >> iterations or C, C++, Java, etc are too fast to have comprehensible results. >> >> I can tell if any of the implementations is incorrect by the final nsum. All >> implementations must produce the same result. >> >> Thanks for the comments. >> >> Jimmie >> >> >> On 1/7/22 07:40, Sven Van Caekenberghe wrote: >>> Hi Jimmie, >>> >>> I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 >>> >>> I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) >>> (not done in Python either) as well as the MessageTally spyOn: from #run >>> (slows things down). >>> >>> Then I ran your code with: >>> >>> [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. >>> >>> which gave me "0:00:09:31.338" >>> >>> The console output was: >>> >>> === >>> Starting test for array size: 28800 iterations: 10 >>> >>> Creating array of size: 28800 timeToRun: 0:00:00:00.031 >>> >>> Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 >>> Loop 1 time: nil >>> nsum: 11234.235001659388 >>> navg: 0.39007760422428434 >>> >>> Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 >>> Loop 2 time: 0:00:04:44.593 >>> nsum: 11245.697629561537 >>> navg: 0.3904756121375534 >>> >>> End of test. TotalTime: 0:00:09:31.338 >>> === >>> >>> Which would be twice as fast as Python, if I got the parameters correct. >>> >>> Sven >>> >>>> On 7 Jan 2022, at 13:19, Jimmie Houchin wrote: >>>> >>>> As I stated this is a micro benchma
[Pharo-dev] Re: Array sum. is very slow
> On 7 Jan 2022, at 16:05, Jimmie Houchin wrote: > > Hello Sven, > > I went and removed the Stdouts that you mention and other timing code from > the loops. > > I am running the test now, to see if that makes much difference. I do not > think it will. > > The reason I put that in there is because it take so long to run. It can be > frustrating to wait and wait and not know if your test is doing anything or > not. So I put the code in to let me know. > > One of your parameters is incorrect. It is 100 iterations not 10. Ah, I misread the Python code, on top it says, reps = 10, while at the bottom it does indeed say, doit(100). So the time should be multiplied by 10. The logging, esp. the #flush will slow things down. But the removing the message tally spy is important too. The general implementation of #sum is not optimal in the case of a fixed array. Consider: data := Array new: 1e5 withAll: 0.5. [ data sum ] bench. "'494.503 per second'" [ | sum | sum := 0. data do: [ :each | sum := sum + each ]. sum ] bench. "'680.128 per second'" [ | sum | sum := 0. 1 to: 1e5 do: [ :each | sum := sum + (data at: each) ]. sum ] bench. "'1033.180 per second'" As others have remarked: doing #average right after #sum is doing the same thing twice. But maybe that is not the point. > I learned early on in this experiment that I have to do a large number of > iterations or C, C++, Java, etc are too fast to have comprehensible results. > > I can tell if any of the implementations is incorrect by the final nsum. All > implementations must produce the same result. > > Thanks for the comments. > > Jimmie > > > On 1/7/22 07:40, Sven Van Caekenberghe wrote: >> Hi Jimmie, >> >> I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 >> >> I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) >> (not done in Python either) as well as the MessageTally spyOn: from #run >> (slows things down). >> >> Then I ran your code with: >> >> [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. >> >> which gave me "0:00:09:31.338" >> >> The console output was: >> >> === >> Starting test for array size: 28800 iterations: 10 >> >> Creating array of size: 28800 timeToRun: 0:00:00:00.031 >> >> Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 >> Loop 1 time: nil >> nsum: 11234.235001659388 >> navg: 0.39007760422428434 >> >> Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 >> Loop 2 time: 0:00:04:44.593 >> nsum: 11245.697629561537 >> navg: 0.3904756121375534 >> >> End of test. TotalTime: 0:00:09:31.338 >> === >> >> Which would be twice as fast as Python, if I got the parameters correct. >> >> Sven >> >>> On 7 Jan 2022, at 13:19, Jimmie Houchin wrote: >>> >>> As I stated this is a micro benchmark and very much not anything resembling >>> a real app, Your comments are true if you are writing your app. But if you >>> want to stress the language you are going to do things which are seemingly >>> non-sense and abusive. >>> >>> Also as I stated. The test has to be sufficient to stress faster languages >>> or it is meaningless. >>> >>> If I remove the #sum and the #average calls from the inner loops, this is >>> what we get. >>> >>> Julia 0.2256 seconds >>> Python 5.318 seconds >>> Pharo3.5seconds >>> >>> This test does not sufficiently stress the language. Nor does it provide >>> any valuable insight into summing and averaging which is done a lot, in >>> lots of places in every iteration. >>> >>> If you notice that inner array changes the array every iteration. So every >>> call to #sum and #average is getting different data. >>> >>> Full Test >>> >>> Julia 1.13 minutes >>> Python 24.02 minutes >>> Pharo2:09:04 >>> >>> Code for the above is now published. You can let me know if I am doing >>> something unequal to the various languages. >>> >>> And just remember anything you do which sufficiently changes the test has >>> to be done in all the languages to give a fair test. This isn't a lets make >>> Pharo look good test. I do want Pharo to look good, but honestly. >>> >>> Yes, I know that I can bind to BLAS or other external libraries. But that >>> is not a test of Pharo. The Python is plain Python3 no
[Pharo-dev] Re: Array sum. is very slow
Hi Jimmie, I loaded your code in Pharo 9 on my MacBook Pro (Intel i5) macOS 12.1 I commented out the Stdio logging from the 2 inner loops (#loop1, #loop2) (not done in Python either) as well as the MessageTally spyOn: from #run (slows things down). Then I ran your code with: [ (LanguageTest newSize: 60*24*5*4 iterations: 10) run ] timeToRun. which gave me "0:00:09:31.338" The console output was: === Starting test for array size: 28800 iterations: 10 Creating array of size: 28800 timeToRun: 0:00:00:00.031 Starting loop 1 at: 2022-01-07T14:10:35.395394+01:00 Loop 1 time: nil nsum: 11234.235001659388 navg: 0.39007760422428434 Starting loop 2 at: 2022-01-07T14:15:22.108433+01:00 Loop 2 time: 0:00:04:44.593 nsum: 11245.697629561537 navg: 0.3904756121375534 End of test. TotalTime: 0:00:09:31.338 === Which would be twice as fast as Python, if I got the parameters correct. Sven > On 7 Jan 2022, at 13:19, Jimmie Houchin wrote: > > As I stated this is a micro benchmark and very much not anything resembling a > real app, Your comments are true if you are writing your app. But if you want > to stress the language you are going to do things which are seemingly > non-sense and abusive. > > Also as I stated. The test has to be sufficient to stress faster languages or > it is meaningless. > > If I remove the #sum and the #average calls from the inner loops, this is > what we get. > > Julia 0.2256 seconds > Python 5.318 seconds > Pharo3.5seconds > > This test does not sufficiently stress the language. Nor does it provide any > valuable insight into summing and averaging which is done a lot, in lots of > places in every iteration. > > If you notice that inner array changes the array every iteration. So every > call to #sum and #average is getting different data. > > Full Test > > Julia 1.13 minutes > Python 24.02 minutes > Pharo2:09:04 > > Code for the above is now published. You can let me know if I am doing > something unequal to the various languages. > > And just remember anything you do which sufficiently changes the test has to > be done in all the languages to give a fair test. This isn't a lets make > Pharo look good test. I do want Pharo to look good, but honestly. > > Yes, I know that I can bind to BLAS or other external libraries. But that is > not a test of Pharo. The Python is plain Python3 no Numpy, just using the the > default list [] for the array. > > Julia is a whole other world. It is faster than Numpy. This is their domain > and they optimize, optimize, optimize all the math. In fact they have reached > the point that some pure Julia code beats pure Fortran. > > In all of this I just want Pharo to do the best it can. > > With the above results unless you already had an investment in Pharo, you > wouldn't even look. :( > > Thanks for exploring this with me. > > > Jimmie > > > > > On 1/6/22 18:24, John Brant wrote: >> On Jan 6, 2022, at 4:35 PM, Jimmie Houchin wrote: >>> No, it is an array of floats. The only integers in the test are in the >>> indexes of the loops. >>> >>> Number random. "generates a float 0.8188008774329387" >>> >>> So in the randarray below it is an array of 28800 floats. >>> >>> It just felt so wrong to me that Python3 was so much faster. I don't care >>> if Nim, Crystal, Julia are faster. But... >>> >>> >>> I am new to Iceberg and have never shared anything on Github so this is all >>> new to me. I uploaded my language test so you can see what it does. It is a >>> micro-benchmark. It does things that are not realistic in an app. But it >>> does stress a language in areas important to my app. >>> >>> >>> https://github.com/jlhouchin/LanguageTestPharo >>> >>> >>> Let me know if there is anything else I can do to help solve this problem. >>> >>> I am a lone developer in my spare time. So my apologies for any ugly code. >>> >> Are you sure that you have the same algorithm in Python? You are calling sum >> and average inside the loop where you are modifying the array: >> >> 1 to: nsize do: [ :j || n | >> n := narray at: j. >> narray at: j put: (self loop1calc: i j: j n: n). >> nsum := narray sum. >> navg := narray average ] >> >> As a result, you are calculating the sum of the 28,800 size array 28,800 >> times (plus another 28,800 times for the average). If I write a similar loop >> in Python, it looks like it would take almost 9 minutes on my machine >> without using numpy to calculate the sum. The Pharo code takes ~40 seconds. >> If this is really how the code should be, then I would change it to not call >> sum twice (once for sum and once in average). This will almost result in a >> 2x speedup. You could also modify the algorithm to update the nsum value in >> the loop instead of summing the array each time. I think the updating would >> require <120,000 math ops vs the >1.6 billion that you are performing. >> >>
[Pharo-dev] Re: Zinc redirect?
There is now a solution described in the issue: https://github.com/svenvc/zinc/issues/69 > On 10 Dec 2021, at 10:58, Sven Van Caekenberghe wrote: > > There is an issue that Ronie created, I am looking into it: > > https://github.com/svenvc/zinc/issues/69 > >> On 10 Dec 2021, at 08:25, stephane ducasse wrote: >> >> FYI >> >> >> running catching my train. >> from >> https://github.com/ronsaldo/form-scene-graph/issues/5#issuecomment-990353154 >> >> >> >> >> >> This seems to be a Zinc error while following a redirect. The following >> snippet can be used for replicating the issue: >> >> ZnClient new accept: 'application/octet-stream'; "Required by GitHub" url: >> 'https://api.github.com/repos/ronsaldo/abstract-gpu/releases/assets/49144132'; >> followRedirects: true; downloadTo: 'agpu.tar.gz' >> >
[Pharo-dev] Re: Zinc redirect?
There is an issue that Ronie created, I am looking into it: https://github.com/svenvc/zinc/issues/69 > On 10 Dec 2021, at 08:25, stephane ducasse wrote: > > FYI > > > running catching my train. > from > https://github.com/ronsaldo/form-scene-graph/issues/5#issuecomment-990353154 > > > > > > This seems to be a Zinc error while following a redirect. The following > snippet can be used for replicating the issue: > > ZnClient new accept: 'application/octet-stream'; "Required by GitHub" url: > 'https://api.github.com/repos/ronsaldo/abstract-gpu/releases/assets/49144132'; > followRedirects: true; downloadTo: 'agpu.tar.gz' >
[Pharo-dev] Re: Hardening Zinc's Core HTTP Server
Thx. > On 31 Oct 2021, at 10:09, Tim Mackinnon wrote: > > Hey Sven - really appreciate your thorough approach to this, and it’s been > interesting seeing the results (I did try a few script injection attempts but > didn’t get up to running burp suite on it, which I may try it while you have > the instances). > > Kudos to Levente for taking a more reasoned approach and using inside > knowledge… this community is awesome! > > Tim > >> On 29 Oct 2021, at 19:42, Sven Van Caekenberghe wrote: >> >> Here is yet another update: >> >> The instances >> >> - On Amazon AWS: http://34.245.183.130:1701 >> >> - On Microsoft Azure: http://51.137.72.94:8080 >> >> have now been running for 18+ days. >> >> Having observed the overall amounts of critical resources inside the Pharo >> images, things look good. >> >> Seaside WASSession cleanup goes slowly but does work over time. >> >> Eventually I will stop at least the Azure instance because it costs me >> personal money. >> >>> On 11 Oct 2021, at 19:37, Sven Van Caekenberghe wrote: >>> >>> Here is an update on the status of both instances. >>> >>> A couple of people tried fiddling with invalid input which did result in >>> exceptions, but everything was handled correctly so that the server kept on >>> functioning. >>> >>> There were no spurious crashes. >>> >>> There was one attack by Levente Uzonyi that was successful though. >>> >>> He used the fact that Seaside sessions are long lived (cached for a certain >>> time) combined with the fact that the Reddit.st app kept one GLORP database >>> connection through P3 open to PostgreSQL and fired off a small denial of >>> service (DOS) attack. Eventually either the VM ran out of space in the >>> ExternalSemaphoreTable, making Socket creation, either for database >>> connections or for handling new HTTP request impossible or the database's >>> own connection limit was reached. At that point the HTTP server or the >>> Pharo VM stopped functioning. >>> >>> Although Levente used a sequential number of requests, a concurrent number >>> of request could cause similar problems (related, but differently). >>> >>> In looking at what he reported it also became clear that I accidentally >>> left a debugging exception handler active, which is not good in a >>> production image. >>> >>> Both instances are now redeployed with the following changes: >>> >>> - the Reddit.st code was changed to not keep the database connection open >>> all the time, but instead connect/disconnect per request. this might be a >>> bit slower, but it conserves resources much better and solves the original >>> issue Levente reported >>> >>> https://github.com/svenvc/Reddit/commit/f2e0a0dc00b9cbb68cfa4fb007906365ae66ab1b >>> >>> - a new feature was added to Zinc HTTP Components' >>> ZnManagingMultiThreadedServer (the default) to enforce a maximum number of >>> concurrent connections being allowed (the limit is 32 by default, but >>> changeable if you know what you are doing). when the limit is reached, 503 >>> Service Unavailable responses are sent to the excess clients and the >>> connection is closed. this should help protect against concurrent >>> connection DOS attacks >>> >>> https://github.com/svenvc/zinc/commit/ac0f06e74e7ab129610c466cb1d7ea9533d29b4c >>> >>> - the deploy script was changed to use the more primitive WAErrorHandler >>> >>> https://github.com/svenvc/Reddit/commit/874b631e6dc0c04c8c0b687ef770d00540d282df >>> >>> Thanks again to Levente for taking the time to try an attack and for >>> reporting it clearly. >>> >>> Sven >>> >>>>> On 29 Sep 2021, at 17:10, Sven Van Caekenberghe wrote: >>>> >>>> Both instances have been up for 5 days now, looking for more testers. >>>> >>>>> On 23 Sep 2021, at 17:03, Sven Van Caekenberghe wrote: >>>>> >>>>> Hi, >>>>> >>>>> Zinc HTTP Components [https://github.com/svenvc/zinc] has been a part of >>>>> Pharo since version 1.3 (2011). It is an open-source framework to deal >>>>> with the HTTP networking protocol, modelling all aspects involved. It >>>>> also offers both client and server functionality. >>>>> >&
[Pharo-dev] Re: Hardening Zinc's Core HTTP Server
Here is yet another update: The instances - On Amazon AWS: http://34.245.183.130:1701 - On Microsoft Azure: http://51.137.72.94:8080 have now been running for 18+ days. Having observed the overall amounts of critical resources inside the Pharo images, things look good. Seaside WASSession cleanup goes slowly but does work over time. Eventually I will stop at least the Azure instance because it costs me personal money. > On 11 Oct 2021, at 19:37, Sven Van Caekenberghe wrote: > > Here is an update on the status of both instances. > > A couple of people tried fiddling with invalid input which did result in > exceptions, but everything was handled correctly so that the server kept on > functioning. > > There were no spurious crashes. > > There was one attack by Levente Uzonyi that was successful though. > > He used the fact that Seaside sessions are long lived (cached for a certain > time) combined with the fact that the Reddit.st app kept one GLORP database > connection through P3 open to PostgreSQL and fired off a small denial of > service (DOS) attack. Eventually either the VM ran out of space in the > ExternalSemaphoreTable, making Socket creation, either for database > connections or for handling new HTTP request impossible or the database's own > connection limit was reached. At that point the HTTP server or the Pharo VM > stopped functioning. > > Although Levente used a sequential number of requests, a concurrent number of > request could cause similar problems (related, but differently). > > In looking at what he reported it also became clear that I accidentally left > a debugging exception handler active, which is not good in a production image. > > Both instances are now redeployed with the following changes: > > - the Reddit.st code was changed to not keep the database connection open all > the time, but instead connect/disconnect per request. this might be a bit > slower, but it conserves resources much better and solves the original issue > Levente reported > > https://github.com/svenvc/Reddit/commit/f2e0a0dc00b9cbb68cfa4fb007906365ae66ab1b > > - a new feature was added to Zinc HTTP Components' > ZnManagingMultiThreadedServer (the default) to enforce a maximum number of > concurrent connections being allowed (the limit is 32 by default, but > changeable if you know what you are doing). when the limit is reached, 503 > Service Unavailable responses are sent to the excess clients and the > connection is closed. this should help protect against concurrent connection > DOS attacks > > https://github.com/svenvc/zinc/commit/ac0f06e74e7ab129610c466cb1d7ea9533d29b4c > > - the deploy script was changed to use the more primitive WAErrorHandler > > https://github.com/svenvc/Reddit/commit/874b631e6dc0c04c8c0b687ef770d00540d282df > > Thanks again to Levente for taking the time to try an attack and for > reporting it clearly. > > Sven > >> On 29 Sep 2021, at 17:10, Sven Van Caekenberghe wrote: >> >> Both instances have been up for 5 days now, looking for more testers. >> >>> On 23 Sep 2021, at 17:03, Sven Van Caekenberghe wrote: >>> >>> Hi, >>> >>> Zinc HTTP Components [https://github.com/svenvc/zinc] has been a part of >>> Pharo since version 1.3 (2011). It is an open-source framework to deal with >>> the HTTP networking protocol, modelling all aspects involved. It also >>> offers both client and server functionality. >>> >>> The reliability of the code base has improved steadily over the years, >>> thanks to virtually all Pharo developers using it, directly or indirectly. >>> Over the summer a number of issues that popped up after Pharo 9 was >>> released were resolved. >>> >>> The robustness of the core HTTP server is one important aspect. To put this >>> quality further to the test, I deployed two servers with the same demo >>> Seaside application, Reddit.st, open to the internet, without any further >>> protections. >>> >>> - On Amazon AWS: http://34.245.183.130:1701 >>> >>> - On Microsoft Azure: http://51.137.72.94:8080 >>> >>> The application's source code can be found at >>> [https://github.com/svenvc/Reddit]. For the technically curious there are >>> also deploy instructions at >>> [https://github.com/svenvc/Reddit/blob/main/DEPLOY.md]. The demo app itself >>> is described in an older article >>> [https://medium.com/@svenvc/reddit-st-in-10-cool-pharo-classes-1b5327ca0740]. >>> Note that, by definition, there is no HTTPS/TLS variant. >>> >>> If you manage to break this server with (a) malicious request(s) in such a >>> way that you can explain what you did for others to confirm your approach, >>> you not only help me/us improve the code, but earn eternal fame as well ;-) >>> >>> Sven >>> >>> PS: I hope I won't regret this, I am looking for constructive criticism. >>> >>> >>> -- >>> Sven Van Caekenberghe >>> Proudly supporting Pharo >>> http://pharo.org >>> http://association.pharo.org >>> http://consortium.pharo.org >>> >> >
[Pharo-dev] Re: Something weird in VirtualMachine>>#maxExternalSemaphores:
I copied your comment to the issue thread. Maybe it is best to hold the conversation there as to leave a better trace. > On 13 Oct 2021, at 13:29, Henrik Sperre Johansen > wrote: > > Seens it used to be in the «old» PharoVM before the reconcilliation: > https://forum.world.st/external-semaphores-again-tp4712934p4713396.html > > Maybe it’s still possible to find it in the git history of the current > pharovm repository? > > Cheers, > Henry > > On Wednesday, October 13, 2021, Sven Van Caekenberghe wrote: > Hi Henry, > > Thanks for your reply. > > I created an issue: https://github.com/pharo-project/pharo/issues/10156 > > Can we still find the file 'sqExternalSemaphores.c' from the old emails ? > > Sven > > > On 12 Oct 2021, at 19:20, Henrik Sperre Johansen > > wrote: > > > > I believe Igor Stasenko made changes so signals could no longer get lost, > > but I’m not sure it ever got integrated in Cog… > > > > https://forum.world.st/Issue-5965-in-pharo-Can-t-allocate-semaphores-VirtualMachine-maxExternalSemaphores-aSize-Not-enough--tp4631607p4632388.html > > > > Maybe worth revisiting/reviewing? > > If it works, the limitation would be gone, and the image warnings could be > > removed with no ill side-effects. > > > > Cheers, > > Henry > > > >> On 12 Oct 2021, at 18:55, Henrik Sperre Johansen > >> wrote: > >> > >> Hi Sven - you’re correct the main intent was to make it clear one should > >> not really be setting this from the image during runtime - and make it as > >> obvious as possible when it had. > >> > >> The way I remember it, the Error was only raised if the image is in > >> development mode - so a developer running into this, would have the limit > >> raised, but also get a debugger (which is hard to ignore) explaining the > >> issue, while in runtime mode, the SystemNotification would get handled by > >> logging so it would at least be possible to spot that it had happened. > >> > >> If there’s a less confusing way to achieve that it would be a good thing, > >> but I’m not sure removing all warnings would be the best way, as the > >> related i/o problems would be nearly impossible to determine the root > >> cause of. > >> > >> Why forked? No idea/can’t remember… > >> > >> Cheers, > >> Henry > >> > >>> On 12 Oct 2021, at 16:01, Sven Van Caekenberghe wrote: > >>> > >>> Hi, > >>> > >>> The ExternalSemaphoreTable is an important resource since it holds 3 > >>> Semaphores per Socket (network connection). The size of > >>> ExternalSemaphoreTable determines how many such resources can be used > >>> concurrently. > >>> > >>> VM parameter 49 holds this limit, it is accessed by > >>> VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to > >>> be 256 currently. > >>> > >>> There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that > >>> can be used to set this limit higher. This seems to work. > >>> > >>> Now, there are some limitations here: the number must be a power of 2, > >>> higher that the one set before and lower than 64K. BTW the number of file > >>> descriptors for files and sockets is also limited by the OS for its > >>> processes. > >>> > >>> Also, while growing the table, the Semaphores become unavailable to the > >>> VM for a very short period of time, so they could miss signals, leading > >>> to IO issues. > >>> > >>> For this reason, it seems that the design goal was to not allow this to > >>> grow during normal use, only during startup or upfront configuration. > >>> > >>> Now comes the weird thing: the implementation/code of > >>> VirtualMachine>>#maxExternalSemaphores: > >>> > >>> This is the method called when the current table is too small and when it > >>> wants to grow larger than the current > >>> VirtualMachine>>#maxExternalSemaphores. > >>> > >>> The implementation calls #maxExternalSemaphoresSilently: in a forked > >>> thread, after signalling a SystemNotification with a full explanation. > >>> But why should this be forked in the first place ? > >>> > >>> Then in the main thread, it signals an error ! Presumably because the > >>> design goals mentioned before. > >>> > >>> These two actions are in conflict. > >>> > >>> All this is probably the result of well meant evolution and refactoring, > >>> but it is quite confusing. > >>> > >>> My first idea would be: let the caller raise the error and don't try > >>> raising the limit, remove the current implementation and replace it by > >>> what is now #maxExternalSemaphoresSilently: > >>> > >>> What do you think ? > >>> > >>> Sven > >>> > >>> > >>> > >>>
[Pharo-dev] Re: Something weird in VirtualMachine>>#maxExternalSemaphores:
Hi Henry, Thanks for your reply. I created an issue: https://github.com/pharo-project/pharo/issues/10156 Can we still find the file 'sqExternalSemaphores.c' from the old emails ? Sven > On 12 Oct 2021, at 19:20, Henrik Sperre Johansen > wrote: > > I believe Igor Stasenko made changes so signals could no longer get lost, but > I’m not sure it ever got integrated in Cog… > > https://forum.world.st/Issue-5965-in-pharo-Can-t-allocate-semaphores-VirtualMachine-maxExternalSemaphores-aSize-Not-enough--tp4631607p4632388.html > > Maybe worth revisiting/reviewing? > If it works, the limitation would be gone, and the image warnings could be > removed with no ill side-effects. > > Cheers, > Henry > >> On 12 Oct 2021, at 18:55, Henrik Sperre Johansen >> wrote: >> >> Hi Sven - you’re correct the main intent was to make it clear one should >> not really be setting this from the image during runtime - and make it as >> obvious as possible when it had. >> >> The way I remember it, the Error was only raised if the image is in >> development mode - so a developer running into this, would have the limit >> raised, but also get a debugger (which is hard to ignore) explaining the >> issue, while in runtime mode, the SystemNotification would get handled by >> logging so it would at least be possible to spot that it had happened. >> >> If there’s a less confusing way to achieve that it would be a good thing, >> but I’m not sure removing all warnings would be the best way, as the related >> i/o problems would be nearly impossible to determine the root cause of. >> >> Why forked? No idea/can’t remember… >> >> Cheers, >> Henry >> >>> On 12 Oct 2021, at 16:01, Sven Van Caekenberghe wrote: >>> >>> Hi, >>> >>> The ExternalSemaphoreTable is an important resource since it holds 3 >>> Semaphores per Socket (network connection). The size of >>> ExternalSemaphoreTable determines how many such resources can be used >>> concurrently. >>> >>> VM parameter 49 holds this limit, it is accessed by >>> VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to be >>> 256 currently. >>> >>> There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that can >>> be used to set this limit higher. This seems to work. >>> >>> Now, there are some limitations here: the number must be a power of 2, >>> higher that the one set before and lower than 64K. BTW the number of file >>> descriptors for files and sockets is also limited by the OS for its >>> processes. >>> >>> Also, while growing the table, the Semaphores become unavailable to the VM >>> for a very short period of time, so they could miss signals, leading to IO >>> issues. >>> >>> For this reason, it seems that the design goal was to not allow this to >>> grow during normal use, only during startup or upfront configuration. >>> >>> Now comes the weird thing: the implementation/code of >>> VirtualMachine>>#maxExternalSemaphores: >>> >>> This is the method called when the current table is too small and when it >>> wants to grow larger than the current >>> VirtualMachine>>#maxExternalSemaphores. >>> >>> The implementation calls #maxExternalSemaphoresSilently: in a forked >>> thread, after signalling a SystemNotification with a full explanation. But >>> why should this be forked in the first place ? >>> >>> Then in the main thread, it signals an error ! Presumably because the >>> design goals mentioned before. >>> >>> These two actions are in conflict. >>> >>> All this is probably the result of well meant evolution and refactoring, >>> but it is quite confusing. >>> >>> My first idea would be: let the caller raise the error and don't try >>> raising the limit, remove the current implementation and replace it by what >>> is now #maxExternalSemaphoresSilently: >>> >>> What do you think ? >>> >>> Sven >>> >>> >>> >>>
[Pharo-dev] Something weird in VirtualMachine>>#maxExternalSemaphores:
Hi, The ExternalSemaphoreTable is an important resource since it holds 3 Semaphores per Socket (network connection). The size of ExternalSemaphoreTable determines how many such resources can be used concurrently. VM parameter 49 holds this limit, it is accessed by VirtualMachine>>#maxExternalSemaphores. The default for Pharo 9 seems to be 256 currently. There is a method VirtualMachine>>#maxExternalSemaphoresSilently: that can be used to set this limit higher. This seems to work. Now, there are some limitations here: the number must be a power of 2, higher that the one set before and lower than 64K. BTW the number of file descriptors for files and sockets is also limited by the OS for its processes. Also, while growing the table, the Semaphores become unavailable to the VM for a very short period of time, so they could miss signals, leading to IO issues. For this reason, it seems that the design goal was to not allow this to grow during normal use, only during startup or upfront configuration. Now comes the weird thing: the implementation/code of VirtualMachine>>#maxExternalSemaphores: This is the method called when the current table is too small and when it wants to grow larger than the current VirtualMachine>>#maxExternalSemaphores. The implementation calls #maxExternalSemaphoresSilently: in a forked thread, after signalling a SystemNotification with a full explanation. But why should this be forked in the first place ? Then in the main thread, it signals an error ! Presumably because the design goals mentioned before. These two actions are in conflict. All this is probably the result of well meant evolution and refactoring, but it is quite confusing. My first idea would be: let the caller raise the error and don't try raising the limit, remove the current implementation and replace it by what is now #maxExternalSemaphoresSilently: What do you think ? Sven
[Pharo-dev] Re: Hardening Zinc's Core HTTP Server
Here is an update on the status of both instances. A couple of people tried fiddling with invalid input which did result in exceptions, but everything was handled correctly so that the server kept on functioning. There were no spurious crashes. There was one attack by Levente Uzonyi that was successful though. He used the fact that Seaside sessions are long lived (cached for a certain time) combined with the fact that the Reddit.st app kept one GLORP database connection through P3 open to PostgreSQL and fired off a small denial of service (DOS) attack. Eventually either the VM ran out of space in the ExternalSemaphoreTable, making Socket creation, either for database connections or for handling new HTTP request impossible or the database's own connection limit was reached. At that point the HTTP server or the Pharo VM stopped functioning. Although Levente used a sequential number of requests, a concurrent number of request could cause similar problems (related, but differently). In looking at what he reported it also became clear that I accidentally left a debugging exception handler active, which is not good in a production image. Both instances are now redeployed with the following changes: - the Reddit.st code was changed to not keep the database connection open all the time, but instead connect/disconnect per request. this might be a bit slower, but it conserves resources much better and solves the original issue Levente reported https://github.com/svenvc/Reddit/commit/f2e0a0dc00b9cbb68cfa4fb007906365ae66ab1b - a new feature was added to Zinc HTTP Components' ZnManagingMultiThreadedServer (the default) to enforce a maximum number of concurrent connections being allowed (the limit is 32 by default, but changeable if you know what you are doing). when the limit is reached, 503 Service Unavailable responses are sent to the excess clients and the connection is closed. this should help protect against concurrent connection DOS attacks https://github.com/svenvc/zinc/commit/ac0f06e74e7ab129610c466cb1d7ea9533d29b4c - the deploy script was changed to use the more primitive WAErrorHandler https://github.com/svenvc/Reddit/commit/874b631e6dc0c04c8c0b687ef770d00540d282df Thanks again to Levente for taking the time to try an attack and for reporting it clearly. Sven > On 29 Sep 2021, at 17:10, Sven Van Caekenberghe wrote: > > Both instances have been up for 5 days now, looking for more testers. > >> On 23 Sep 2021, at 17:03, Sven Van Caekenberghe wrote: >> >> Hi, >> >> Zinc HTTP Components [https://github.com/svenvc/zinc] has been a part of >> Pharo since version 1.3 (2011). It is an open-source framework to deal with >> the HTTP networking protocol, modelling all aspects involved. It also offers >> both client and server functionality. >> >> The reliability of the code base has improved steadily over the years, >> thanks to virtually all Pharo developers using it, directly or indirectly. >> Over the summer a number of issues that popped up after Pharo 9 was released >> were resolved. >> >> The robustness of the core HTTP server is one important aspect. To put this >> quality further to the test, I deployed two servers with the same demo >> Seaside application, Reddit.st, open to the internet, without any further >> protections. >> >> - On Amazon AWS: http://34.245.183.130:1701 >> >> - On Microsoft Azure: http://51.137.72.94:8080 >> >> The application's source code can be found at >> [https://github.com/svenvc/Reddit]. For the technically curious there are >> also deploy instructions at >> [https://github.com/svenvc/Reddit/blob/main/DEPLOY.md]. The demo app itself >> is described in an older article >> [https://medium.com/@svenvc/reddit-st-in-10-cool-pharo-classes-1b5327ca0740]. >> Note that, by definition, there is no HTTPS/TLS variant. >> >> If you manage to break this server with (a) malicious request(s) in such a >> way that you can explain what you did for others to confirm your approach, >> you not only help me/us improve the code, but earn eternal fame as well ;-) >> >> Sven >> >> PS: I hope I won't regret this, I am looking for constructive criticism. >> >> >> -- >> Sven Van Caekenberghe >> Proudly supporting Pharo >> http://pharo.org >> http://association.pharo.org >> http://consortium.pharo.org >> >
[Pharo-dev] Re: Hardening Zinc's Core HTTP Server
Both instances have been up for 5 days now, looking for more testers. > On 23 Sep 2021, at 17:03, Sven Van Caekenberghe wrote: > > Hi, > > Zinc HTTP Components [https://github.com/svenvc/zinc] has been a part of > Pharo since version 1.3 (2011). It is an open-source framework to deal with > the HTTP networking protocol, modelling all aspects involved. It also offers > both client and server functionality. > > The reliability of the code base has improved steadily over the years, thanks > to virtually all Pharo developers using it, directly or indirectly. Over the > summer a number of issues that popped up after Pharo 9 was released were > resolved. > > The robustness of the core HTTP server is one important aspect. To put this > quality further to the test, I deployed two servers with the same demo > Seaside application, Reddit.st, open to the internet, without any further > protections. > > - On Amazon AWS: http://34.245.183.130:1701 > > - On Microsoft Azure: http://51.137.72.94:8080 > > The application's source code can be found at > [https://github.com/svenvc/Reddit]. For the technically curious there are > also deploy instructions at > [https://github.com/svenvc/Reddit/blob/main/DEPLOY.md]. The demo app itself > is described in an older article > [https://medium.com/@svenvc/reddit-st-in-10-cool-pharo-classes-1b5327ca0740]. > Note that, by definition, there is no HTTPS/TLS variant. > > If you manage to break this server with (a) malicious request(s) in such a > way that you can explain what you did for others to confirm your approach, > you not only help me/us improve the code, but earn eternal fame as well ;-) > > Sven > > PS: I hope I won't regret this, I am looking for constructive criticism. > > > -- > Sven Van Caekenberghe > Proudly supporting Pharo > http://pharo.org > http://association.pharo.org > http://consortium.pharo.org >
[Pharo-dev] Re: About / and ZnUrl
Because ZnUrl>>#/ is pretty simple and not very intelligent. It is meant for path extension. Read the code if you are curious. What you probably want is ZnUrl>>#withRelativeReference: 'http://www.pharo.org/figures/' asUrl withRelativeReference: '/documentation/p.png'. "http://www.pharo.org/documentation/p.png; This implementation is based on an actual specification [RFC 3986 Section 5] with a big test suite. But it might still surprise you. It seems to work best when the base URL (the left one) points to an actual document (the current HTTP page) and the relative URL (the right one) is meant to refer from there. See the method comment and #testReferenceResolution. Sven > On 5 Sep 2021, at 22:01, stephane ducasse wrote: > > Hi sven > > may be you know the answer :) > > > 'http://www.pharo.org/figures/' asUrl / 'documentation/p.png' > "http://www.pharo.org/figures/documentation/p.png” > > so far so good. > > > 'http://www.pharo.org/figures/' asUrl / '/documentation/p.png' > "http://www.pharo.org/figures//documentation/p.png; > > Why not having > > "http://www.pharo.org/documentation/p.png” > > Because /documentation is absolute so the resolution could take that into > account. > > S
[Pharo-dev] Re: about URL isAbsolute
Stef, Yes, this is confusing. The first 2 forward slashed are part of the scheme, file:// or http:// Next comes the host:port, but that does not make too much sense for a File URL. Finally comes the path. For the syntax to make sense, you need a third / (to indicate that there is no host:port). If you inspect the ZnUrl object in your second example, 'file://toto.png' asUrl, you will see that is has toto.png as host, not as path. This is in fact an invalid File URL. The annoying thing is, there is no such thing as a relative File URL. https://en.wikipedia.org/wiki/File_URI_scheme Now, for your problem, I would not allow the use of file:// there, and just simpler file references (using #asFileReference), which can be relative. Sven PS: Also, the methods #isAbsolute and #isRelative do not make much sense on ZnUrl, see the method comment/implementation. > On 31 Aug 2021, at 14:54, stephane ducasse wrote: > > Hi > > I’m a bit confused. When I use isAbsolute I get > > 'file:///toto.png' asUrl isAbsolute > >>> false > > > 'file://toto.png' asUrl isAbsolute > >>> true > > > Now I confused because as Unix / is absolute so file://toto.png would be > relative. > and /foo is absolute. > > Are URLs so different than files? > Are the results I get correct? > > I ask this question in the context of Microdown if people write > > . >  > . > > S > >
[Pharo-dev] Re: FileLocator uninitialised ?
https://github.com/pharo-project/pharo/issues/9682
> On 23 Jul 2021, at 13:37, Marcus Denker wrote:
>
> Hi,
>
> yes, this looks strange.
>
> It seems that even calling #initialize does not fix it, thus it might be
> initialised on Startup, but this does not
> set the desktop location.
>
> Can you open an issue?
>
> Marcus
>
>
>> On 21 Jul 2021, at 21:54, Sven Van Caekenberghe wrote:
>>
>> Hi,
>>
>> It seems that in the latest Pharo 9 (downloaded minutes ago) the following:
>>
>> FileLocator desktop resolve.
>>
>> Pops up a dialog asking to set the location of {desktop} manually.
>>
>> I am guessing FileLocator did not properly initialise at startup. It seems
>> it is missing from the session manager.
>>
>> This is bad.
>>
>> Maybe the full list of classes that are supposed to be part of the session
>> manager's start up sequence need to be checked. Such a failure to initialise
>> are sometimes hard to find and can lead to subtle bugs.
>>
>> Sven
[Pharo-dev] FileLocator uninitialised ?
Hi,
It seems that in the latest Pharo 9 (downloaded minutes ago) the following:
FileLocator desktop resolve.
Pops up a dialog asking to set the location of {desktop} manually.
I am guessing FileLocator did not properly initialise at startup. It seems it
is missing from the session manager.
This is bad.
Maybe the full list of classes that are supposed to be part of the session
manager's start up sequence need to be checked. Such a failure to initialise
are sometimes hard to find and can lead to subtle bugs.
Sven
[Pharo-dev] Re: [Zdc]SocketStream>>#openConnectionToHostNamed:port: no longer signals am exception on macOS
> On 21 Jul 2021, at 13:37, Sven Van Caekenberghe wrote: > > Thank you, I will test this soon. I can confirm that it now works again as before: opening a TCP connection to a non-existing host:port gives a NetworkError (ConnectionTimedOut cannot connect to host:port). Thanks again! Sven
[Pharo-dev] Re: [Zdc]SocketStream>>#openConnectionToHostNamed:port: no longer signals am exception on macOS
Hi Pablo, Thank you, I will test this soon. Is there any chance that there were also network changes in the Linux VM ? Perhaps also related to "detecting exceptions and out-of-band data". We keep on struggling with https://github.com/pharo-project/pharo/issues/9565 where it seems that the same code runs fine on Pharo 7 and 8, while it seems to fail on Pharo 9. Sven > On 21 Jul 2021, at 13:06, teso...@gmail.com wrote: > > I have done a new release with the fix for the socket connection. > It was an issue in the kqueue code in OSX. > It was wrongly detecting exceptions and out-of-band data. > Please if you can update the VM and test it should be working. > > Cheers > Pablo > > On Mon, Jul 19, 2021 at 1:13 PM Torsten Bergmann wrote: > Hi Pablo, > > yes - Sven is right on that it could be improved. Thanks for taking > care of this important topic. > > Cheers, > Torsten > > > Gesendet: Montag, 19. Juli 2021 um 11:46 Uhr > Von: "teso...@gmail.com" > An: "Pharo Development List" > Betreff: [Pharo-dev] Re: [Zdc]SocketStream>>#openConnectionToHostNamed:port: > no longer signals am exception on macOS > Thanks, yes it is true, the version name is a mess. > We need to improve the version display. > I will take note of that and I will fix it. > > Cheers, > Pablo > > On Mon, Jul 19, 2021 at 11:08 AM Sven Van Caekenberghe wrote: > Hi Pablo, > > prometheus:2021-07-16 sven$ ./pharo --version > Pharo 9.0.0 built on Jul 6 2021 10:36:56 Compiler: 4.2.1 Compatible Apple > LLVM 11.0.3 (clang-1103.0.32.29) > Built from: ea8a3bfc - Commit: ea8a3bfc - Date: 2021-07-06 10:23:13 +0200 > prometheus:2021-07-16 sven$ ./pharo Pharo.image printVersion > [version] 'Pharo9.0.0' > 'Pharo-9.0.0+build.1532.sha.e58ef49051bf06cad56a2dda174b8e091a45d5df (64 Bit)' > prometheus:2021-07-16 sven$ ./pharo Pharo.image eval "(ZdcSocketStream > openConnectionToHostNamed: 'localhost' port: 12335) nextPut: 0; close. #foo" > > I have said this before, but the VM version is incomprehensible for normal > people, why can't it just be something like 10.1 like the rest of the world ? > > Also, for mere mortals it is impossible to find out if something changed > about the VM, let alone what, and how that all relates to specific versions. > > I know that you guys have a lot of work, and we are very grateful for it, but > this is how it looks from the outside. > > Thx, > > Sven > > > On 19 Jul 2021, at 10:56, teso...@gmail.com wrote: > > > > Hi, > >with the new VM I could not reproduce it. I think it was related with > > the issue of the NetNameResolver. > > Which VM version are you using? > > > > Thanks > > > > On Sun, Jul 18, 2021 at 3:51 PM Sven Van Caekenberghe wrote: > > Hi, > > > > It seems that the following consistently crashes a recent/latest Pharo 9 > > image/vm on macOS: > > > > (ZdcSocketStream openConnectionToHostNamed: 'localhost' port: 12335) > > nextPut: 0; close. > > > > Command line, you can try it as follows: > > > > $ ./pharo Pharo.image eval "(ZdcSocketStream openConnectionToHostNamed: > > 'localhost' port: 12335) nextPut: 0; close. #foo" > > > > You can replace ZdcSocketStream by SocketStream, same result. > > > > Before, trying to connect to a non-existing host:port resulted in a > > ConnectionTimedOut: Cannot connect to 127.0.0.1:12335. > > > > Now, the code just returns and the bad stream is then used as if it is OK, > > which results in a hard crash. > > > > Furthermore, there is no backtrace nor log. > > > > Sven > > > > > > -- > > Pablo Tesone. > > teso...@gmail.com > > > -- > Pablo Tesone. > teso...@gmail.com > > > -- > Pablo Tesone. > teso...@gmail.com
[Pharo-dev] Re: [Zdc]SocketStream>>#openConnectionToHostNamed:port: no longer signals am exception on macOS
Hi Pablo, prometheus:2021-07-16 sven$ ./pharo --version Pharo 9.0.0 built on Jul 6 2021 10:36:56 Compiler: 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.29) Built from: ea8a3bfc - Commit: ea8a3bfc - Date: 2021-07-06 10:23:13 +0200 prometheus:2021-07-16 sven$ ./pharo Pharo.image printVersion [version] 'Pharo9.0.0' 'Pharo-9.0.0+build.1532.sha.e58ef49051bf06cad56a2dda174b8e091a45d5df (64 Bit)' prometheus:2021-07-16 sven$ ./pharo Pharo.image eval "(ZdcSocketStream openConnectionToHostNamed: 'localhost' port: 12335) nextPut: 0; close. #foo" I have said this before, but the VM version is incomprehensible for normal people, why can't it just be something like 10.1 like the rest of the world ? Also, for mere mortals it is impossible to find out if something changed about the VM, let alone what, and how that all relates to specific versions. I know that you guys have a lot of work, and we are very grateful for it, but this is how it looks from the outside. Thx, Sven > On 19 Jul 2021, at 10:56, teso...@gmail.com wrote: > > Hi, >with the new VM I could not reproduce it. I think it was related with the > issue of the NetNameResolver. > Which VM version are you using? > > Thanks > > On Sun, Jul 18, 2021 at 3:51 PM Sven Van Caekenberghe wrote: > Hi, > > It seems that the following consistently crashes a recent/latest Pharo 9 > image/vm on macOS: > > (ZdcSocketStream openConnectionToHostNamed: 'localhost' port: 12335) > nextPut: 0; close. > > Command line, you can try it as follows: > > $ ./pharo Pharo.image eval "(ZdcSocketStream openConnectionToHostNamed: > 'localhost' port: 12335) nextPut: 0; close. #foo" > > You can replace ZdcSocketStream by SocketStream, same result. > > Before, trying to connect to a non-existing host:port resulted in a > ConnectionTimedOut: Cannot connect to 127.0.0.1:12335. > > Now, the code just returns and the bad stream is then used as if it is OK, > which results in a hard crash. > > Furthermore, there is no backtrace nor log. > > Sven > > > -- > Pablo Tesone. > teso...@gmail.com
[Pharo-dev] [Zdc]SocketStream>>#openConnectionToHostNamed:port: no longer signals am exception on macOS
Hi, It seems that the following consistently crashes a recent/latest Pharo 9 image/vm on macOS: (ZdcSocketStream openConnectionToHostNamed: 'localhost' port: 12335) nextPut: 0; close. Command line, you can try it as follows: $ ./pharo Pharo.image eval "(ZdcSocketStream openConnectionToHostNamed: 'localhost' port: 12335) nextPut: 0; close. #foo" You can replace ZdcSocketStream by SocketStream, same result. Before, trying to connect to a non-existing host:port resulted in a ConnectionTimedOut: Cannot connect to 127.0.0.1:12335. Now, the code just returns and the bad stream is then used as if it is OK, which results in a hard crash. Furthermore, there is no backtrace nor log. Sven
[Pharo-dev] Re: Serious NetNameResolver regression
thx. > On 16 Jul 2021, at 12:18, teso...@gmail.com wrote: > > I have updated the VMs, the issue should be resolved now. > > On Fri, Jul 16, 2021 at 12:10 PM Guillermo Polito > wrote: > Hi all, > > it seems we need a new VM release. > The issue seems fixed since ~6 months ago > > https://github.com/pharo-project/opensmalltalk-vm/commit/bff77946691617acce104d8f38d60242fa1cc2bb > > Pablo is updating the stable VMs just in this moment. > > G > > > El 16 jul 2021, a las 12:07, Sven Van Caekenberghe escribió: > > > > Anyway, for now, I added guards: > > > > https://github.com/svenvc/zinc/commit/cac2cb36410c24e9070f850b641e0cd02a05deb8 > > https://github.com/svenvc/zodiac/commit/b12ba07f93ab73afad2523d75149c8b5440b2c64 > > > > but the core problem remains. > > > >> On 15 Jul 2021, at 21:35, Gabriel Cotelli wrote: > >> > >> You're right Sven. The code in the image looks exactly the same betwen > >> Pharo 8 and 9 but the behavior is different. > >> > >> On Thu, Jul 15, 2021 at 3:40 PM Sven Van Caekenberghe wrote: > >> > >> > >>> On 15 Jul 2021, at 16:42, Gabriel Cotelli wrote: > >>> > >>> Just doing > >>> NetNameResolver > >>> primStartLookupOfName:'unknown-stfx.eu';primNameLookupResult > >>> produces the bogus result. And both methods only call primitives in the > >>> SocketPlugin. So I think that no image code is responsible for the > >>> behavior change. > >> > >> I don't know, but your example is not good. You have to wait else the > >> result is always 0.0.0.0 > >> > >> Pharo 7 > >> > >> NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; > >> waitForCompletionUntil: 5 > >> > >> false (signal exception) > >> > >> Pharo 9 > >> > >> NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; > >> waitForCompletionUntil: 5 > >> > >> true (bogus 0.0.0.0 result) > >> > >>> On Thu, Jul 15, 2021 at 11:36 AM Sven Van Caekenberghe > >>> wrote: > >>> Hi, > >>> > >>> It seems that we have a serious NetNameResolver regression: instead of > >>> signalling an exception, a bogus value is returned. > >>> > >>> Pharo 7: > >>> > >>> [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: > >>> NameLookupFailure do: [ :exception | exception ]. > >>> > >>> "NameLookupFailure: cannot resolve 'unknown-stfx.eu'" > >>> > >>> Pharo 9: > >>> > >>> [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: > >>> NameLookupFailure do: [ :exception | exception ]. > >>> > >>> "0.0.0.0" > >>> > >>> This is bad. What is even worse is that the following kills your image > >>> without leaving any trace (on macOS): > >>> > >>> ZnClient new get: 'http://unknown-stfx.eu'. > >>> > >>> Because it tries to connect to 0.0.0.0 > >>> > >>> On linux, at least there is a backtrace: > >>> > >>> sven@stfx-1:~/pharo$ rlwrap ./pharo reddit.image NeoConsole repl > >>> NeoConsole > >>> Pharo-9.0.0+build.1528.sha.29269ecfac2cbf6a56dfee232b7d3355e5cb77bd (64 > >>> Bit) > >>> pharo> NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10. > >>> > >>> 0.0.0.0 > >>> pharo> ZnClient new get: 'http://unknown-stfx.eu'. > >>> > >>> ConnectionTimedOut: Cannot connect to 0.0.0.0:80 > >>> [ConnectionTimedOut signal: 'Cannot connect to ' > >>>, (NetNameResolver > >>> stringFromAddress: hostAddress) , ':' , port asString] in > >>> Socket>>connectTo:port:waitForConnectionFor: > >>>Receiver: a Socket[unconnected] > >>>Arguments and temporary variables: > >>>hostAddress:0.0.0.0 > >>>port: 80 > >>>timeout:30 > >>>Receiver's instance variables: > >>>semaphore: a Semaphore() > >>>socketHandle: #[198 113 243 96 0 0 0 0 240 65 61 2 0 0 0 > >>> 0] > >>>readSemaphore: a Semaphore() > >>>writeSemaphore: a Semaphore() > >>> > >>> Socket>>waitForConnectionFor:ifTimedOut: > >>> Socket>>connectTo:port:waitForConnectionFor: > >>> ZdcSocketStream(ZdcAbstractSocketStream)>>socketConnectTo:port: > >>> ZdcSocketStream(ZdcSimpleSocketStream)>>connectTo:port: > >>> ZdcSocketStream class(ZdcSimpleSocketStream > >>> class)>>openConnectionToHost:port:timeout: > >>> ZnNetworkingUtils>>socketStreamToUrlDirectly: > >>> ZnNetworkingUtils>>socketStreamToUrl: > >>> ZnNetworkingUtils class>>socketStreamToUrl: > >>> > >>> I had a quick look at changes to Network-Kernel but had no luck yet. > >>> > >>> Because of this running Zinc HTTP Components tests on macOS Pharo 9 also > >>> kills the image (ZnClientTest>>#testIfFailNonExistingHost). > >>> > >>> Of course, NetNameResolverTest does not do enough. > >>> > >>> Sven > > > -- > Pablo Tesone. > teso...@gmail.com
[Pharo-dev] Re: Serious NetNameResolver regression
Thank you! > On 16 Jul 2021, at 12:10, Guillermo Polito wrote: > > Hi all, > > it seems we need a new VM release. > The issue seems fixed since ~6 months ago > > https://github.com/pharo-project/opensmalltalk-vm/commit/bff77946691617acce104d8f38d60242fa1cc2bb > > Pablo is updating the stable VMs just in this moment. > > G > >> El 16 jul 2021, a las 12:07, Sven Van Caekenberghe escribió: >> >> Anyway, for now, I added guards: >> >> https://github.com/svenvc/zinc/commit/cac2cb36410c24e9070f850b641e0cd02a05deb8 >> https://github.com/svenvc/zodiac/commit/b12ba07f93ab73afad2523d75149c8b5440b2c64 >> >> but the core problem remains. >> >>> On 15 Jul 2021, at 21:35, Gabriel Cotelli wrote: >>> >>> You're right Sven. The code in the image looks exactly the same betwen >>> Pharo 8 and 9 but the behavior is different. >>> >>> On Thu, Jul 15, 2021 at 3:40 PM Sven Van Caekenberghe wrote: >>> >>> >>>> On 15 Jul 2021, at 16:42, Gabriel Cotelli wrote: >>>> >>>> Just doing >>>> NetNameResolver >>>> primStartLookupOfName:'unknown-stfx.eu';primNameLookupResult >>>> produces the bogus result. And both methods only call primitives in the >>>> SocketPlugin. So I think that no image code is responsible for the >>>> behavior change. >>> >>> I don't know, but your example is not good. You have to wait else the >>> result is always 0.0.0.0 >>> >>> Pharo 7 >>> >>> NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; >>> waitForCompletionUntil: 5 >>> >>> false (signal exception) >>> >>> Pharo 9 >>> >>> NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; >>> waitForCompletionUntil: 5 >>> >>> true (bogus 0.0.0.0 result) >>> >>>> On Thu, Jul 15, 2021 at 11:36 AM Sven Van Caekenberghe >>>> wrote: >>>> Hi, >>>> >>>> It seems that we have a serious NetNameResolver regression: instead of >>>> signalling an exception, a bogus value is returned. >>>> >>>> Pharo 7: >>>> >>>> [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: >>>> NameLookupFailure do: [ :exception | exception ]. >>>> >>>> "NameLookupFailure: cannot resolve 'unknown-stfx.eu'" >>>> >>>> Pharo 9: >>>> >>>> [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: >>>> NameLookupFailure do: [ :exception | exception ]. >>>> >>>> "0.0.0.0" >>>> >>>> This is bad. What is even worse is that the following kills your image >>>> without leaving any trace (on macOS): >>>> >>>> ZnClient new get: 'http://unknown-stfx.eu'. >>>> >>>> Because it tries to connect to 0.0.0.0 >>>> >>>> On linux, at least there is a backtrace: >>>> >>>> sven@stfx-1:~/pharo$ rlwrap ./pharo reddit.image NeoConsole repl >>>> NeoConsole >>>> Pharo-9.0.0+build.1528.sha.29269ecfac2cbf6a56dfee232b7d3355e5cb77bd (64 >>>> Bit) >>>> pharo> NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10. >>>> >>>> 0.0.0.0 >>>> pharo> ZnClient new get: 'http://unknown-stfx.eu'. >>>> >>>> ConnectionTimedOut: Cannot connect to 0.0.0.0:80 >>>> [ConnectionTimedOut signal: 'Cannot connect to ' >>>> , (NetNameResolver >>>> stringFromAddress: hostAddress) , ':' , port asString] in >>>> Socket>>connectTo:port:waitForConnectionFor: >>>> Receiver: a Socket[unconnected] >>>> Arguments and temporary variables: >>>> hostAddress:0.0.0.0 >>>> port: 80 >>>> timeout:30 >>>> Receiver's instance variables: >>>> semaphore: a Semaphore() >>>> socketHandle: #[198 113 243 96 0 0 0 0 240 65 61 2 0 0 0 0] >>>> readSemaphore: a Semaphore() >>>> writeSemaphore: a Semaphore() >>>> >>>> Socket>>waitForConnectionFor:ifTimedOut: >>>> Socket>>connectTo:port:waitForConnectionFor: >>>> ZdcSocketStream(ZdcAbstractSocketStream)>>socketConnectTo:port: >>>> ZdcSocketStream(ZdcSimpleSocketStream)>>connectTo:port: >>>> ZdcSocketStream class(ZdcSimpleSocketStream >>>> class)>>openConnectionToHost:port:timeout: >>>> ZnNetworkingUtils>>socketStreamToUrlDirectly: >>>> ZnNetworkingUtils>>socketStreamToUrl: >>>> ZnNetworkingUtils class>>socketStreamToUrl: >>>> >>>> I had a quick look at changes to Network-Kernel but had no luck yet. >>>> >>>> Because of this running Zinc HTTP Components tests on macOS Pharo 9 also >>>> kills the image (ZnClientTest>>#testIfFailNonExistingHost). >>>> >>>> Of course, NetNameResolverTest does not do enough. >>>> >>>> Sven
[Pharo-dev] Re: Serious NetNameResolver regression
Anyway, for now, I added guards: https://github.com/svenvc/zinc/commit/cac2cb36410c24e9070f850b641e0cd02a05deb8 https://github.com/svenvc/zodiac/commit/b12ba07f93ab73afad2523d75149c8b5440b2c64 but the core problem remains. > On 15 Jul 2021, at 21:35, Gabriel Cotelli wrote: > > You're right Sven. The code in the image looks exactly the same betwen Pharo > 8 and 9 but the behavior is different. > > On Thu, Jul 15, 2021 at 3:40 PM Sven Van Caekenberghe wrote: > > > > On 15 Jul 2021, at 16:42, Gabriel Cotelli wrote: > > > > Just doing > > NetNameResolver primStartLookupOfName:'unknown-stfx.eu';primNameLookupResult > > produces the bogus result. And both methods only call primitives in the > > SocketPlugin. So I think that no image code is responsible for the behavior > > change. > > I don't know, but your example is not good. You have to wait else the result > is always 0.0.0.0 > > Pharo 7 > > NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; > waitForCompletionUntil: 5 > > false (signal exception) > > Pharo 9 > > NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; > waitForCompletionUntil: 5 > > true (bogus 0.0.0.0 result) > > > On Thu, Jul 15, 2021 at 11:36 AM Sven Van Caekenberghe wrote: > > Hi, > > > > It seems that we have a serious NetNameResolver regression: instead of > > signalling an exception, a bogus value is returned. > > > > Pharo 7: > > > > [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: > > NameLookupFailure do: [ :exception | exception ]. > > > > "NameLookupFailure: cannot resolve 'unknown-stfx.eu'" > > > > Pharo 9: > > > > [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: > > NameLookupFailure do: [ :exception | exception ]. > > > > "0.0.0.0" > > > > This is bad. What is even worse is that the following kills your image > > without leaving any trace (on macOS): > > > > ZnClient new get: 'http://unknown-stfx.eu'. > > > > Because it tries to connect to 0.0.0.0 > > > > On linux, at least there is a backtrace: > > > > sven@stfx-1:~/pharo$ rlwrap ./pharo reddit.image NeoConsole repl > > NeoConsole > > Pharo-9.0.0+build.1528.sha.29269ecfac2cbf6a56dfee232b7d3355e5cb77bd (64 Bit) > > pharo> NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10. > > > > 0.0.0.0 > > pharo> ZnClient new get: 'http://unknown-stfx.eu'. > > > > ConnectionTimedOut: Cannot connect to 0.0.0.0:80 > > [ConnectionTimedOut signal: 'Cannot connect to ' > > , (NetNameResolver > > stringFromAddress: hostAddress) , ':' , port asString] in > > Socket>>connectTo:port:waitForConnectionFor: > > Receiver: a Socket[unconnected] > > Arguments and temporary variables: > > hostAddress:0.0.0.0 > > port: 80 > > timeout:30 > > Receiver's instance variables: > > semaphore: a Semaphore() > > socketHandle: #[198 113 243 96 0 0 0 0 240 65 61 2 0 0 0 > > 0] > > readSemaphore: a Semaphore() > > writeSemaphore: a Semaphore() > > > > Socket>>waitForConnectionFor:ifTimedOut: > > Socket>>connectTo:port:waitForConnectionFor: > > ZdcSocketStream(ZdcAbstractSocketStream)>>socketConnectTo:port: > > ZdcSocketStream(ZdcSimpleSocketStream)>>connectTo:port: > > ZdcSocketStream class(ZdcSimpleSocketStream > > class)>>openConnectionToHost:port:timeout: > > ZnNetworkingUtils>>socketStreamToUrlDirectly: > > ZnNetworkingUtils>>socketStreamToUrl: > > ZnNetworkingUtils class>>socketStreamToUrl: > > > > I had a quick look at changes to Network-Kernel but had no luck yet. > > > > Because of this running Zinc HTTP Components tests on macOS Pharo 9 also > > kills the image (ZnClientTest>>#testIfFailNonExistingHost). > > > > Of course, NetNameResolverTest does not do enough. > > > > Sven
[Pharo-dev] Re: Serious NetNameResolver regression
> On 15 Jul 2021, at 16:42, Gabriel Cotelli wrote: > > Just doing > NetNameResolver primStartLookupOfName:'unknown-stfx.eu';primNameLookupResult > produces the bogus result. And both methods only call primitives in the > SocketPlugin. So I think that no image code is responsible for the behavior > change. I don't know, but your example is not good. You have to wait else the result is always 0.0.0.0 Pharo 7 NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; waitForCompletionUntil: 5 false (signal exception) Pharo 9 NetNameResolver primStartLookupOfName:'unknown-stfx.eu'; waitForCompletionUntil: 5 true (bogus 0.0.0.0 result) > On Thu, Jul 15, 2021 at 11:36 AM Sven Van Caekenberghe wrote: > Hi, > > It seems that we have a serious NetNameResolver regression: instead of > signalling an exception, a bogus value is returned. > > Pharo 7: > > [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: > NameLookupFailure do: [ :exception | exception ]. > > "NameLookupFailure: cannot resolve 'unknown-stfx.eu'" > > Pharo 9: > > [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: > NameLookupFailure do: [ :exception | exception ]. > > "0.0.0.0" > > This is bad. What is even worse is that the following kills your image > without leaving any trace (on macOS): > > ZnClient new get: 'http://unknown-stfx.eu'. > > Because it tries to connect to 0.0.0.0 > > On linux, at least there is a backtrace: > > sven@stfx-1:~/pharo$ rlwrap ./pharo reddit.image NeoConsole repl > NeoConsole > Pharo-9.0.0+build.1528.sha.29269ecfac2cbf6a56dfee232b7d3355e5cb77bd (64 Bit) > pharo> NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10. > > 0.0.0.0 > pharo> ZnClient new get: 'http://unknown-stfx.eu'. > > ConnectionTimedOut: Cannot connect to 0.0.0.0:80 > [ConnectionTimedOut signal: 'Cannot connect to ' > , (NetNameResolver stringFromAddress: > hostAddress) , ':' , port asString] in > Socket>>connectTo:port:waitForConnectionFor: > Receiver: a Socket[unconnected] > Arguments and temporary variables: > hostAddress:0.0.0.0 > port: 80 > timeout:30 > Receiver's instance variables: > semaphore: a Semaphore() > socketHandle: #[198 113 243 96 0 0 0 0 240 65 61 2 0 0 0 0] > readSemaphore: a Semaphore() > writeSemaphore: a Semaphore() > > Socket>>waitForConnectionFor:ifTimedOut: > Socket>>connectTo:port:waitForConnectionFor: > ZdcSocketStream(ZdcAbstractSocketStream)>>socketConnectTo:port: > ZdcSocketStream(ZdcSimpleSocketStream)>>connectTo:port: > ZdcSocketStream class(ZdcSimpleSocketStream > class)>>openConnectionToHost:port:timeout: > ZnNetworkingUtils>>socketStreamToUrlDirectly: > ZnNetworkingUtils>>socketStreamToUrl: > ZnNetworkingUtils class>>socketStreamToUrl: > > I had a quick look at changes to Network-Kernel but had no luck yet. > > Because of this running Zinc HTTP Components tests on macOS Pharo 9 also > kills the image (ZnClientTest>>#testIfFailNonExistingHost). > > Of course, NetNameResolverTest does not do enough. > > Sven
[Pharo-dev] Serious NetNameResolver regression
Hi, It seems that we have a serious NetNameResolver regression: instead of signalling an exception, a bogus value is returned. Pharo 7: [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: NameLookupFailure do: [ :exception | exception ]. "NameLookupFailure: cannot resolve 'unknown-stfx.eu'" Pharo 9: [ NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10 ] on: NameLookupFailure do: [ :exception | exception ]. "0.0.0.0" This is bad. What is even worse is that the following kills your image without leaving any trace (on macOS): ZnClient new get: 'http://unknown-stfx.eu'. Because it tries to connect to 0.0.0.0 On linux, at least there is a backtrace: sven@stfx-1:~/pharo$ rlwrap ./pharo reddit.image NeoConsole repl NeoConsole Pharo-9.0.0+build.1528.sha.29269ecfac2cbf6a56dfee232b7d3355e5cb77bd (64 Bit) pharo> NetNameResolver addressForName: 'unknown-stfx.eu' timeout: 10. 0.0.0.0 pharo> ZnClient new get: 'http://unknown-stfx.eu'. ConnectionTimedOut: Cannot connect to 0.0.0.0:80 [ConnectionTimedOut signal: 'Cannot connect to ' , (NetNameResolver stringFromAddress: hostAddress) , ':' , port asString] in Socket>>connectTo:port:waitForConnectionFor: Receiver: a Socket[unconnected] Arguments and temporary variables: hostAddress:0.0.0.0 port: 80 timeout:30 Receiver's instance variables: semaphore: a Semaphore() socketHandle: #[198 113 243 96 0 0 0 0 240 65 61 2 0 0 0 0] readSemaphore: a Semaphore() writeSemaphore: a Semaphore() Socket>>waitForConnectionFor:ifTimedOut: Socket>>connectTo:port:waitForConnectionFor: ZdcSocketStream(ZdcAbstractSocketStream)>>socketConnectTo:port: ZdcSocketStream(ZdcSimpleSocketStream)>>connectTo:port: ZdcSocketStream class(ZdcSimpleSocketStream class)>>openConnectionToHost:port:timeout: ZnNetworkingUtils>>socketStreamToUrlDirectly: ZnNetworkingUtils>>socketStreamToUrl: ZnNetworkingUtils class>>socketStreamToUrl: I had a quick look at changes to Network-Kernel but had no luck yet. Because of this running Zinc HTTP Components tests on macOS Pharo 9 also kills the image (ZnClientTest>>#testIfFailNonExistingHost). Of course, NetNameResolverTest does not do enough. Sven
[Pharo-dev] Pharo 9 Spotter Input field does not track font changes
Hi, When using a large font in Pharo 9, the new Spotter input field does not seem to be tracking this change. Sven
[Pharo-dev] Metacello / Iceberg / GitHub master to main renaming
Hi, For new GitHub projects the default branch is now main instead of master. There is however code in Metacello / Iceberg / ... that tries a number of options if no branch is specified, but it is not yet aware of this change. Specifically: This does not work ./pharo reddit.image metacello install github://svenvc/Reddit BaselineOfReddit instead you have to say ./pharo reddit.image metacello install github://svenvc/Reddit:main/ BaselineOfReddit It took me half an hour to figure this out ;-) Sven
[Pharo-dev] Allowed Characters in a Class name ?
Hi, What are the allowed characters in a Class name ? Where is this defined/enforced in the system ? Does anyone know ? This is in the context of the following: https://github.com/svenvc/ston/issues/29 Thx, Sven
[Pharo-dev] Re: Open Sound Control Package for Pharo 9.0
I am 100% confident that we can fix this easily (i.e. portability across 7, 8 or 9), there is not a lot of code and it looks pretty straightforward at first sight. > On 21 May 2021, at 18:23, ducasse wrote: > > Thanks. > I do not think that it is good to reintroduce this super ugly class. at the > minimum it should be in a separate branch. > > S > >> On 21 May 2021, at 18:06, Günter Khyo via Pharo-dev >> wrote: >> >> Hi Stef, >> >> Great! I'll send you a pull-request for the "hotfix" this evening. As I >> mentioned to Sven, I'll do some rework on the implementation and >> documentation when I find the time. >> >> Günter >> >> On 5/21/21 5:55 PM, ducasse wrote: >>> I checked and all the tests pass on Pharo80. >>> Now I did a little pass on the tests. >>> I do not have the time to migrate this lib to P9. >>> But I would accept Pull Requests :) >>> >>> S. >>> On 21 May 2021, at 17:46, ducasse wrote: The old travis was telling that the code works in 6.1, 7 and 8.0 I added GithubActions for Pharo 80 and I will see. S > On 21 May 2021, at 17:40, ducasse wrote: > > Hi gunter > > I’m maintaining OSC > https://github.com/Ducasse/OSC > and TUIO > https://github.com/Ducasse/TUIO > > Please do some PR if something does not work. > > S > >> On 21 May 2021, at 14:02, Günter Khyo via Pharo-dev >> wrote: >> >> Hi, >> >> I have ported the Open Sound Control package to Pharo 9.0. Since the >> Catalog Browser is marked as legacy, I was wondering how to submit the >> patch. Is there a git repository? I attached the fileout to this email >> in case anybody wants to take a look at the package or knows how/where >> to add it to Pharo 9.0, >> >> Patch Note: >> >> OSC relies on the removed RWTextOrBinaryStream class which has very >> specific behavior that I could not emulate using the available stream >> and codec classes, so I pulled in the class from Squeak 5, renamed it to >> OSCStream and added it to the OSC package. This seemed to be the easiest >> way to do it without changing the OSC implementation, but I welcome any >> suggestions for a cleaner solution. >> >> Günter >> >> > >>> >
[Pharo-dev] Re: Open Sound Control Package for Pharo 9.0
Hi Günter, Thanks for answering, I expected that you would mention this aspect. I don't know about OSC, but from your description it sounds like it basically is a binary protocol, where some sequences of bytes are to be interpreted as ASCII encoded strings. I would just work on binary streams and do the encoding/decoding as needed. ASCII is trivial, you can either do: ZnCharacterEncoder ascii encodeString: 'OSC' or 'OSC' asByteArray And the inverse is either ZnCharacterEncoder ascii decodeBytes: #[79 83 67]. or #[79 83 67] asString. The OSC spec says: OSC-string A sequence of non-null ASCII characters followed by a null, followed by 0-3 additional null characters to make the total number of bits a multiple of 32. One approach could be: #writeOSCString: string on: stream stream nextPutAll: (ZnCharacterEncoder ascii encodeString: string). stream nextPut: 0. (string size + 1) // 4 timesRepeat: [ stream nextPut: 0 ] #readOSCStringFrom: stream | string | string := ByteArray streamContents: [ :out | [ stream peekFor: 0 ] whileFalse: [ out nextPut: stream next ] ]. (string size + 1) // 4 timesRepeat: [ stream next ]. ^ ZnCharacterEncoder ascii decodeBytes: string My point being that you probably don't need RWTextOrBinaryStream at all. Of course, if you ported existing code and don't want to change it much, your solution is acceptable. Sven > On 21 May 2021, at 15:39, Günter Khyo wrote: > > A RWTextOrBinaryStream instance can operate in binary or ascii mode, the mode > can be switched at any time by sending ascii or binary, and some methods are > mode-sensitive, e.g., contents will respond with a ByteArray in binary mode > and a ByteString in ascii mode. (OSC uses ASCII encoding for addresses and > strings, and binary formats for numbers and blobs.) > > The OSC testcases and the parser rely on this behavior. > > On 5/21/21 3:00 PM, Sven Van Caekenberghe wrote: >> >>> On 21 May 2021, at 14:02, Günter Khyo via Pharo-dev >>> >>> wrote: >>> >>> Hi, >>> >>> I have ported the Open Sound Control package to Pharo 9.0. Since the >>> Catalog Browser is marked as legacy, I was wondering how to submit the >>> patch. Is there a git repository? I attached the fileout to this email in >>> case anybody wants to take a look at the package or knows how/where to add >>> it to Pharo 9.0, >>> >>> Patch Note: >>> >>> OSC relies on the removed RWTextOrBinaryStream class which has very >>> specific behavior that I could not emulate using the available stream and >>> codec classes, so I pulled in the class from Squeak 5, renamed it to >>> OSCStream and added it to the OSC package. This seemed to be the easiest >>> way to do it without changing the OSC implementation, but I welcome any >>> suggestions for a cleaner solution. >>> >> I am curious, what was the specific behaviour that you could not emulate ? >> >> >>> Günter >>> >>> >>>
[Pharo-dev] Re: Open Sound Control Package for Pharo 9.0
> On 21 May 2021, at 14:02, Günter Khyo via Pharo-dev > wrote: > > Hi, > > I have ported the Open Sound Control package to Pharo 9.0. Since the Catalog > Browser is marked as legacy, I was wondering how to submit the patch. Is > there a git repository? I attached the fileout to this email in case anybody > wants to take a look at the package or knows how/where to add it to Pharo > 9.0, > > Patch Note: > > OSC relies on the removed RWTextOrBinaryStream class which has very specific > behavior that I could not emulate using the available stream and codec > classes, so I pulled in the class from Squeak 5, renamed it to OSCStream and > added it to the OSC package. This seemed to be the easiest way to do it > without changing the OSC implementation, but I welcome any suggestions for a > cleaner solution. I am curious, what was the specific behaviour that you could not emulate ? > Günter > >
[Pharo-dev] Re: Pharo 8.0 crash on resizing with Moom
Hi Bernard, I run a dual monitor macOS based setup as well. In general there is indeed a hard to track down issue with going full screen. What does work for me is going 'almost' full screen, that is, leave the menu bar visible. You can do that with Option-click on the green expand button. I think this is called Zoom. That way everything remains rock solid for me (as in images are running for days on end). I like this better than real full screen because the menu bar has useful functionality (especially the right hand side icons), even though Pharo does not need or use it. HTH, Sven > On 10 Apr 2021, at 11:46, Bernhard Pieber wrote: > > Hi everyone, > > Is this the right place to report a Pharo 8.0 crash? > > I am on a MacBookPro16,4 on macOS Big Sur Version 11.2.3 (20D91) with an > external 30" Apple Cinema Display (2500 x 1600). > > Here is what I did: > > I downloaded PharoLauncher. > > I created a new image using the official distribution Pharo 8.0 - 64bit > (stable). > > I launched the new image. > > I clicked on light theme. > > I resized to full screen using Moom. => It crashed. > > I could reproduce it three out of five times I tried those steps. > > Let me add that Moom works very reliably with all other applications. > Resizing like that is in my muscular memory. > > Cheers, > Bernhard > I dow >
[Pharo-dev] Re: Broken window alert
And here is the PR, I hope it is OK: https://github.com/pharo-project/pharo/pull/8977 > On 7 Apr 2021, at 11:53, Sven Van Caekenberghe wrote: > > Hi Pablo, Guille, > > Thanks for the links. > > You can consider the Zn/Zdc failures fixed. Something changed recently with > the Google query service so that it now includes a redirect. The full > ZnClient follows those without any problems, but these tests are much more > primitive and can't handle a redirect. I now use DuckDuckGo for the test at > this level. > > Here is the first change > > > https://github.com/svenvc/zinc/commit/75710f9832e6ddad8a477d061978149b1d08c6b8 > > and the second > > > https://github.com/svenvc/zodiac/commit/cd4c4ee7e7cc8eb3624b4c7b4f8af861481cd37e > > my own tests for Pharo 7, 8 and 9 are green > > https://github.com/svenvc/zinc/actions/runs/725549812 > > I will now try to make a PR on a current Pharo 9 image. > > Sven > >> On 6 Apr 2021, at 22:58, Guillermo Polito wrote: >> >> >> >>> El 6 abr 2021, a las 22:55, Sven Van Caekenberghe escribió: >>> >>> I agree, could you post a link to or a list of the broken tests ? >> >> Sure :) >> >> The CI: >> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/ >> >> Last Three Jobs: >> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/1286/ >> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/1285/ >> https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/1284/ >> >> >>> >>>> On 6 Apr 2021, at 22:54, Guillermo Polito >>>> wrote: >>>> >>>> Hi Guys, >>>> >>>> With Pablo we were discussing that it would be nice to stop with the >>>> integrations until we fix the broken tests. >>>> We have crossed the 20 broken tests for more than 20 builds. >>>> We should put some energy in fixing those, but we are afraid that if we >>>> continue this way, eventually it will become unmanageable. >>>> >>>> Cheers, >>>> Guille >> >
[Pharo-dev] Re: Broken window alert
Hi Pablo, Guille, Thanks for the links. You can consider the Zn/Zdc failures fixed. Something changed recently with the Google query service so that it now includes a redirect. The full ZnClient follows those without any problems, but these tests are much more primitive and can't handle a redirect. I now use DuckDuckGo for the test at this level. Here is the first change https://github.com/svenvc/zinc/commit/75710f9832e6ddad8a477d061978149b1d08c6b8 and the second https://github.com/svenvc/zodiac/commit/cd4c4ee7e7cc8eb3624b4c7b4f8af861481cd37e my own tests for Pharo 7, 8 and 9 are green https://github.com/svenvc/zinc/actions/runs/725549812 I will now try to make a PR on a current Pharo 9 image. Sven > On 6 Apr 2021, at 22:58, Guillermo Polito wrote: > > > >> El 6 abr 2021, a las 22:55, Sven Van Caekenberghe escribió: >> >> I agree, could you post a link to or a list of the broken tests ? > > Sure :) > > The CI: > https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/ > > Last Three Jobs: > https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/1286/ > https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/1285/ > https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/Pharo9.0/1284/ > > >> >>> On 6 Apr 2021, at 22:54, Guillermo Polito wrote: >>> >>> Hi Guys, >>> >>> With Pablo we were discussing that it would be nice to stop with the >>> integrations until we fix the broken tests. >>> We have crossed the 20 broken tests for more than 20 builds. >>> We should put some energy in fixing those, but we are afraid that if we >>> continue this way, eventually it will become unmanageable. >>> >>> Cheers, >>> Guille >
[Pharo-dev] Re: Broken window alert
I agree, could you post a link to or a list of the broken tests ? > On 6 Apr 2021, at 22:54, Guillermo Polito wrote: > > Hi Guys, > > With Pablo we were discussing that it would be nice to stop with the > integrations until we fix the broken tests. > We have crossed the 20 broken tests for more than 20 builds. > We should put some energy in fixing those, but we are afraid that if we > continue this way, eventually it will become unmanageable. > > Cheers, > Guille
[Pharo-dev] Re: Call for Beta-testers Pharo ARM64 JIT
give more details of the current status, and the > following steps including Apple Silicon, Windows ARM64 and Linux Open Build > System support. > > ## Current Status > > Our objective is to have a running JIT for the new aarch64 architecture (ARM > 64bits). This task includes not only a new backend for the JIT compiler but > also adding support for external libraries, dependencies and the build > process. > This means having a working VM with comparable features as the one existing > in Intel X64. We are targeting all the major operating systems running in > this platform (Linux, OSX, Windows). > Each of them present different restrictions and conditions. > > This is the current status: > > - We implemented a full backend for the JIT compiler targeting aarch64. > - All the image side was adapted to run on it, tested on Ubuntu ARM 64 bits. > - We added support for: Iceberg (Libgit) / Athens (Cairo) / SDL / GTK > - We implemented a LibFFI-based FFI backend as the default one for Pharo 9 in > aarch64 (next to come in all platforms). > This opens the door to ease port the features to other platforms and OSes. > > ## Following Steps and Open Betas: Linux Open Build System (OBS), Windows > ARM64 and Apple Silicon > > Linux Systems: In the following days, we will also support Raspbian (Debian) > and Manjaro on ARM64. For doing so, we are pushing the last details in having > a single Linux build system through OBS. So, if you want to start doing > beta-testing of these versions please contact us. A public beta will be open > in around two weeks. > > Windows Systems: We have extended the build process to fully support > Microsoft Visual Studio compilers and more flexibility to select the targets, > also we are building it to run in Windows ARM. To correctly run the VM in > Windows it is needed to build all dependencies for aarch64. In the following > weeks, we expect to have a working Non-JIT version and a JIT version. The > remaining points to have a JIT version are related with the build process as > the API of the operating system has not changed from X64 to aarch64. > > OSX Systems: Our third target is to have a working version for the newest > Apple silicon. We are acquiring the corresponding hardware to test and to > address the differences in the API exposed to JIT applications. As it is the > case of the Windows VM, there is not need to change the machine code > generation backend; but to compile external libraries, and particularities of > the new OS version. > > Thanks for your support, and again, if you like to start beta testing the VM > please contact us. In the meantime, we will continue giving you news about > the current state and where are we going. > > The consortium would like to particularly thank Schmidt Buro and Lifeware for > their contracts. > > Regards, > > Pablo in behalf of the Pharo Consortium Engineers > > Stéphane Ducasse > http://stephane.ducasse.free.fr / http://www.pharo.org > 03 59 35 87 52 > Assistant: Aurore Dalle > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley, > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France > -- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
[Pharo-dev] Re: P3Converter and DateAndTime>>#readFrom: performance
Hi Estaban,
I took some time to properly look at your remarks/suggestions.
You are right, the chronology conversions could be improved and made faster.
I committed the following
https://github.com/svenvc/P3/commit/dd06eb33d0147060e0b6f44c704e2e298f55c0d4
"As noted/analysed by Esteban A. Maringolo, incoming conversions for
Timestamp/DateAndTime and Date especially were slow. He suggested to take
advantage of the known ISO format.
Introduction of P3ISO8601Parser that implements a fast, non-checking parser
specifically for the PSQL 8601 format.
Use P3ISO8601Parser for the chronology conversions in P3Converter, for a 5x to
10x speedup.
Note that for this to work, the connection output format must be ISO, the
default."
I did the following small test to confirm the improvements:
p3client := P3Client url: 'psql://sven@localhost:5432/'.
p3client execute: 'DROP TABLE IF EXISTS table1'.
p3client execute: 'CREATE TABLE table1 (id SERIAL PRIMARY KEY, name TEXT, t1
TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, t2 TIMESTAMP WITH TIME
ZONE DEFAULT CURRENT_TIMESTAMP, t0 TIME DEFAULT CURRENT_TIME, d1 DATE DEFAULT
CURRENT_DATE)'.
1 to: 100 do: [ :each | p3client execute: 'INSERT INTO table1 (name) VALUES
(''n-' , (each + 999 atRandom) asString , ''')' ].
[ p3client query: 'SELECT * FROM table1' ] bench.
p3client close.
This query (6 columns, 100 rows) went from 68.332 per second to 446.711 per
second on my machine.
Thanks again for your feedback, it was most useful.
Do these changes work for you ?
Regards,
Sven
> On 12 Oct 2020, at 15:57, Sven Van Caekenberghe wrote:
>
> Hi Esteban,
>
> Thanks for this detailed feedback, this is most interesting.
>
> I don't have enough time to look at this in detail, I will do so later, but
> here is some initial feedback.
>
> Note that PostreSQL supports many different date/time formats and that the
> user can set preferences per connection, hence some flexibility in parsing is
> probably needed, and that is one reason why the parsing is implemented the
> way it is.
>
> Check section 8.5.2. from
> https://www.postgresql.org/docs/13/datatype-datetime.html
>
> Note that all of the following give the same result:
>
> Date readFrom: 'March 2 2020' readStream.
> Date readFrom: '3/2/2020' readStream.
> Date readFrom: '2020-03-02' readStream.
>
> Date parsing is probably slow because it has to deal with all these formats.
>
> Of course, parsing a fixed well-known format is always faster.
>
> The timezones are tricky too, as preferences can be set per connection, so we
> have to be careful.
>
> I will come back to this.
>
> Sven
>
>> On 12 Oct 2020, at 15:04, Esteban Maringolo wrote:
>>
>> Hi,
>>
>> I recently stumbled into some performance issue I had, so I profiled
>> it and got it fixed, but by doing that I also noticed a huge time
>> spent when reading DateAndTimes and Dates (which also uses the former)
>> and the culprit seems to be `Date class>>#readFrom:`.
>>
>> Digging into this, I noticed that this was called from
>> `DateAndTime>>#readFrom:`, but in particular 90% of the time spent was
>> when instantiating the Date part of it, because it does unnecessary
>> lookup of the index of Month when input stream only have a month
>> index.
>>
>> What I did then was to change the default behavior of DateAndTime
>> class>>#readFrom:
>>
>> readFrom: aStream defaultOffset: defaultOffset
>> "Parse and return a new DateAndTime instance from stream,
>> as a Date, an optional Time and an optional TimeZone offset.
>> The time defaults to midnight, the timezone to defaultOffset"
>> "self readFrom: '2013-03-04T23:47:52.876+01:00' readStream"
>>
>> "From this"
>> date := Date readFrom: aStream.
>>
>> "To this"
>> date := Date readFrom: (aStream next: 10) pattern: '-mm-dd'.
>>
>> And also in P3Converter to specify the pattern directly.
>>
>> P3Converter>>#convertDateFrom: bytes length: length description: description
>> ^Date readFrom: (self asciiStreamFor: bytes length: length) pattern:
>> '-mm-dd'
>>
>> With these two changes I got a 8x-10x speedup when reading
>> Date/DateAndTime from the database.
>>
>> Then I dig a little further and in P3Converter added a little check to
>> avoid translating the offset of a DateAndTime when it is not needed.
>>
>> P3Converter>>#convertDateAndTimeWithoutTimezoneFrom: bytes length:
>> length description: description
>> "TIMESTAMP WITHOUT TIME ZONE (TIMESTAMP) is stored internally in
>> Postgres the way it was inserted, its r
[Pharo-dev] Re: P3Converter and DateAndTime>>#readFrom: performance
Hi Esteban, Thanks for this detailed feedback, this is most interesting. I don't have enough time to look at this in detail, I will do so later, but here is some initial feedback. Note that PostreSQL supports many different date/time formats and that the user can set preferences per connection, hence some flexibility in parsing is probably needed, and that is one reason why the parsing is implemented the way it is. Check section 8.5.2. from https://www.postgresql.org/docs/13/datatype-datetime.html Note that all of the following give the same result: Date readFrom: 'March 2 2020' readStream. Date readFrom: '3/2/2020' readStream. Date readFrom: '2020-03-02' readStream. Date parsing is probably slow because it has to deal with all these formats. Of course, parsing a fixed well-known format is always faster. The timezones are tricky too, as preferences can be set per connection, so we have to be careful. I will come back to this. Sven > On 12 Oct 2020, at 15:04, Esteban Maringolo wrote: > > Hi, > > I recently stumbled into some performance issue I had, so I profiled > it and got it fixed, but by doing that I also noticed a huge time > spent when reading DateAndTimes and Dates (which also uses the former) > and the culprit seems to be `Date class>>#readFrom:`. > > Digging into this, I noticed that this was called from > `DateAndTime>>#readFrom:`, but in particular 90% of the time spent was > when instantiating the Date part of it, because it does unnecessary > lookup of the index of Month when input stream only have a month > index. > > What I did then was to change the default behavior of DateAndTime > class>>#readFrom: > > readFrom: aStream defaultOffset: defaultOffset > "Parse and return a new DateAndTime instance from stream, > as a Date, an optional Time and an optional TimeZone offset. > The time defaults to midnight, the timezone to defaultOffset" > "self readFrom: '2013-03-04T23:47:52.876+01:00' readStream" > > "From this" > date := Date readFrom: aStream. > > "To this" > date := Date readFrom: (aStream next: 10) pattern: '-mm-dd'. > > And also in P3Converter to specify the pattern directly. > > P3Converter>>#convertDateFrom: bytes length: length description: description > ^Date readFrom: (self asciiStreamFor: bytes length: length) pattern: > '-mm-dd' > > With these two changes I got a 8x-10x speedup when reading > Date/DateAndTime from the database. > > Then I dig a little further and in P3Converter added a little check to > avoid translating the offset of a DateAndTime when it is not needed. > > P3Converter>>#convertDateAndTimeWithoutTimezoneFrom: bytes length: > length description: description > "TIMESTAMP WITHOUT TIME ZONE (TIMESTAMP) is stored internally in > Postgres the way it was inserted, its representation remains constant, > with no offset added. We use the timezone of the connection to do the > necessary shifting. This assumes that the timezones used during > insert/update and query are the same." > > | timestamp offset | > timestamp := DateAndTime readFrom: (self asciiStreamFor: bytes > length: length) defaultOffset: Duration zero. > offset := self timezone offsetForTimestamp: timestamp. > ^offset isZero "<-- added this check to avoid translation" >ifTrue: [ timestamp ] >ifFalse: [ timestamp translateTo: offset ] > > And the speedup went even higher. But I don't know whether this last > part is correct. > > Of course this seems as a quick and dirty fix, but I don't know > whether DateAndTime>>readFrom: should read any format or just ISO 8601 > formats nor whether the Date/DateAndTime coming from PostgreSQL are > always formatted that way. I ran my app's small test suite and haven't > had any issues, neither during development with these changes applied. > > I think there is a significant slowdown involved in the instantiation > of Dates that should be fixed, whether this fix is definitive or not > is why I'm asking here. > > Regards! > > Esteban A. Maringolo
Re: [Pharo-dev] 'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com
> On 30 Aug 2020, at 12:07, Christophe Demarey > wrote: > > Hi Dale, > >> Le 29 août 2020 à 19:40, Dale Henrichs a >> écrit : >> >> So close, the interactive tests are passing, but there is a use case >> that is popping up in a travis job[1] where this url >> http://smalltalkhub.com/mc/Swazoo/Swazoo/main is being used (no trailing >> slash) and this form of the url still produces the `raw` listing, >> > > Are you sure? For me, http://smalltalkhub.com/mc/Swazoo/Swazoo/main produces > the html listing For me as well.
Re: [Pharo-dev] 'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com
Thanks looks fine to me, Christophe, thank you for the great work. The non format=raw case results in a redirect but that should be acceptable, both URL works both and without trailing / > On 29 Aug 2020, at 00:05, Christophe Demarey > wrote: > > I understand that he new site is unusable if tools do not get the expected > input. Sorry for the inconvenience. I was just not aware of this as Pharo > uses the raw format. > I had to set up a CGI to produce an html listing from the raw file. > It should now be ok. Could you tell me if it works in GemStone now? > > Christophe > >> Le 28 août 2020 à 19:21, Dale Henrichs a >> écrit : >> >> Christophe, >> >> I appreciate your effort, but I assume that you aren't finished yet? >> >> When I use the url from your message I get a `Not Found` error (note that I >> explicitly included a trailing slash in the url): >> >> >> >> This is a bug (or omission) as well, since other Monticello sites produce >> the expected HTML output with or without a trailing slash. >> >> Without the trailing slash I get the same (non-html) response that was >> causing a problem before ... although something must have changed, because >> the GemStone command is now producing an error instead of an empty list ... >> Once you get the static site to produce the expected output, I'm confident >> that the GemStone errors will go away (this code has been running for about >> a decade on all Monticello html repositories). >> >> I suggest that you look at what is produced on SqueakSource >> (http://www.squeaksource.com/MooseSQL/) as an example of expected output of >> the mcz file listing ... this html page format has been used since 2003 for >> ALL (valid) monticello repositories. >> >> If it is not clear, the `?format=raw` option is a recent Pharo only option >> and when the dynamic site was running and the `?format-raw` option was >> OMITTED it produced output compatible with >> (http://www.squeaksource.com/MooseSQL/) ... >> >> Restoring HTML is *REQUIRED* for the static site to be a faithful >> replacement of the dynamic site... the format that is produced today is only >> compatible with the Pharo only `?format=raw` option. >> >> Bug or omission ... the static site is currently in worse shape than the >> dynamic site before the swap ... >> >> Dale >> >> On 8/28/20 2:12 AM, Christophe Demarey wrote: >>>> Le 28 août 2020 à 10:15, Sven Van Caekenberghe >>>> a écrit : >>>> >>>> >>>> >>>> >>>>> On 28 Aug 2020, at 10:13, Christophe Demarey >>>>> wrote: >>>>> >>>>> Smalltalkhub is now to able to distinguish between raw and not raw mcz >>>>> listing requests. >>>>> Ex: >>>>> >>>>> http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main?format=raw >>>>> >>>>> >>>>> http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main/ >>>>> >>>>> >>>>> I already use the apache directory index for another page so I will not >>>>> be able to modify this one. I guess the current index will « not work » >>>>> because glass expect some HTML structure. >>>>> Could you confirm and tell me what is the expected structure? >>>>> >>>> See my earlier mail with code. >>>> >>> Thanks Sven >>> >>> >>> >
Re: [Pharo-dev] 'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com
> On 28 Aug 2020, at 10:13, Christophe Demarey
> wrote:
>
> Smalltalkhub is now to able to distinguish between raw and not raw mcz
> listing requests.
> Ex:
> http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main?format=raw
> http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main/
>
> I already use the apache directory index for another page so I will not be
> able to modify this one. I guess the current index will « not work » because
> glass expect some HTML structure.
> Could you confirm and tell me what is the expected structure?
See my earlier mail with code.
> Regards,
> Christophe
>
>> Le 28 août 2020 à 09:37, Christophe Demarey a
>> écrit :
>>
>> Hi Dale,
>>
>> I would not call it a bug but an omission ;)
>> It is hard to guess that a tool is scraping an html page to get data.
>>
>> Nevertheless, I will take a look today to find a solution for Glass to work
>> again with smalltalkhub.
>> Could you tell me what is the expected output? What is the html structure
>> that is expected?
>>
>> I think I can find a solution to produce an html page through apache index
>> listing and rewrite rules to catch URLs like
>> http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main.
>>
>> Thanks,
>> Christophe.
>>
>>> Le 27 août 2020 à 23:50, Dale Henrichs a
>>> écrit :
>>>
>>> That makes sense and confirms that the static site has a bug ... portions
>>> of my work are on hold until the SmalltalkHub issue is resolved and at
>>> least one other GemStone user is impacted, so far...
>>>
>>> Dale
>>>
>>> On 8/27/20 2:41 PM, Sven Van Caekenberghe wrote:
>>>> The static version of StHub seems to always assume the client is Pharo,
>>>> while the dynamic version used format=raw (which non-Pharo implementation
>>>> did not include in their request for the listing) to send the Pharo
>>>> specific response only then.
>>>>
>>>>> On 27 Aug 2020, at 23:34, Dale Henrichs
>>>>> wrote:
>>>>>
>>>>> Depends upon how old that optimized code is ... as little as a 15 days
>>>>> ago, the last time my travis cron job ran successfully[1], the pharo code
>>>>> presumably was handling html page returns ... I'm pretty certain I
>>>>> haven't touched the Monticello HTTP handling code for nearly a decade:)
>>>>>
>>>>> Dale
>>>>>
>>>>> [1] https://travis-ci.org/github/GsDevKit/GsDevKit_home/builds/717364651
>>>>>
>>>>> On 8/27/20 1:44 PM, Sven Van Caekenberghe wrote:
>>>>>> Hmm, this is going to be a hard one.
>>>>>>
>>>>>> SmalltalkHub got optimised in Pharo, consider
>>>>>>
>>>>>> MCHttpRepository>>#parseFileNamesFromStream: aStream
>>>>>> | names fullName |
>>>>>> names := OrderedCollection new.
>>>>>> [aStream atEnd] whileFalse:
>>>>>> [[aStream upTo: $<. {$a. $A. nil} includes: aStream next]
>>>>>> whileFalse.
>>>>>> aStream upTo: $".
>>>>>> aStream atEnd ifFalse: [
>>>>>> fullName := aStream upTo: $".
>>>>>> names add: fullName urlDecoded ]].
>>>>>> ^ names
>>>>>>
>>>>>> vs.
>>>>>>
>>>>>> MCSmalltalkHubRepository>>#parseFileNamesFromStream:
>>>>>> aNewLineDelimitedString
>>>>>> ^ aNewLineDelimitedString
>>>>>> ifNil: [ ^ OrderedCollection new ]
>>>>>> ifNotNil: [ aNewLineDelimitedString substrings: String crlf ]
>>>>>>
>>>>>> In the old server code there was probably a way to detect what kind of
>>>>>> client was making the request to determine how to respond. I am not sure
>>>>>> a static server can do that (it is the format=raw query parameter, see
>>>>>> MCSmalltalkHubRepository>>#loadAllFileNames). I also believe GZIP
>>>>>> compressed files were returned in the optimised case.
>>>>>>
>>>>>> BTW, there exists code to generate the listing in
>>>>>>
>>>>>> ZnMonticelloRepository>>#repositoryListing
>>>>>> ^ ZnHtmlOutputStream streamContents:
Re: [Pharo-dev] 'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com
The static version of StHub seems to always assume the client is Pharo, while
the dynamic version used format=raw (which non-Pharo implementation did not
include in their request for the listing) to send the Pharo specific response
only then.
> On 27 Aug 2020, at 23:34, Dale Henrichs
> wrote:
>
> Depends upon how old that optimized code is ... as little as a 15 days ago,
> the last time my travis cron job ran successfully[1], the pharo code
> presumably was handling html page returns ... I'm pretty certain I haven't
> touched the Monticello HTTP handling code for nearly a decade:)
>
> Dale
>
> [1] https://travis-ci.org/github/GsDevKit/GsDevKit_home/builds/717364651
>
> On 8/27/20 1:44 PM, Sven Van Caekenberghe wrote:
>> Hmm, this is going to be a hard one.
>>
>> SmalltalkHub got optimised in Pharo, consider
>>
>> MCHttpRepository>>#parseFileNamesFromStream: aStream
>> | names fullName |
>> names := OrderedCollection new.
>> [aStream atEnd] whileFalse:
>> [[aStream upTo: $<. {$a. $A. nil} includes: aStream next]
>> whileFalse.
>> aStream upTo: $".
>> aStream atEnd ifFalse: [
>> fullName := aStream upTo: $".
>> names add: fullName urlDecoded ]].
>> ^ names
>>
>> vs.
>>
>> MCSmalltalkHubRepository>>#parseFileNamesFromStream: aNewLineDelimitedString
>> ^ aNewLineDelimitedString
>> ifNil: [ ^ OrderedCollection new ]
>> ifNotNil: [ aNewLineDelimitedString substrings: String crlf ]
>>
>> In the old server code there was probably a way to detect what kind of
>> client was making the request to determine how to respond. I am not sure a
>> static server can do that (it is the format=raw query parameter, see
>> MCSmalltalkHubRepository>>#loadAllFileNames). I also believe GZIP compressed
>> files were returned in the optimised case.
>>
>> BTW, there exists code to generate the listing in
>>
>> ZnMonticelloRepository>>#repositoryListing
>> ^ ZnHtmlOutputStream streamContents: [ :html |
>> html page: 'Monticello Repository' do: [
>> html tag: #ul do: [
>> self mczEntries do: [ :each |
>> html tag: #li do: [
>> html
>> tag: #a
>> attributes: { #href.
>> each }
>> with: each ] ] ] ] ]
>>
>> Sven
>>
>>> On 27 Aug 2020, at 22:29, Dale Henrichs
>>> wrote:
>>>
>>> My guess is lies in the difference in the payload returned.
>>>
>>> http://www.squeaksource.com/MooseSQL/ produces a html page:
>>>
>>>
>>>
>>> and the static smalltalkhub site does not:
>>>
>>>
>>>
>>> I think that all of the monticello web sites return an html web page
>>> listing of packages and presumably the static site should produce html ...
>>> I'm sure that the dynamic version of smalltalkhub produced html pages as
>>> well and for now we are caught between a rock and a hard place ... the
>>> dynamic site is flakey and the static site breaks existing Monticello
>>> package list reading code:)
>>>
>>> Dale
>>>
>>> On 8/27/20 1:04 PM, Dale Henrichs wrote:
>>>> As I've started digging around, I have found that this url[1] does produce
>>>> the correct list of mcz files in the browser, but is currently failing to
>>>> produce any list at all in GLASS ... so there is a different mystery ...
>>>> other than the fact that this url[1] was working prior(?) to the
>>>> switchover (if in fact the DNS has propagated to all the right spots) and
>>>> has been working for all of the other http Monticello repositories for
>>>> over a decade:)
>>>>
>>>> I will continue digging ...
>>>>
>>>> Dale
>>>>
>>>> [1] http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main
>>>>
>>>> On 8/27/20 12:48 PM, Dale Henrichs wrote:
>>>>> Christophe,
>>>>>
>>>>> There is a new(?) problem that we are having that has been reported in
>>>>> this thread on the GLASS list[1] where I am able to successfully download
>>>
Re: [Pharo-dev] 'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com
Hmm, this is going to be a hard one.
SmalltalkHub got optimised in Pharo, consider
MCHttpRepository>>#parseFileNamesFromStream: aStream
| names fullName |
names := OrderedCollection new.
[aStream atEnd] whileFalse:
[[aStream upTo: $<. {$a. $A. nil} includes: aStream next]
whileFalse.
aStream upTo: $".
aStream atEnd ifFalse: [
fullName := aStream upTo: $".
names add: fullName urlDecoded ]].
^ names
vs.
MCSmalltalkHubRepository>>#parseFileNamesFromStream: aNewLineDelimitedString
^ aNewLineDelimitedString
ifNil: [ ^ OrderedCollection new ]
ifNotNil: [ aNewLineDelimitedString substrings: String crlf ]
In the old server code there was probably a way to detect what kind of client
was making the request to determine how to respond. I am not sure a static
server can do that (it is the format=raw query parameter, see
MCSmalltalkHubRepository>>#loadAllFileNames). I also believe GZIP compressed
files were returned in the optimised case.
BTW, there exists code to generate the listing in
ZnMonticelloRepository>>#repositoryListing
^ ZnHtmlOutputStream streamContents: [ :html |
html page: 'Monticello Repository' do: [
html tag: #ul do: [
self mczEntries do: [ :each |
html tag: #li do: [
html
tag: #a
attributes: { #href.
each }
with: each ] ] ] ] ]
Sven
> On 27 Aug 2020, at 22:29, Dale Henrichs
> wrote:
>
> My guess is lies in the difference in the payload returned.
>
> http://www.squeaksource.com/MooseSQL/ produces a html page:
>
>
>
> and the static smalltalkhub site does not:
>
>
>
> I think that all of the monticello web sites return an html web page listing
> of packages and presumably the static site should produce html ... I'm sure
> that the dynamic version of smalltalkhub produced html pages as well and for
> now we are caught between a rock and a hard place ... the dynamic site is
> flakey and the static site breaks existing Monticello package list reading
> code:)
>
> Dale
>
> On 8/27/20 1:04 PM, Dale Henrichs wrote:
>> As I've started digging around, I have found that this url[1] does produce
>> the correct list of mcz files in the browser, but is currently failing to
>> produce any list at all in GLASS ... so there is a different mystery ...
>> other than the fact that this url[1] was working prior(?) to the switchover
>> (if in fact the DNS has propagated to all the right spots) and has been
>> working for all of the other http Monticello repositories for over a decade:)
>>
>> I will continue digging ...
>>
>> Dale
>>
>> [1] http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main
>>
>> On 8/27/20 12:48 PM, Dale Henrichs wrote:
>>> Christophe,
>>>
>>> There is a new(?) problem that we are having that has been reported in this
>>> thread on the GLASS list[1] where I am able to successfully download an mcz
>>> file [2], but get a `Not Found` error when I try to list the mcz files in a
>>> project[3]. The missing mcz list is consistent with the failed builds that
>>> I am now seeing on travis [4] and that are being reported by Brodbeck[1]. I
>>> have yet to get to a point where I can debug the problems directly and
>>> determine what is actually going on and of course I can't tell if these are
>>> the results of slow DNS propagation.
>>>
>>> In this case [2][3], the list of file shows up on the dynamic(?) site:
>>>
>>>
>>>
>>> and can be downloaded by pressing the download for the selected mcz file,
>>> but the missing list of packages[3] is likely to be the root cause of the
>>> problem.
>>>
>>> Dale
>>>
>>> [1]
>>> http://forum.world.st/SmalltalkHub-packages-not-accessible-tt5120932.html
>>> [2]
>>> http://smalltalkhub.com/mc/Seaside/Seaside30LGPL/main/Seaside-Swazoo-jf.19.mcz
>>> [3] http://smalltalkhub.com/mc/Seaside/Seaside30LGPL
>>> [4] https://travis-ci.org/github/GsDevKit/GsDevKit_home/jobs/721523221#L2411
>>>
>>> On 8/27/20 5:36 AM, Christophe Demarey wrote:
Hi Dale,
Sorry, I did not see your message before.
Yesterday, I switched smalltalkhub to the static version (a bit earlier
than announced) to avoid frequent downtimes we had with smalltalkhub.
I did not measure but downloads should now be faster and reliable.
Do not hesitate to ping if you have any problem.
Cheers,
Christophe
> Le 26 août 2020 à 18:12, Dale Henrichs
> a écrit :
>
> Well, I haven't see any email response, but today (after two days of
> brokenness),
>
Re: [Pharo-dev] 'VOMongoConnectionError' when dowloading mcz from smalltalkhub.com
> On 27 Aug 2020, at 14:36, Christophe Demarey > wrote: > > Hi Dale, > > Sorry, I did not see your message before. > Yesterday, I switched smalltalkhub to the static version (a bit earlier than > announced) to avoid frequent downtimes we had with smalltalkhub. Are you sure ? For me, http://smalltalkhub.com is different from http://static.smalltalkhub.com And the download links http://www.smalltalkhub.com/mc/SvenVanCaekenberghe/ZincHTTPComponents/main/ConfigurationOfZincHTTPComponents-SvenVanCaekenberghe.120.mcz and http://static.smalltalkhub.com/SvenVanCaekenberghe/ZincHTTPComponents/mc/ConfigurationOfZincHTTPComponents-SvenVanCaekenberghe.120.mcz are coming from a different server. > I did not measure but downloads should now be faster and reliable. > > Do not hesitate to ping if you have any problem. > > Cheers, > Christophe > >> Le 26 août 2020 à 18:12, Dale Henrichs a >> écrit : >> >> Well, I haven't see any email response, but today (after two days of >> brokenness), >> http://smalltalkhub.com/mc/dkh/metacello/main/Metacello-Base-dkh.109.mcz is >> now downloading successfully, so THANK YOU, to whoever fixed the problem! >> >> Dale >> >> On 8/25/20 9:02 AM, Dale Henrichs wrote: >>> SmalltalkHub mcz downloads are broken ... looks like a mongo server has >>> gone down? I ran into this problem running production tests yesterday >>> and today I find that while the smalltalkhub site is up, I cannot download >>> an mcz file, using this url: >>> http://smalltalkhub.com/mc/dkh/metacello/main/Metacello-Base-dkh.109.mcz. >>> >>> If you are not going to keep the current smalltalkhub site functional, why >>> don't you switch to the static site and give those of us who DEPEND upon >>> static access to mcz files a reliable site to connect to ... I have plans >>> to move completely away from mcz files, but I didn't plan on doing that >>> this week ... and frankly I don't have the cycles to do that ... right now >>> ... >>> >>> Here's a screenshot of a manual login and navigation to the mcz file that >>> is failing to download: >>> >>> >>> >>> And when I press the `Download .mcz` button, I get the following "response" >>> after a delay: >>> >>> >>> >
Re: [Pharo-dev] Microdown ?
Thank you, Pierre and Norbert. > On 7 Jul 2020, at 14:32, Norbert Hartl wrote: > > Microdown is a subset of markdown that can parse to a pillar model. So the > discussions about pillar markup have finally settled and we have a syntax > format that reads well in ascii which can be used in class comments even if > it is a basic image. > To integrate pillar core in pharo is not new just the change in syntax. > >> Am 07.07.2020 um 11:27 schrieb Sven Van Caekenberghe : >> >> Hi, >> >> What is microdown ? >> What are its goals and objectives ? >> Where does it come from ? >> Is there any documentation about it ? >> >> There probably is a background story here, and it is probably a good idea, >> but unless I missed something, for me reading the mailing lists daily, it >> came from nowhere and got immediately integrated into Pharo 9. >> >> Sven >> >> > >
[Pharo-dev] Microdown ?
Hi, What is microdown ? What are its goals and objectives ? Where does it come from ? Is there any documentation about it ? There probably is a background story here, and it is probably a good idea, but unless I missed something, for me reading the mailing lists daily, it came from nowhere and got immediately integrated into Pharo 9. Sven
[Pharo-dev] Zinc HTTP Components, P3, MQTT and NeoUniversalBinaryJSON now require Pharo 7.0.5 as minimum version
Hi, Here is a little heads up: in order to support Pharo 9, I made a change in a couple of my libraries/frameworks (using Integer #byteAt: instead of #digitAt:) that requires Pharo 7.0.5 (released recently) as minimum version. https://github.com/svenvc/zinc https://github.com/svenvc/P3 https://github.com/svenvc/mqtt https://github.com/svenvc/NeoUniversalBinaryJSON If you are unable or unwilling to upgrade, you can patch this by adding the follow alias to Pharo versions < 7.0.5: Integer>>#byteAt: index ^ self digitAt: index Sven
Re: [Pharo-dev] Pharo 7.0.5
Thanks again, Esteban, it works well ! Travis CI picked up the latest version as well. I got one project using Integer>>#byteAt: now, https://github.com/svenvc/NeoUniversalBinaryJSON , that cleanly builds on Pharo 7, 8 and 9 using one code base: https://travis-ci.org/github/svenvc/NeoUniversalBinaryJSON Cool. I'll be moving my other projects using that method (MQTT, P3 and Zinc) in the coming days. Sven > On 29 Jun 2020, at 11:07, Esteban Lorenzano wrote: > > yes > > On lun., juin 29, 2020 at 11:05, Sven Van Caekenberghe wrote: >> And will I get that version using >> >> $ curl get.pharo.org/64/70 | bash >> >> and will that version be available in the Pharo Launcher as well ? >> >> >> On 29 Jun 2020, at 10:51, Esteban Lorenzano > > wrote: >> >> ok, in theory... you have Pharo 7.0.5 now, as final version. >> >> Esteban >> >> On lun., juin 29, 2020 at 09:11, Sven Van Caekenberghe < >> s...@stfx.eu >> > wrote: >> >> OK, thanks. >> >> >> On 29 Jun 2020, at 06:58, Esteban Lorenzano < >> esteba...@netc.eu >> >> > wrote: >> >> Hi Sven, >> >> In general, I would say Pharo 7 is not maintained anymore, but in this >> case I think we should have done this as last task before closing it. >> I will try to do a final release now (we'll see how it works, it should be >> an easy task but two years happened since P7 so... we will not know until we >> try ;) ) >> >> Esteban >> >> On dim., juin 28, 2020 at 20:32, Sven Van Caekenberghe < >> >> s...@stfx.eu >> >> > wrote: >> >> PING: I still did not yet get any answer. >> >> >>On 17 May 2020, at 12:27, Sven Van Caekenberghe < >> >> s...@stfx.eu >> >> >> > wrote: >> >>Hi, >> >>Is there any intention of officially releasing Pharo 7.0.5 ? >> >>There are 18 commits after 7.0.4 >> >> >> >> >> https://github.com/pharo-project/pharo/compare/v7.0.4...Pharo7.0 >> >> >> >> >>I need the following >> >> >> >> >> https://github.com/pharo-project/pharo/pull/5079 >> >> >> >> >>which is one of the 18 commits >> >>Zinc HTTP Components is currently partially broken on Pharo 9 because >> #digitAt: was replaced by #byteAt: and #digitAt: is now removed. >> >>In order to fix this I could start using #byteAt: as far back as 7.0.5 >> and add it myself for even older Pharo versions using my >> Zinc-Pharo-Forward-Compatibility package. >> >>What has to happen to make it possible to release 7.0.5 ? More testing ? >> Where is the artefact to test ? >> >>Thx, >> >>Sven >> >> >> >> >> >> >> >> >> >> >> >> >>
Re: [Pharo-dev] Pharo 7.0.5
And will I get that version using $ curl get.pharo.org/64/70 | bash and will that version be available in the Pharo Launcher as well ? > On 29 Jun 2020, at 10:51, Esteban Lorenzano wrote: > > ok, in theory... you have Pharo 7.0.5 now, as final version. > > Esteban > > On lun., juin 29, 2020 at 09:11, Sven Van Caekenberghe wrote: >> OK, thanks. >> >> >> On 29 Jun 2020, at 06:58, Esteban Lorenzano > > wrote: >> >> Hi Sven, >> >> In general, I would say Pharo 7 is not maintained anymore, but in this case >> I think we should have done this as last task before closing it. >> I will try to do a final release now (we'll see how it works, it should be >> an easy task but two years happened since P7 so... we will not know until we >> try ;) ) >> >> Esteban >> >> On dim., juin 28, 2020 at 20:32, Sven Van Caekenberghe < >> s...@stfx.eu >> > wrote: >> >> PING: I still did not yet get any answer. >> >> >> On 17 May 2020, at 12:27, Sven Van Caekenberghe < >> s...@stfx.eu >> >> > wrote: >> >> Hi, >> >> Is there any intention of officially releasing Pharo 7.0.5 ? >> >> There are 18 commits after 7.0.4 >> >> >> >> https://github.com/pharo-project/pharo/compare/v7.0.4...Pharo7.0 >> >> >> >> I need the following >> >> >> >> https://github.com/pharo-project/pharo/pull/5079 >> >> >> >> which is one of the 18 commits >> >> Zinc HTTP Components is currently partially broken on Pharo 9 because >> #digitAt: was replaced by #byteAt: and #digitAt: is now removed. >> >> In order to fix this I could start using #byteAt: as far back as 7.0.5 and >> add it myself for even older Pharo versions using my >> Zinc-Pharo-Forward-Compatibility package. >> >> What has to happen to make it possible to release 7.0.5 ? More testing ? >> Where is the artefact to test ? >> >> Thx, >> >> Sven >> >> >> >> >> >> >> >> >>
Re: [Pharo-dev] Pharo 7.0.5
OK, thanks. > On 29 Jun 2020, at 06:58, Esteban Lorenzano wrote: > > Hi Sven, > > In general, I would say Pharo 7 is not maintained anymore, but in this case I > think we should have done this as last task before closing it. > I will try to do a final release now (we'll see how it works, it should be an > easy task but two years happened since P7 so... we will not know until we try > ;) ) > > Esteban > > On dim., juin 28, 2020 at 20:32, Sven Van Caekenberghe wrote: >> PING: I still did not yet get any answer. >> >> >> On 17 May 2020, at 12:27, Sven Van Caekenberghe > > wrote: >> >> Hi, >> >> Is there any intention of officially releasing Pharo 7.0.5 ? >> >> There are 18 commits after 7.0.4 >> >> >> https://github.com/pharo-project/pharo/compare/v7.0.4...Pharo7.0 >> >> >> I need the following >> >> >> https://github.com/pharo-project/pharo/pull/5079 >> >> >> which is one of the 18 commits >> >> Zinc HTTP Components is currently partially broken on Pharo 9 because >> #digitAt: was replaced by #byteAt: and #digitAt: is now removed. >> >> In order to fix this I could start using #byteAt: as far back as 7.0.5 and >> add it myself for even older Pharo versions using my >> Zinc-Pharo-Forward-Compatibility package. >> >> What has to happen to make it possible to release 7.0.5 ? More testing ? >> Where is the artefact to test ? >> >> Thx, >> >> Sven >> >> >> >> >>
Re: [Pharo-dev] Pharo 7.0.5
PING: I still did not yet get any answer. > On 17 May 2020, at 12:27, Sven Van Caekenberghe wrote: > > Hi, > > Is there any intention of officially releasing Pharo 7.0.5 ? > > There are 18 commits after 7.0.4 > > https://github.com/pharo-project/pharo/compare/v7.0.4...Pharo7.0 > > I need the following > > https://github.com/pharo-project/pharo/pull/5079 > > which is one of the 18 commits > > Zinc HTTP Components is currently partially broken on Pharo 9 because > #digitAt: was replaced by #byteAt: and #digitAt: is now removed. > > In order to fix this I could start using #byteAt: as far back as 7.0.5 and > add it myself for even older Pharo versions using my > Zinc-Pharo-Forward-Compatibility package. > > What has to happen to make it possible to release 7.0.5 ? More testing ? > Where is the artefact to test ? > > Thx, > > Sven >
Re: [Pharo-dev] [Pharo-users] [ANN] PharoPro
Hi Norbert, I would say 'finally'. This is great news and a good initiative. This will certainly help move Pharo forward. I wish you all the best. Sven > On 17 Jun 2020, at 10:34, Norbert Hartl wrote: > > Dear community, > > we are very proud to announce the availability of PharoPro, a company that > offers professional support for the pharo platform. Our mission is to enable > people and companies to secure their business when building products with > pharo. > > We have been tinkering with the idea creating it for a long time. We talked > to a lot of people, especially at last ESUG in cologne and finally decided to > do it. So at the end of last year we founded the company PharoPro. Obstacles > (one being covid-19) kept us from make it public until today. > > In close relationship with the pharo consortium our mission is to extend the > market opportunities for pharo. > Although PharoPro is new and still small we want to show our commitment to > the consortium and the community from the start. We decided therefor that > PharoPro will join the pharo consortium as platinum member. > > Our plans apart from support contracts is to have an LTS (long term support) > version of pharo, create a professional pharo ecosystem with libraries and > frameworks and to help out building the infrastructure pharo needs. > > We are still shaping up. We want to know your requirements and needs that > help us deciding what are the most needed things. So you might visit > > http://pharo-pro.com > > and then come talk to us. > > Best regards and health, > > Norbert > PharoPro GmbH -- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
Re: [Pharo-dev] question about Zinc ZnUrl or ZnResourceMetaUtils
https://github.com/svenvc/zinc/commit/7a42881f14144e0d9dcc2b3aa98a075f189eae69 > On 4 Jun 2020, at 23:06, Sven Van Caekenberghe wrote: > > Theoretically, the order is not important (should not matter), but for > practical purposes (like being able to parse a URL and print it to the same > representation) it would be a good idea. > > I'll see if I can add it. > >> On 28 May 2020, at 11:24, Stéphane Ducasse wrote: >> >> Hi sven >> >> Are the dictionaries for the query part conserving the order? >> Because this would be really nice. >> >> S. >> >> Stéphane Ducasse >> http://stephane.ducasse.free.fr / http://www.pharo.org >> 03 59 35 87 52 >> Assistant: Aurore Dalle >> FAX 03 59 57 78 50 >> TEL 03 59 35 86 16 >> S. Ducasse - Inria >> 40, avenue Halley, >> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza >> Villeneuve d'Ascq 59650 >> France >> >
Re: [Pharo-dev] question about Zinc ZnUrl or ZnResourceMetaUtils
Theoretically, the order is not important (should not matter), but for practical purposes (like being able to parse a URL and print it to the same representation) it would be a good idea. I'll see if I can add it. > On 28 May 2020, at 11:24, Stéphane Ducasse wrote: > > Hi sven > > Are the dictionaries for the query part conserving the order? > Because this would be really nice. > > S. > > Stéphane Ducasse > http://stephane.ducasse.free.fr / http://www.pharo.org > 03 59 35 87 52 > Assistant: Aurore Dalle > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley, > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France >
Re: [Pharo-dev] [Pharo-users] About strange email related to smalltalkhub read-only on squeak-dev
> On 1 Jun 2020, at 06:39, Jerry Kott wrote: > > If you read the text of the EAR and take into account all other facts, I > think that the notion that anyone should get into trouble by copying open > source Smalltalk crypto libraries to other repositories is just a pure FUD. > This software is open source, it is publicly available including the source > code, it is hosted on a domain that is controlled by a non-US entity, and > it’s easily accessible in its current form from countries that are currently > on the US ‘vorboten’ list. Indeed.
Re: [Pharo-dev] would it make sense to have ZnCharacterReadStream and other readstream reply to readstream
https://github.com/svenvc/zinc/commit/7a4b4bda502092186d03b467dc6baf9d1a76baed > On 13 May 2020, at 17:49, Sven Van Caekenberghe wrote: > > Yes, I know the pattern and yes it would make sense. I will study this a bit. > >> On 13 May 2020, at 17:21, Stéphane Ducasse wrote: >> >> Hi sven >> >> I like the pattern >> >> >> on: aStringOrStream >> >> | stream | >> stream := aStringOrStream readStream >> >> but I cannot apply with Zn because ZnCharacterReadStream and friends do not >> answer readstream >> >> What is your point of view on it. >> >> >> S. >> >> Stéphane Ducasse >> http://stephane.ducasse.free.fr / http://www.pharo.org >> 03 59 35 87 52 >> Assistant: Julie Jonas >> FAX 03 59 57 78 50 >> TEL 03 59 35 86 16 >> S. Ducasse - Inria >> 40, avenue Halley, >> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza >> Villeneuve d'Ascq 59650 >> France >> >
[Pharo-dev] pharo-contributions/pharo-unicode
Hi, I am not 100% sure how things are organised, but I would like access to pharo-contributions/pharo-unicode, so that I can commit directly. I see that Alistair did the initial commits (copying from history), but I can't see how he could have done that. I access granted at the pharo-contributions level, or below ? Thx, Sven
[Pharo-dev] Pharo 7.0.5
Hi, Is there any intention of officially releasing Pharo 7.0.5 ? There are 18 commits after 7.0.4 https://github.com/pharo-project/pharo/compare/v7.0.4...Pharo7.0 I need the following https://github.com/pharo-project/pharo/pull/5079 which is one of the 18 commits Zinc HTTP Components is currently partially broken on Pharo 9 because #digitAt: was replaced by #byteAt: and #digitAt: is now removed. In order to fix this I could start using #byteAt: as far back as 7.0.5 and add it myself for even older Pharo versions using my Zinc-Pharo-Forward-Compatibility package. What has to happen to make it possible to release 7.0.5 ? More testing ? Where is the artefact to test ? Thx, Sven
Re: [Pharo-dev] would it make sense to have ZnCharacterReadStream and other readstream reply to readstream
Yes, I know the pattern and yes it would make sense. I will study this a bit. > On 13 May 2020, at 17:21, Stéphane Ducasse wrote: > > Hi sven > > I like the pattern > > > on: aStringOrStream > > | stream | > stream := aStringOrStream readStream > > but I cannot apply with Zn because ZnCharacterReadStream and friends do not > answer readstream > > What is your point of view on it. > > > S. > > Stéphane Ducasse > http://stephane.ducasse.free.fr / http://www.pharo.org > 03 59 35 87 52 > Assistant: Julie Jonas > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley, > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France >
Re: [Pharo-dev] why GT forked the VM and how to kill our fork
Hi Doru, > On 26 Apr 2020, at 09:57, Tudor Girba wrote: > > Hi, > > Currently, at feenk we have feenkcom/opensmalltalk-vm: > https://github.com/feenkcom/opensmalltalk-vm > > This is a small fork of the headless branch from > pharo-project/opensmalltalk-vm that appeared out of practical necessities, > but that we would like to avoid having. This post briefly describes the > changes in the feenkcom/opensmalltalk-vm repo and the functionality those > changes provide for Glamorous Toolkit. > > For Glamorous Toolkit we aimed for the following functionality: > • Open the GUI natively and have native display quality (GUI opened > through FFI calls) > • Have a Glamorous Toolkit app for Mac OS that works as any other apps > for Mac OS > • Create end-user applications that are fully customisable (executable > name, menus, etc) > • Use Github actions for doing all compilations of external libraries > and the vm instead of Travis CI. > • Have Iceberg running in native windows (which requires nested FFI > callbacks) > > There has been work on these issues in both OpenSmalltalk/opensmalltalk-vm > and pharo-project/opensmalltalk-vm but they were not entirely addressed. We > needed to have something reliable a few months ago, and forking and doing > some quick changes made that possible. > > Ideally we want to be able to run Glamorous Toolkit on both > OpenSmalltalk/opensmalltalk-vm and pharo-project/opensmalltalk-vm. > > To have native GUIs we relied on Ronie Salgado’s work on the headless vm and > started with pharo-project/opensmalltalk-vm - headless branch: > https://github.com/pharo-project/opensmalltalk-vm/tree/headless > That provided a solution for opening the GUI from the image through FFI > calls. Currently we use Glutin (a library for OpenGL context creation, > written in pure Rust) and this made it possible to run the entire Glamorous > Toolkit inside a callback. > > On macOS when running an app, even a notarized one, the OS warns the user > that the app is downloaded from the internet, and the user needs to confirm > that they agree. Once the user agrees the app should automatically start. > This is not currently possible with Pharo apps (for example > PharoLaunched.app) and users have to again run the app manually after giving > permission. Also Gatekeeper in macOS runs applications downloaded from zips > in a randomized read-only DMG. We do not want this behaviour as users not > copying Glamorous Toolkit to the Applications folder on macOS would then > experience incorrect application behaviour. > > To create end-user applications we also need to fully customize the > executable name (what the user sees in the Task Runner/Activity monitor), > icons, native menus. Part of this work is already integrated in the > pharo-project/opensmalltalk-vm - headless branch (Customizing the OS X icons, > Brand the VM executable and package). > > Since last year Github offers Github Actions similar to Travis. We found it > much easier to use than Travis for external libraries and the vm. Also we get > to manage the code and the builds in the same place. This work is already > integrated in the pharo-project/opensmalltalk-vm - headless branch (Build the > VM under GitHub actions: > https://github.com/pharo-project/opensmalltalk-vm/pull/56). > > The issues related to running Iceberg is a bit more technical. By moving to > the headless vm we are running the entire image computation inside a callback > from Glutin (https://github.com/rust-windowing/glutin/). When using Iceberg > we get nested callbacks which we could not get to work using Alien. Instead > we are using the ThreadedFFI Plugin and running all callback from Iceberg and > Glutin using the Threaded FFI plugin > (https://github.com/pharo-project/threadedFFI-Plugin). Currently we have a > small fork of this plugin (feenkcom/threadedFFI-Plugin) and we also ship a > custom plugin with the VM to fix a race condition due to having two copies of > the callback stack (a pull request is here: > https://github.com/pharo-project/threadedFFI-Plugin/pull/17). > > While not specific to our environment, openssl1.0 is no longer supported, and > we are seeing users who are unable to run Pharo due to version conflicts, as > reported in https://github.com/pharo-project/opensmalltalk-vm/issues/62. Thanks for the detailed writeup. > To sum up, a fork was the easiest way to get all this running. Now some > changes are already in the pharo-project/opensmalltalk-vm - headless branch. > What we are still missing are the changes that get the VM to work nicely with > Mac OS and a bug fix in ThreadedFFI. I would also love to have *all* your changes back into pharo-project/opensmalltalk-vm ! Sven > We would also love it to have all these changes integrated in > OpenSmalltalk/opensmalltalk-vm in the headless vm. This requires additional > coordination as the required changes are somewhat
Re: [Pharo-dev] Programmatically change display size in Pharo 8 / 9
Hi Max, Not an answer, but do you really need VNC to deploy on a server ? This is often unreliably and a bit overkill, it is also alien to the rest of the server world. Provided you can build your project automatically from a single Metacello invocation, you can build server side, command line as well. I use a simple command line REPL for those harder problems that require interacting with the image. https://github.com/svenvc/NeoConsole You can type arbitrary expressions, look at code and even patch a method. Yes, it is all primitive, but it can make a night and day difference if you don't understand what is happening in real time. I have never felt a need to use VNC for a full interactive IDE on a server. But maybe you have other reasons to go in that direction. Sven > On 24 Apr 2020, at 10:51, Max Leske wrote: > > Hi all, > > I'm trying to change the size of the display programmatically but am having > some trouble. It used to be possible to do Display setExtent: 50@50 depth: 16 > and that would take care of everything. I can still do that but 1. the > renderer reverts the changes immediately and 2. if I prevent the renderer > from reverting the changes the rendering is broken (which I can understand). > > So: how am I supposed to change the display size? Note: this is for the > remote frame buffer in headless mode, so technically there's no OS window. > Still the VM returns a fixed size of something close to 900x700. > > I can live with the fixed size if I have to but I'd rather be able to change > the size (and also the bit depth as I don't need 32 bits for VNC). > > Cheers, > Max >
Re: [Pharo-dev] FW: [Pharo-users] Label truncation in new Pharo 8.0 64-bit (stable)
Did you try another font ? Another size ? Did you try closing & reopening the Settings Browser ? > On 19 Apr 2020, at 16:31, shap...@uurda.org wrote: > > On 2020-04-19 03:01, Sven Van Caekenberghe wrote: > Works for me (this font is larger than the default one). macOS, Pharo > 9 64bit >> The screenshot below is taken on Windows 10 Pro. This time I did not change >> the theme. I did also update the fonts listed in the dialog before choosing >> FiraCode. The labels are still truncated. This is another new Pharo 8 >> 64-bit (stable) with only the font changed. Why would formatting not work >> on Windows 10? >> Can any Windows 10 Pharo users verify this problem? >> <07e94bb1.png> >> >> >>> On 19 Apr 2020, at 06:05, Shaping wrote: >>> >>> Can anyone suggest a fix for the truncated labels shown below in the >>> screenshot of the Setting window? This happens in the Git client >>> too. >>> >>> Will Pharo tools be rewritten in Spec2, and does its formatting >>> scheme solve generally the text-formatting problem, as font size is >>> changed? >>> >>> Why don’t we opt for easy elastic rendering of GUIs by >>> implementing Pharo as a web app? Has this been discussed? Willow >>> is looking very good. >>> >>> Shaping >>> From: Pharo-users [mailto:pharo-users-boun...@lists.pharo.org] On >>> Behalf Of Shaping >>> Sent: Saturday, 18 April, 2020 04:16 >>> To: 'Any question about pharo is welcome' >>> ; 'Pharo Development List' >>> >>> Subject: [Pharo-users] Label truncation in new Pharo 8.0 64-bit >>> (stable) >>> >>> Hi everyone. >>> >>> Below is a Pharo 8.0 64-bit (stable) image (installed with the new >>> Launcher 2.0) with two changes: >>> >>> -Light theme selected. >>> -FiraCode 11 chosen as the default font. >>> >>> >>> >>> Truncated labels in the GUI on font change has always been a >>> problem. Am I the only one experiencing this? >>> >>> Shaping >>> From: Pharo-users [mailto:pharo-users-boun...@lists.pharo.org] On >>> Behalf Of Sanjay Minni >>> Sent: Friday, 17 April, 2020 11:05 >>> To: Any question about pharo is welcome >>> >>> Subject: Re: [Pharo-users] issues in setting up a new project on >>> Github/Iceberg >>> >>> Thanks tried just https and this worked too >>> I suppose once cloned it does not really matter whether it was done >>> using "clone from github.com" or "clone remote repository" >>> --- >>> Sanjay Minni >>> +91-9900-902902 >>> >>> On Fri, 17 Apr 2020 at 19:43, Guillermo Polito >>> wrote: >>> Hi, >>> >>> El 17 abr 2020, a las 15:30, Sanjay Minni >>> escribió: >>> >>> Hi Guillermo, >>> >>> Ok then given your comments - right now got it working as such >>> - initially created as private in Github >>> - changed to public >>> - created clone from github >>> - (did a repair repository (dont know why) and created "src" and >>> designated it as source folder) >>> - did a commit and push >>> - changed repository back to private >>> seems to work >>> >>> can you give an example on how the "Remote URL" field is to be >>> filled up - note its not taking HTTP as i am not sure about the >>> syntax, assume SSH, Github, MyName, MyRepository, >>> will try that as well >>> >>> you should be able to use the URLs provided by the clone buttons in >>> Github/gitlab/bitbucket, both https and ssh. >>> >>> >>> >>> note Just for Info to other users: >>> I am using Windows 10, it seems to have SSH tools inbuilt and was >>> able to use ssh (with passphrase) without hiccup, by putting my key >>> files path in pharo setting-> ...-> use custom SSH keys, and >>> uploading the .pub key in Github >>> >>> :) >>> >>> Thanks >>> >>> --- >>> Sanjay Minni >>> +91-9900-902902 >>> >>> On Fri, 17 Apr 2020 at 18:36, Guillermo Polito >>> wrote: >>> Hi Sanjay, >>> >>> This is a bug in Iceberg github integration that tries to access the >>> repository to get metadata using Github REST API using anonymous >>> access apparently. >>> And since your projec
Re: [Pharo-dev] [Pharo-users] [ANN] Pharo Launcher 2.0 released!
> On 18 Apr 2020, at 10:40, Stéphane Ducasse wrote: > > Super cool > Thanks a lot christophe for your effort. > PharoLauncher changes our life. +10 > S. > >> On 17 Apr 2020, at 18:08, Christophe Demarey >> wrote: >> >> Hi all, >> >> Pharo Launcher 2.0 has just been released! It is available from >> http://pharo.org/download. >> >> This new version introduces major changes: >> • The UI has been fully rewritten using the new Spec2 framework and the >> Commander2 library. UI has been revamped to increase usability, especially >> for newcomers. The main window is now composed of a toolbar and the list of >> images. The template list is now available when clicking on the new image >> button. >> • Documentation web site : All Pharo Launcher features are now >> explained in the new documentation available at >> https://pharo-project.github.io/pharo-launcher. You can contribute easily by >> clicking the *edit on GitHub* button. >> • You can now have many launch configurations for an image (VM to use, >> vm and image arguments). It means you can use headless Pharo VM from Pharo >> Launcher. >> • When creating a new image, you can specify an initialisation script >> that will be run once at the first image launch. It is useful to load your >> project code in a stock Pharo image for example. See >> https://pharo-project.github.io/pharo-launcher/create-images/#image-initialization-script >> • You can now define your own template sources in addition to official >> sources (see >> https://pharo-project.github.io/pharo-launcher/templates/#create-your-own-list-of-template-categories), >> including authenticated sources. >> • Improved image metadata. Pharo Launcher now manages all image >> metadata in a single STON file (including description, Pharo version). >> >> Big thanks to all contributors, including issue reports. It is also the >> opportunity to thanks Damien Cassou, the original author of Pharo Launcher. >> >> Here is the changelog: >> Pharo Launcher v2.0 >> >> The list of closed issue is too long (68) to be listed but you can browse it >> here: >> https://github.com/pharo-project/pharo-launcher/issues?q=is%3Aissue+is%3Aclosed+closed%3A2019-07-09..2020-04-17 >> Here are some highlights: >> New features: >> >> • Documentation web site >> • Image initialisation script >> • Launch configurations, headless VM support >> • User template file in addition to the official template file >> • Jenkins server template now support pipeline projects >> • Support of private Jenkins server >> • Support of authenticated HTTP server >> Improvements: >> >> • Monitoring of image launch failures to give back the error message >> (if any) >> • Newly created image is automatically selected in the image list >> • Allow to set image description at creation time >> • Better error management (you will have the choice to ignore them or >> debug them) >> • Add a poor version column in image list >> • Speedup (especially when image repository has a lot of images) >> >> Bug fixes: >> >> • Fix use of system unzip on Windows >> >> Regards, >> The Pharo team. > > > Stéphane Ducasse > http://stephane.ducasse.free.fr / http://www.pharo.org > 03 59 35 87 52 > Assistant: Julie Jonas > FAX 03 59 57 78 50 > TEL 03 59 35 86 16 > S. Ducasse - Inria > 40, avenue Halley, > Parc Scientifique de la Haute Borne, Bât.A, Park Plaza > Villeneuve d'Ascq 59650 > France >
Re: [Pharo-dev] How to execute code
Sometimes things are hard to find. Not everything can be easy. It also depends on what you already know, what your field of expertise is, what your experiences are. What I meant is: doing a couple of searches for class or selector names is always the first step. I guess you are more a UI person and started by disassembling something. *That* is harder. > On 11 Apr 2020, at 15:18, Hilaire wrote: > > Once you know it is obvious :) > > Nevertheless, It took me about one hour to discover it, and only with Squeak > I can discover it. > > My observations regarding the complexity of Pharo remain valid, though. > > > Le 11/04/2020 à 13:05, Sven Van Caekenberghe a écrit : >> Hmmm ... >> >> Shift-Return (Spotter) type 'Compiler', first hit is AbstractCompiler which >> has a nice public API, it has only one concrete subclass that is being used: >> OpalCompiler >> >> Same for evaluate: > > -- > Dr. Geo > http://drgeo.eu > > >
Re: [Pharo-dev] How to execute code
Hmmm ... Shift-Return (Spotter) type 'Compiler', first hit is AbstractCompiler which has a nice public API, it has only one concrete subclass that is being used: OpalCompiler Same for evaluate: > On 11 Apr 2020, at 12:05, Hilaire wrote: > > Thanks to Squeak, it tooks me 5 min to discover this simple things from its > workspace menu: > > Compiler evaluate: ... > > Oh, it does not work on Pharo. Ah, I remember I read something about a new > compiler named Opal. I tried prefixing it with Opal. Yes, OpalCompiler > evaluate:... > > But why prefixing with Opal, it makes it impossible to discover, again ! > > Hum, I should have know how to do this simple thing really (and I did know > but forget), but for me it illustrates how complex Pharo is becoming, and so > more and more unpleasant to code with sometime. But I am clear I have no hope > it will ever change though. > > Hilaire > > Le 11/04/2020 à 11:37, Hilaire a écrit : >> >> How to write the code to execute a piece of code in code presenter? > > -- > Dr. Geo > http://drgeo.eu > > >
Re: [Pharo-dev] Cover for PBE8
Not bad at all, made me smile. > On 7 Apr 2020, at 12:21, Nicolas Anquetil wrote: > > > :-) > > > > > On Tue, 2020-04-07 at 09:45 +0200, Stéphane Ducasse wrote: >> Hello >> >> We are working on PBE80 and we are looking for a new cool cover. >> If you think that you can do this we are all ears. >> >> S >> >> Stéphane Ducasse >> http://stephane.ducasse.free.fr / http://www.pharo.org >> 03 59 35 87 52 >> Assistant: Julie Jonas >> FAX 03 59 57 78 50 >> TEL 03 59 35 86 16 >> S. Ducasse - Inria >> 40, avenue Halley, >> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza >> Villeneuve d'Ascq 59650 >> France >> > -- > Nicolas Anquetil > RMod team -- Inria Lille >
Re: [Pharo-dev] New VM promoted to Stable for Pharo 8
A side note, VM version strings are an absolute disaster for end users, it is simply impossible to talk about a specific version. $ ./pharo-ui --version 5.0 5.0.201901051900 Mac OS X built on Jan 5 2019 19:11:02 UTC Compiler: 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31) [Production Spur 64-bit VM] CoInterpreter VMMaker.oscog-eem.2504 uuid: a00b0fad-c04c-47a6-8a11-5dbff110ac11 Jan 5 2019 StackToRegisterMappingCogit VMMaker.oscog-eem.2504 uuid: a00b0fad-c04c-47a6-8a11-5dbff110ac11 Jan 5 2019 VM: 201901051900 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Sat Jan 5 20:00:11 2019 CommitHash: 7a3c6b6 Plugins: 201901051900 https://github.com/OpenSmalltalk/opensmalltalk-vm.git WTH ? > On 3 Mar 2020, at 19:07, tbrunz wrote: > > Pablo, does this only affect Macs? I just tried this on a Linux machine, > updating/launching a Pharo 8 64-bit image. > > It still showed the older VM. I tried restarting the VM Manager, restarting > Pharo Launcher (1.9.2), and then re-installing Pharo Launcher, then creating > a new image, each time trying everything again. > > But the version of the VM only ever says "2020-02-12", not "2020-02-20". > When I tried the same thing for a 32-bit Pharo 8 image, it uses a VM dated > "2020-02-06". > > What am I doing wrong? > > > > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >
[Pharo-dev] The expanding Pharo ecosystem
Hi, Let's take a moment to contemplate on the breath and the depth of the expanding Pharo ecosystem. So much is happening that it is very hard to keep track, let alone look at everything or try it out. Marcus' excellent curated Pharo Newsletter is one place to see this. Take the February 2020 list of New/Updated Libraries and Frameworks at the end: https://mailchi.mp/pharo/pharo-newsletter-february-2020 This is one of the longest enumerations I have seen so far. The great thing is that it is like that almost every month. Check out the archive: https://us11.campaign-archive.com/home/?u=6f667565c2569234585a7be77=048680a940 Of course, GitHub is also a good way to see this happening. The main entry being: https://github.com/pharo-project/ Which contains several overviews: https://github.com/pharo-project/PharoMap https://github.com/pharo-open-documentation/awesome-pharo Topic tags automatically organise some projects: https://github.com/topics/pharo https://github.com/topics/pharo-smalltalk What all this comes down to is that Pharo has many recently developed, actively maintained options to get your job done: to model your domain, to build your user interface, to talk to other systems, to speak other protocols, to interface with the world, to deploy and to deliver your applications. Similarly, Pavel recently did a very good job at the describing why Pharo itself is so great: https://github.com/pavel-krivanek/pharoMaterials/blob/master/features/PharoKeyFeatures.md So thanks to all of you for helping to make Pharo into what it is. Thanks to our users: for your questions, you're feedback, your bug reports. Thanks to those helping out others on the mailing lists. Thanks to everyone who ever blogged or otherwise wrote about their experiences with Pharo. Thanks to the contributors involved in constantly improving Pharo with Pull Requests, to those working hard to maintain the process and the machinery behind the development process. Thanks to all the developers producing and maintaining the many libraries and frameworks that help us in our day to day work. Thanks to those writing documentation. Thank you. Sven PS: This is already a long email, but I know that I forgot many important points, feel free to add them in reply. -- Sven Van Caekenberghe Proudly supporting Pharo http://pharo.org http://association.pharo.org http://consortium.pharo.org
Re: [Pharo-dev] Wrong behavior of receiveDataInto:fromHost:port:
> On 26 Jan 2020, at 09:42, HilaireFernandes wrote: > > Sven Van Caekenberghe wrote >> Sounds nice, and indeed a (more) fun way to teach programming. >> >> But I still fail to see why you have to filter incoming UDP packets on >> their origin address. > > If third party software on the workstation are connected to other network > services, don't you want to filter out? Or am I missing something? I think so: you listen on an UDP socket on a specific port (and optionally bound to a specific network interface on your machine), then you get only datagrams directed to you - this happens at the OS level. A (at host:port) sends a datagram to B (at host:port), to reply, B looks at the datagram to figure out the sender's info and replies. You can study the example UDPSocketEchoTest to see how this works. HTH, Sven
Re: [Pharo-dev] Fwd: [GitHub] Deprecation Notice
We probably have to change something. Do you know which operation (GitHub API access from Pharo code) is responsible for this ? > On 21 Jan 2020, at 21:05, ducasse wrote: > > what will be the implication? > > >> Begin forwarded message: >> >> From: GitHub >> Subject: [GitHub] Deprecation Notice >> Date: 21 January 2020 at 21:03:28 CET >> To: StéphaneDucasse >> >> Hi @Ducasse, >> >> You recently used a password to access an endpoint through the GitHub API >> using Zinc HTTP Components 1.0 (Pharo/9.0). We will deprecate basic >> authentication using password to this endpoint soon: >> >> https://api.github.com/repositories/169849137 >> >> We recommend using a personal access token (PAT) with the appropriate scope >> to access this endpoint instead. Visit https://github.com/settings/tokens >> for more information. >> >> Thanks, >> The GitHub Team > >
Re: [Pharo-dev] Fed up
I also like the use of symbols but more for casual code. I know using blocks can be faster, but the difference is not massive. What I don't understand is why it is so super bad. Polymorphism will always be there, that is really powerful, when used wisely. I can't immediately see why one or the other would make analysis easier or better. Can you explain ? > On 21 Jan 2020, at 23:37, Sean P. DeNigris wrote: > > ducasse wrote >> in Pharo we should write >> aCol do: [ :each | each store ] > > I always enjoyed the Symbol/Block polymorphism because I thought it was such > a clever and visible example of the power of Smalltalk, and, let's face it, > I'm lazy and enjoyed saving a few key strokes! > > That said, I had no idea that there was a dramatic performance cost. Also, > the issues you raise about analysis seem important. > > Since people are free to still use it in their own projects, it doesn't seem > to controversial. Can/should we add a lint rule? Can/should it be scoped to > Pharo core? > > > > - > Cheers, > Sean > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >
Re: [Pharo-dev] Wrong behavior of receiveDataInto:fromHost:port:
> On 20 Jan 2020, at 17:39, HilaireFernandes wrote: > > It is indeed about *receiveDataInto:fromHost:port:* I am talking about. > > Its *fromHost* parameter is not of the same nature as the *toHost* parameter > in the message *sendUDPData:toHost:port: * > In the former it is expected a ByteArray to work, in the later a > SocketAdress. I don't see any indication in the code to prove your statement "Socket>>#sendUDPData:toHost:port: is assuming hostAddress to be a SocketAddress". Like I said, all primitives take host addresses to be a byte array (of which SocketAddress is a subclass). > *About the context. > *This school year I am doing basic Pharo programming with Dr.Geo (12 and 13 > years old). I wrote a school book[1] for that. But It is a bit tedious to > teach and not enough appealing. It may just be my fault or the format of the > book. For the next semester, I will explore alternative format, in more > independent teaching unit. > > For the longer term, I want to explore alternative way of teaching Pharo, > more appealing. Something more in the kids interest area. > > Do you know Minetest, it is a free implementation of Minecraft? It comes > with a server software and client instances communicating with the server > with a dedicated UDP protocol. In the past, we use it in an architecture > project in connexion with math. > > Written with Pharo, I envision a Minetest client using the UPD protocol to > interact with a server. This Pharo written client will be used as a > programming environment to control the kid avatar in the Minetest 3D world, > like a BotsInc of 3D. The Pharo client will not render the view, it will be > done by the usual Minetest client. > > For example, a kid could code the behaviour of his avatar to build a tower > with a Minetest DSL: > >../.. >100 timesRepeat: [ >avatar goUp >4 timesRepeat: [ >10 timesRepeat: [avatar dropBrick; moveForward]. >avatar turnLeft] ] > > > There are some extensions in Minetest client (Mod in Minetest terminology) > to do some basic programming but it is not as interesting as could be Pharo > programming with a dedicated DSL[3]. > > I speculate it could be pretty cool and it should meet a lot of success > among kids. Plus provide good teaching interest and more Pharo awareness. > > Hilaire > > [1] > https://launchpad.net/drgeo/trunk/19.09/+download/programmer-avec-drgeo.pdf > [2] http://www.minetest.net > [3] > https://forum.minetest.net/viewtopic.php?f=14=365620=5d825763b9cb5579c5e195c0bf4e3b28#p365113 Sounds nice, and indeed a (more) fun way to teach programming. But I still fail to see why you have to filter incoming UDP packets on their origin address. I do agree that using ByteArray and SocketAddress interchanging fails in some cases, even though one is a subclass of the other, because #= fails, due to the class check. I can't see an easy way to fix this (as #= is defined in SequenceableCollection). One solution might be to add #species ByteArray to SocketAddress, but it is hard to see the implications of that. Another option might be to add: SocketAddress>>#= anObject ^ self == anObject or: [ self hasEqualElements: anObject ] This way, #= would work when the SocketAddress is the receiver (but not the other way around which is ugly). If you then put hostAddress first in: Socket>>#receiveDataInto: aStringOrByteArray fromHost: hostAddress port: portNumber "Receive a UDP packet from the given hostAddress/portNumber, storing the data in the given buffer, and return the number of bytes received. Note the given buffer may be only partially filled by the received data." [ | datagram | datagram := self receiveUDPDataInto: aStringOrByteArray. (hostAddress = (datagram at: 2) and: [ (datagram at: 3) = portNumber ]) ifTrue: [ ^ datagram at: 1 ] ifFalse: [ ^0 ] ] repeat it should work. Sven
Re: [Pharo-dev] [Pharo-users] [ANN] Pharo 8.0 Released!
> On 20 Jan 2020, at 19:01, ducasse wrote: > > I would like to also thanks all the consortium members and the association > members so their continuous > support. In addition we would like to thank SchmidtPro and Lifeware for the > contracts we sign to improve Pharo. > Pharo 80 and Pharo 90 will strongly benefit from this effort. +10
Re: [Pharo-dev] printing Symbol
#foo printString.
'#foo'
String streamContents: [ :out | out print: #foo ].
'#foo'
String streamContents: [ :out | out << #foo ].
'foo'
I would not use #storeOn: directly, I would consider the fact that
String>>#printOn: uses it an implementation detail, and a confusing one at that.
> On 20 Jan 2020, at 03:54, Eliot Miranda wrote:
>
>
>
>> On Jan 19, 2020, at 1:50 PM, Stéphane Ducasse
>> wrote:
>>
>>
>> The idea that is that I would like to be able to
>>
>> text -> tokens -> text
>>
>> For text -> tokens
>>
>> (RBScanner on: 'self classVariables: { #A . #B }' readStream)
>> contents collect: #value
>>
>>
>> I wrote a little method that takes the result of the RBScanner and recreate
>> the text
>> But I cannot get this method to work.
>> I’m puzzled because the symbols are eaten.
>>
>>
>> expressionStringFrom: aLine
>>
>> "self new
>> expressionStringFrom: #('self' 'classVariables:' ${ #A $. #B
>> $})
>> >>>
>> 'self classVariables: { A . B }'
>> "
>> ^ String streamContents: [ :s |
>> aLine
>> do: [ :each | s << each ]
>> separatedBy: [ s space ]]
>>
>> I tried with print:, printOn:, but I failed.
>>
>> Any idea?
>
> With Symbols one needs to use storeOn:
>
>>
>> S.
>>
>>
>>
>>
>>
>> Stéphane Ducasse
>> http://stephane.ducasse.free.fr / http://www.pharo.org
>> 03 59 35 87 52
>> Assistant: Julie Jonas
>> FAX 03 59 57 78 50
>> TEL 03 59 35 86 16
>> S. Ducasse - Inria
>> 40, avenue Halley,
>> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
>> Villeneuve d'Ascq 59650
>> France
Re: [Pharo-dev] Wrong behavior of receiveDataInto:fromHost:port:
I am confused, you talk about three different methods, receiveDataInto:fromHost:port: receiveUDPData:toHost:port: sendUDPData... (unspecified) which one do you want to change and how ? In any case, as answered by the primitives, the address seems to be a byte array. So for filtering incoming packets, I would not convert each packet's address, but the target address. > On 20 Jan 2020, at 10:10, HilaireFernandes wrote: > > IMHO, this method is usefull when you are connected to a networked service > (IP and PORT). It helps to filter the datagrams you are interested by. How would you have a setup where you need to do that ? Are you building a firewall ? Can you elaborate on what you are trying to do ? You do not have to answer, but I am interested in learning where this comes from. > Turning SocketAdress as ByteArray could help without the need to convert > each IP ByteArray instance, as I proposed. Not sure about the implication, > though. > > Hilaire > > > > - > http://drgeo.eu > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >
Re: [Pharo-dev] Wrong behavior of receiveDataInto:fromHost:port:
Hi Hilaire, Given the following comment: Socket>>#primSocket: socketID receiveUDPDataInto: aStringOrByteArray startingAt: startIndex count: count "Receive data from the given socket into the given array starting at the given index. Return an Array containing the amount read, the host address byte array, the host port, and the more flag" self primitiveFailed I would says that the primitive result uses a ByteArray to represent the (other party's) host address in its second slot. I find the method Socket>>#receiveDataInto:fromHost:port: quite weird: why would you restrict from whom you receive datagrams, especially since non-matching datagrams are simply thrown away. But I do agree that SocketAddress is a weird class, maybe its #species should be set to ByteArray so that #= would work between them, but I am not sure. Sven > On 19 Jan 2020, at 20:24, HilaireFernandes wrote: > > Hello, > > I would like to discuss one point regarding this method. In my opinion its > parameter hostAddress is expected to be a SocketAdress, but the method > interprets it as a ByteArray. So in the code bellow, the returned result is > 0, although the host responds a datagram: > > server := NetNameResolver addressForName: 'localhost'. > socket := Socket newUDP. > socket sendUDPData: packet toHost: server port: 3. > socket waitForData. > packet := ByteArray new: 200. > result := socket receiveDataInto: packet fromHost: server port: 3. > > The method receiveUDPData:toHost:port: should be amended as follow to make > it symmetric with sendUDPData... : > > [ | datagram | > datagram := self receiveUDPDataInto: aStringOrByteArray. > ((datagram at: 2) *asSocketAddress* = hostAddress and: [ > (datagram at: 3) > = portNumber ]) > ifTrue: [ ^ datagram at: 1 ] > ifFalse: [ ^0 ] ] repeat > > What do you think? > > > > - > http://drgeo.eu > -- > Sent from: http://forum.world.st/Pharo-Smalltalk-Developers-f1294837.html >
Re: [Pharo-dev] Negative number syntax
For the record, should we want to remove the option of having a (possible second) minus sign after the radix specification, the expression self peekSignIsMinus ifTrue: [ neg := neg not ]. should be removed from NumberParser>>#nextNumber and #nextInteger But I haven't tested this. Some unit tests might fail since this 'feature' seems to be supported explicitly. > On 15 Jan 2020, at 17:38, ducasse wrote: > > > >> On 15 Jan 2020, at 14:30, Sven Van Caekenberghe wrote: >> >> Hi, >> >> I came across the following quirk that surprised me. >> >> Positive number in a arbitrary radix >> >> 16rA = 10 >> >> Same number but negative >> >> -16rA = -10 >> >> I guess, we all know that. >> >> However, the following is also possible >> >> 16r-A = -10 > > I would not support this one either >> >> I did not know that, but OK. >> >> But what about this one ? >> >> -16r-A = 10 >> >> I understand that the double negation is positive again, >> but do we really want to support such a syntax ? > > I would not > >> >> Sven
[Pharo-dev] Negative number syntax
Hi, I came across the following quirk that surprised me. Positive number in a arbitrary radix 16rA = 10 Same number but negative -16rA = -10 I guess, we all know that. However, the following is also possible 16r-A = -10 I did not know that, but OK. But what about this one ? -16r-A = 10 I understand that the double negation is positive again, but do we really want to support such a syntax ? Sven
Re: [Pharo-dev] about signal
Hi Alistair, > On 12 Jan 2020, at 09:33, Alistair Grant wrote: > > On Thu, 9 Jan 2020 at 13:01, ducasse wrote: >> >> Hi >> >> I wanted to explain >> >> | semaphore p1 p2 | >> semaphore := Semaphore new. >> p1 := [ semaphore wait. >>'p1' crTrace ] fork. >> >> p2 := [semaphore signal. >> 'p2' crTrace ] fork. >> >> displays p2 and p1. >> but I would like explain clearly but it depends on the semantics of signal. > > The way this is phrased seems to imply that 'p2' will always be > displayed before 'p1', however in Pharo this is not guaranteed (when > the processes are at the same priority, as they are this example). > > As Eliot implied in another reply, Pharo has #processPreemptionYields > set to true, which means that any time a higher priority process > preempts, the current process will be moved to the back of the queue. > > So in the case above, after p2 signals the semaphore, if a timer was > delivered or keystroke pressed, p2 would be suspended and moved to the > back of the queue. When the timer / keystroke / etc. had finished > processing p1 would be at the front of the queue and would complete > first. > > Since time and input events are (for practical purposes) unpredictable > it means that the execution order of processes at a given priority is > also unpredictable. > > While this isn't likely to happen in the example above, I have seen it > regularly with TaskIt and multiple entries being run concurrently. > > I agree with Eliot that changing #processPreemptionYields to true by > default would be an improvement in Pharo. It would make it easier to > predict what is happening in a complex environment. I don't understand, in your second paragraph you say 'Pharo has #processPreemptionYields set to true' and now you say it should become the default. Is that already the case or not then ? > Running the following variant, and then typing in to another window, > demonstrates the behaviour: I am not sure what you want to demonstrate: that it is totally random depending on external factors ;-) ? Which is pretty bad: how should semaphores be used (safely) ? What are good examples of real world correct semaphore usage ? Right now, all the explanations around scheduling of processes and their priorities make it seem as if the answer is 'it all depends' and 'there is no way to be 100% sure what will happen'. Sven > | semaphore p1 p2 | > semaphore := Semaphore new. > [ 100 timesRepeat: [ > p1 := [ | z | > semaphore wait. > z := SmallInteger maxVal. > 1000 timesRepeat: [ z := z + 1 ]. > 'p1' crTrace ] fork. > > p2 := [ | z | 1 second wait. > semaphore signal. > z := SmallInteger maxVal. > 1000 timesRepeat: [ z := z + 1 ]. > 'p2' crTrace ] fork. > 1 second wait. > ] ] fork. > > > The tail of transcript: > > 'p2' > 'p1' > 'p1' > 'p1' > 'p1' > 'p2' > 'p2' > 'p2' > 'p1' > 'p1' > 'p2' > 'p1' > 'p2' > 'p2' > 'p1' > 'p1' > 'p2' > 'p1' > > > > Cheers, > Alistair
