On 26 May 2010 01:58, Schwab,Wilhelm K <[email protected]> wrote: > Sig, > > *nix/mac is **essential** - otherwise, we are not cross-platform. What is > involved in writing the interfaces? I fear it is a little out of my league, > but you have my attention :) > > You mentioned that anything can be done because one can use native code: true > enough and a strength, but I don't want to be forced to resort to it, > especially for simple things. In fact, therein might lurk a weakness of > Alien; IIRC, the author(s) of its documentation declined to boast of its > call-out capabilities, leaving me wondering whether one is left to manage the > stack rather than having the callout mechanism do it. > > I am working on a wrapper for the Gnu Scientific Library. Given the unix > code, I could take a shot at transforming my code to use it. If you have > callbacks, I would have a big reason to dive into it. As it is, I am working > around the need for callbacks, but it will eventually become unavoidable. > One thing I am considering for curve fitting is to create a .so with a large > number of suitable functions whose addresses can be obtained and given to the > library. Ultimately, we need callbacks though. >
For porting it onto other platform, the basic procedure is following: - subclass NativeBoost class - implement a bootstrap routines: #pointerSize #newAssembler #initializeExternalHeap #generateCallgateCode #basicAllocate: aSize on Win32 i am using a HeapCreate/HeapAlloc kernel functions for working with extenal heap. On unixes same functionality could be provided by mmap + own heap management code, or, link with some library, which provides a basic heap operations (allocate/free) on a heap with execution enabled. The main feature, what a platform-specific code should provide is to enable putting a native code into external heap so, other NB parts could be able to make calls to it. This means that memory, which allocated by calls to NativeBoost>>allocate: should always have an execution enabled. http://linux.die.net/man/2/mmap http://linux.die.net/man/2/mprotect > Bill > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Igor > Stasenko > Sent: Tuesday, May 25, 2010 4:51 PM > To: [email protected] > Subject: Re: [Pharo-project] FFI Documentation > > 2010/5/25 Schwab,Wilhelm K <[email protected]>: >> I too am waiting to find something about NB that we should not like. >> No offense to Sig, there has to be _something_ wrong with it, right? >> :) >> > Right. :) > Wrong, is lack of Unix and mac support. It should be written. > > Examples is missing too. > >> A draft list of stuff we should check: >> >> (1) many examples of calls with int, float/double, pointers, structs, etc. >> (2) mac, Linux, Windows >> (3) callbacks >> >> Bonus: >> (4) calls on separate OS threads so the image can defend itself >> against something that does not return in a "reasonable" time, with >> reasonable being up to either the programmer or the end user (which >> the programmer can hopefully enforce if the image is not locked). >> >> Bill >> >> >> >> ________________________________ >> From: [email protected] >> [mailto:[email protected]] On Behalf Of >> Denis Kudriashov >> Sent: Tuesday, May 25, 2010 1:08 PM >> To: Pharo-project >> Subject: Re: [Pharo-project] FFI Documentation >> >> Hi. Igor >> >> Can you compare NB callout functionality with FFI and Alien functionality. >> Can NB replace FFI, Alien? >> >> What absent in NB callout? >> >> You implement callbacks with NB. FFI not support it. >> For me NB foreign interface seems more attractive than something else. >> >> >> 2010/5/21 Igor Stasenko <[email protected]> >>> >>> Very nice topic coverage. I hope, eventually NB will also have >>> something like this :) >>> >>> But it having a little different pros/cons table: >>> >>> NB Pros: >>> * as well as for plugin, you can create new >>> functionality that doesn't exist in any library >>> * no need to recompile plugin/VM when you making a >>> changes >>> * all your code is distributed with the Smalltalk code, >>> but there can be complications with platforms >>> * fast - faster than any FFI implementation written in >>> C, and as fast as plugin primitive or even faster >>> >>> Cons: >>> * ?unsafe? - you have to provide a safety layers >>> (But hey, you have to deal with same sorts of stuff, >>> when writing plugin. No magician workers there) >>> >>> * harder to write >>> Yes, its harder than plain smalltalk - true. >>> But i can't say, that writing an assembler is harder than writing a >>> plugin's slang code. >>> If you writing a plugin, you should have an expertise, in VM >>> internals and how to build VM , etc etc and if you writing an native >>> code, you should have an expertise in assembler as well as VM >>> internals. >>> >>> So, they are different, and definitely much harder comparing to plain >>> smalltalk code, but which one is easier is hard to tell. >>> >>> On 20 May 2010 22:25, Stéphane Ducasse <[email protected]> wrote: >>> > thanks sean >>> > this is cool to see such kind of documentation emerging. >>> > >>> > Stef >>> > >>> > On May 20, 2010, at 9:16 PM, Sean P. DeNigris wrote: >>> > >>> >> >>> >> I want to get this info into the help system and/or collaborative >>> >> book, as similar questions pop up regularly on the lists. >>> >> >>> >> I compiled every piece of info I could find. The focus in on FFI, >>> >> with stubs for other strategies. Please read this *very* rough >>> >> outline for missing info and inaccuracies: >>> >> >>> >> FFI >>> >> >>> >> What is FFI and what is it used for? Calling functions in >>> >> libraries outside of the image... >>> >> >>> >> FFI, the Squeak Foreign Function Interface, is used to call >>> >> functions located in shared libraries that are not part of the >>> >> Squeak VM nor its plugins. It also provides means to read and >>> >> write memory structures that are associated with the use of those >>> >> shared libraries. A typical use is to directly invoke operating >>> >> system APIs. As such, applications that use FFI can only be used >>> >> on the platform(s) that support the particular API being used. C >>> >> conventions are used throughout, though the external function >>> >> could have been written by any language capable of generating >>> >> object code that follows C conventions. FFI is probably the >>> >> easiest way to do the things it does. FFI is pretty fast too. >>> >> Croquet uses FFI calls to OpenGL for all it's drawing routines.[1] >>> >> >>> >> How does FFI work? >>> >> Technically what happens is: >>> >> * you define what the interface is - the parameters, types etc. >>> >> * when you make the call, the FFI logic assembles the data from the >>> >> Squeak >>> >> Objects into the proper structures according to the routine calling >>> >> conventions for your architecture, and of course manages the return >>> >> values. >>> >> So no magic but perhaps just a little assembler in the plugin to >>> >> properly >>> >> deal with all the registers and condition flags. >>> >> >>> >> How do I use it? >>> >> >>> >> 1. make a method (whose structure is similar to a named primitive >>> >> method) >>> >> >>> >> Example: >>> >> system: aString "name (by convention is apiXxx: e.g. apiSystem:)" >>> >> >>> >> <apicall: long 'system' (char*) module: 'libSystem.dylib'> "first >>> >> line >>> >> should be the external function specification" >>> >> ^self externalCallFailed. >>> >> >>> >> Let's take it piece by piece: >>> >> >>> >> system: aString >>> >> Method name - by convention named 'apiXxx' >>> >> >>> >> <apicall: long 'system' (char*) module: 'libSystem.dylib'> >>> >> Function specification >>> >> - should be the first line >>> >> - enclosed in angle brackets: < > containing: >>> >> 1. Calling Convention, either apicall: >>> >> (Pascal convention) or cdecl: (C >>> >> convention) >>> >> - Mac - use either one >>> >> - Unix - use cdecl >>> >> - Windows - use apical >>> >> 2. Return Type (see types) >>> >> 3. External Function Name (literal >>> >> string) >>> >> 4. Argument Types (a literal array) >>> >> 5. Module - "module: " + [filename of the >>> >> external library (literal >>> >> string)] (see below). >>> >> >>> >> self externalCallFailed. >>> >> Failure handler >>> >> - normal smalltalk code >>> >> - executed if the linking to or calling the >>> >> external function fails >>> >> - API calls don't know how to communicate failure >>> >> like Squeak primitives >>> >> do, so: >>> >> - it does not tell you whether the >>> >> external function succeeded >>> >> - the most common code is simply '^self >>> >> externalCallFailed.' >>> >> >>> >> Argument Types >>> >> - must be names of ExternalTypes, either: >>> >> - atomic types (see ExternalType >>> >> class>>initializeFFIConstants and >>> >> ExternalType class>>initializeAtomicTypes): >>> >> void >>> >> bool >>> >> byte (unsigned) >>> >> sbyte (signed) >>> >> ushort (16-bit unsigned) >>> >> short (16-bit signed) >>> >> ulong (32-bit unsigned) >>> >> long (32-bit signed) >>> >> ulonglong (64-bit unsigned) >>> >> longlong (64-bit signed) >>> >> char (unsigned) >>> >> schar (signed) >>> >> float (single-precision float) >>> >> double (double-precision float) >>> >> >>> >> Structure Types [4] >>> >> - subclass of ExternalStructure >>> >> - class>>fields that returns an array of field >>> >> descriptions (see below) >>> >> - Example: >>> >> fields >>> >> ^#((red 'byte')(green >>> >> 'byte')(blue 'byte')) >>> >> - class>>initialize which includes "self defineFields" >>> >> (which must be >>> >> called before using the class) >>> >> - refer to as MyExternalStructure* (to indicate that the argument >>> >> or return >>> >> is a pointer to that structure) >>> >> >>> >> Field description [4] >>> >> - 2-element array (or three but that does something else, I'm not >>> >> sure >>> >> what): >>> >> - first element is the field name >>> >> - second is the type >>> >> >>> >> Mac Memory Allocation Issues [4] (not sure about this) >>> >> >>> >> If you allocate external structures, those with memory outside the >>> >> Squeak >>> >> process space, you may need to increase the amount of memory that is >>> >> reserved outside the object heap for such use. The Mac OS (9 and >>> >> previous) >>> >> needs this, other platforms may be able to dynamically get more memory >>> >> from >>> >> the OS. To see how much memory is currently reserved printIt 'Smalltalk >>> >> extraVMMemory'. To change it, execute 'Smalltalk extraVMMemory: >>> >> someNumberofBytes' Then save your image and quit. When you next start >>> >> up, >>> >> the amount of memory you requested will be reserved. (JMM) Note the OSX >>> >> versions of the VM ignore extraVMMemory because the memory model for >>> >> OS-X/unix applications is quite different. >>> >> >>> >> Module Name >>> >> - depends on the platform >>> >> - Mac >>> >> - pre Snow Leopard: flexible, can eliminate leading lib >>> >> or extension e.g. >>> >> 'libc.dylib' becomes 'libc', 'c.dylib', or 'c' >>> >> - Snow Leopard >>> >> - file name must be exact including extension >>> >> (unless Info.plist is >>> >> altered as in 'Library Location' below) >>> >> - With pre-mach-o VMs >>> >> - For Classic applications, use 'InterfaceLib' >>> >> - For Carbon libs, use 'CarbonLib' >>> >> >>> >> Module Location - where the external library file lives >>> >> - depends on the platform >>> >> - Mac >>> >> - pre Snow Leopard >>> >> - checks VM path and common library paths >>> >> - Snow Leopard >>> >> - only looks in VM bundle's Resources >>> >> file, you must either [5]: >>> >> - store all external libraries >>> >> there >>> >> - ln -s path/to/library >>> >> path/to/VM/Resources/library_name >>> >> - Change the VM's Info.plist >>> >> "SqueakPluginsBuiltInOrLocalOnly" key from >>> >> "true" to "false." >>> >> Caveats >>> >> - security >>> >> - malicious users could call arbitrary functions in the >>> >> OS e.g. "format >>> >> c:" from "system.dll" [7] >>> >> - VMs do not protect against buffer overflow from bad >>> >> parameters [8]: >>> >> "this would require an attacker to execute >>> >> arbitrary Smalltalk >>> >> code on your server. Of course if they can do >>> >> that they own you >>> >> anyway, especially if you allow FFi or use the >>> >> OSProcess plugin" - John >>> >> McIntosh >>> >> >>> >> * difficulty >>> >> - if you make a mistake you'll not drop into the debugger but >>> >> Squeak will >>> >> just crash [2] >>> >> - If you crash Squeak when it is running the garbage collector, >>> >> then you >>> >> know your FFI code is leaking bits into object memory [2] >>> >> >>> >> What do I need to use FFI with Squeak? >>> >> >>> >> You need the FFI plugin, which is included with most VM's as of Squeak >>> >> 3.6 >>> >> or so. >>> >> >>> >> You can also build the plugin yourself. See VMMaker. >>> >> >>> >> References: >>> >> [1] http://wiki.squeak.org/squeak/1414 >>> >> [2] http://wiki.squeak.org/squeak/2424 >>> >> [3] http://wiki.squeak.org/squeak/5716 >>> >> [4] http://wiki.squeak.org/squeak/2426 >>> >> [5] >>> >> >>> >> http://forum.world.st/squeak-dev-Alien-Squeak-FFI-issues-on-Snow-Leopard-td85608.html >>> >> [6] http://wiki.squeak.org/squeak/5846 >>> >> [7] http://forum.world.st/FFI-Callbacks-td54056.html#a54073 >>> >> [8] http://forum.world.st/Security-td99624.html#a99635: >>> >> >>> >> Other choices: >>> >> In the fall of 2008, Alien the FFI interface (written by Cadence Design >>> >> Systems, Inc.) was put into squeaksource: >>> >> http://www.squeaksource.com/Alien.html. This API allows the primitive >>> >> to >>> >> call back to Smalltalk Code, and return error code information, and >>> >> apparently is much faster due to a less complex call sequence. >>> >> * if you need callbacks >>> >> * mac-only? >>> >> >>> >> Plugins - write external code and dynamically link it to the VM >>> >> Pros: >>> >> * safest - users are limited to using the functionality >>> >> you provide and >>> >> can not call other external functions e.g. system rm / >>> >> * fast - faster than FFI >>> >> * you can create new functionality that doesn't exist in >>> >> any library >>> >> Cons: >>> >> * harder to write >>> >> * plugin must be distributed with the Smalltalk code - >>> >> there can be >>> >> complications with platforms >>> >> >>> >> Primitive method - invokes behavior in the VM or a plugin [3] >>> >> >>> >> Questions: >>> >> * why would you want to build the FFI plugin yourself? >>> >> * api prefix or no for method names? >>> >> * not sure about pre Snow Leopard library search >>> >> * OSProcess - how does this fit into the bigger picture? >>> >> * "^self externalCallFailed." or "self externalCallFailed." i.e. return >>> >> self >>> >> or return the result, or doesn't matter? >>> >> * why would a field description have three elements? >>> >> * Mac Memory Allocation Issues? >>> >> -- >>> >> View this message in context: >>> >> http://forum.world.st/FFI-Documentation-tp2225148p2225148.html >>> >> Sent from the Pharo Smalltalk mailing list archive at Nabble.com. >>> >> >>> >> _______________________________________________ >>> >> Pharo-project mailing list >>> >> [email protected] >>> >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> > >>> > >>> > _______________________________________________ >>> > Pharo-project mailing list >>> > [email protected] >>> > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >>> > >>> >>> >>> >>> -- >>> Best regards, >>> Igor Stasenko AKA sig. >>> >>> _______________________________________________ >>> Pharo-project mailing list >>> [email protected] >>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> >> _______________________________________________ >> Pharo-project mailing list >> [email protected] >> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project >> > > > > -- > Best regards, > Igor Stasenko AKA sig. > > _______________________________________________ > Pharo-project mailing list > [email protected] > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > _______________________________________________ > Pharo-project mailing list > [email protected] > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > -- Best regards, Igor Stasenko AKA sig. _______________________________________________ Pharo-project mailing list [email protected] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
