Re: subscribe
Hi Thorsten, I like picolisp (very much) and would like to participate in the mailing list. Good :) I could not figure out how and where to subscribe with Gnus - therefore this email. Perfect. This was the correct way (just a mail with subscribe in the subject). Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Scaling
Hallo, I recently discovered (amazing) picolisp and have a few (I hope not too naive) questions. I write one mail for each question to not mix up things. I read in the documentations about distributed picolisp databases, the ability to make picolisp apps faster and faster by adding hardware cores (and using different pipes of the underlying linux OS?), and the possibility to deploy picolisp-apps in the clouds. But these things are only mentioned, without further explications. Since scaling and concurrency is all the hype in the Java world (scala, clojure) I would like to know a bit more about capabilities and limits of picolisp in this area, and how these things are achieved in practise (ie how to deploy an picolisp-app in the cloud?) Thanks Thorsten
Re: Scaling
Hi Thorsten. Here is a description of a real world example: http://picolisp.com/5000/-2-I.html In that article you will also find some links to functions that might or might now be of use to you, such as (ext). When it comes to distributed data and PicoLisp you don't get much for free (apart from the aforementioned ext functionality). It's more like a framework with which you are able to create something more specific. In short, you won't get something like Cassandra, Hadoop or Riak out of the box but you could certainly create something like them with the tools that you do have. And you could probably create something similar to those three with less hassle than it was to create them in their respective languages (Java / Erlang). /Henrik On Thu, Mar 24, 2011 at 6:11 PM, Thorsten gruenderteam.ber...@googlemail.com wrote: Hallo, I recently discovered (amazing) picolisp and have a few (I hope not too naive) questions. I write one mail for each question to not mix up things. I read in the documentations about distributed picolisp databases, the ability to make picolisp apps faster and faster by adding hardware cores (and using different pipes of the underlying linux OS?), and the possibility to deploy picolisp-apps in the clouds. But these things are only mentioned, without further explications. Since scaling and concurrency is all the hype in the Java world (scala, clojure) I would like to know a bit more about capabilities and limits of picolisp in this area, and how these things are achieved in practise (ie how to deploy an picolisp-app in the cloud?) Thanks Thorsten
Inferior processes and access to foreign libraries
Hello, one thing I like very much about emacs is the ability to run foreign programs (like R and picolisp) as inferior process and communicate with them as if they were part of emacs. I wonder if one could build a picolisp app that fully integrates with R (statistics software, http://www.r-project.org/) and GRASS GIS (a command line GIS that uses a superset of shell commands, http://grass.fbk.eu/) on its linux host machine? Emacs obviously can use foreign programs, like ie ledger mode ( http://www.emacswiki.org/emacs/LedgerMode) which uses a fast program written in C++. I wonder if picolisp can access libraries written in other languages too? It's always claimed that clojure is great because it has access to countless java libraries on the jvm. But clojure is all about functional programming, concurrency and avoiding mutable state, while java is all about objects with mutable state. So it would only make sense for a clojure program to call java libraries like pure functions without side effects and use the return value, otherwise the clojure clean and scalable programming model would be messed up. Can't picolisp do this too? Call Java (and C, C++, Python ...) functions (with a list of data, maybe) and use the return value? Maybe using clojure as man in the middle between java and picolisp who takes care of converting lists in other datastructures and vice versa? Thanks any help on the road to picolisp enlightment Thorsten
Re: Inferior processes and access to foreign libraries
Hi Thorsten,
I wonder if one could build a picolisp app that fully integrates with R
(statistics software, http://www.r-project.org/) and GRASS GIS (a command
line GIS that uses a superset of shell commands, http://grass.fbk.eu/) on
its linux host machine?
There are basically five ways a PicoLisp program can be integrated with
other software:
1. Call existing library functions
This is available only in the 64-bit version with the 'native'
function. C functions can be directly called from Lisp code. For an
extensive example, look at lib/openGl.l.
2. Write a shared object file (DLL)
This is analog in the 32-bit and 64-bit versions, but due to the
underlying implementation such shared object files are written in C
for the 32-bit version, and in assembly for the 64-bit version.
Examples are the 'ext' and 'ht' libraries, built in the Makefile's
(src/ext.c, src/ht.c, src64/ext.l and src64/ht.l).
3. Write inline C code
The 32-bit and 64-bit versions of PicoLisp behave a little different
here. In the 32-bit version, these functions must contain glue code
to convert between C and Lisp data. The file misc/crc.l contains an
example for both versions.
4. Start other programs as sub-processes, and communicate with their
standard I/O via pipes. This can be done with the 'in', 'out' and
'pipe' functions.
5. Use sockets or some other type of IPC (e.g. named pipes) to
communicate with another program.
Did I forget some other way?
It's always claimed that clojure is great because it has access to countless
java libraries on the jvm. But clojure is all about functional programming,
BTW, there is also ErsatzLisp, a subset of PicoLisp written in Java. It
provides for some ways to call the underlying JVM.
concurrency and avoiding mutable state, while java is all about objects with
mutable state. So it would only make sense for a clojure program to call
java libraries like pure functions without side effects and use the return
value, otherwise the clojure clean and scalable programming model would be
messed up.
Yep. The same problems apply to ErsatzLisp.
Can't picolisp do this too? Call Java (and C, C++, Python ...) functions
Several years ago, the PicoLisp GUI depended heavily on Java applets.
This stuff is still available as a separate tarball picoJavaGUI.tgz.
It also contains a file lib/java.l which provides an interface to a
separately running JVM. This was never really used, though.
In one project we communicated with C++ programs via a PLIO library.
PLIO is the format used by PicoLisp internally for marshalling database
objects, for the 'pr' and 'rd' functions, and family IPC ('tell' etc.).
A skeleton for PLIO can be fetched from software-lab.de/plio.tgz.
I'll gladly explain more details if you have specific questions :)
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Inferior processes and access to foreign libraries
Hi Alex,
This overview of the different ways of integration with other software
is very good! It deserves a place in the wiki. ;-)
/Jon
On 24-03-11 15:08 , Alexander Burger wrote:
Hi Thorsten,
I wonder if one could build a picolisp app that fully integrates with R
(statistics software, http://www.r-project.org/) and GRASS GIS (a command
line GIS that uses a superset of shell commands, http://grass.fbk.eu/) on
its linux host machine?
There are basically five ways a PicoLisp program can be integrated with
other software:
1. Call existing library functions
This is available only in the 64-bit version with the 'native'
function. C functions can be directly called from Lisp code. For an
extensive example, look at lib/openGl.l.
2. Write a shared object file (DLL)
This is analog in the 32-bit and 64-bit versions, but due to the
underlying implementation such shared object files are written in C
for the 32-bit version, and in assembly for the 64-bit version.
Examples are the 'ext' and 'ht' libraries, built in the Makefile's
(src/ext.c, src/ht.c, src64/ext.l and src64/ht.l).
3. Write inline C code
The 32-bit and 64-bit versions of PicoLisp behave a little different
here. In the 32-bit version, these functions must contain glue code
to convert between C and Lisp data. The file misc/crc.l contains an
example for both versions.
4. Start other programs as sub-processes, and communicate with their
standard I/O via pipes. This can be done with the 'in', 'out' and
'pipe' functions.
5. Use sockets or some other type of IPC (e.g. named pipes) to
communicate with another program.
Did I forget some other way?
It's always claimed that clojure is great because it has access to countless
java libraries on the jvm. But clojure is all about functional programming,
BTW, there is also ErsatzLisp, a subset of PicoLisp written in Java. It
provides for some ways to call the underlying JVM.
concurrency and avoiding mutable state, while java is all about objects with
mutable state. So it would only make sense for a clojure program to call
java libraries like pure functions without side effects and use the return
value, otherwise the clojure clean and scalable programming model would be
messed up.
Yep. The same problems apply to ErsatzLisp.
Can't picolisp do this too? Call Java (and C, C++, Python ...) functions
Several years ago, the PicoLisp GUI depended heavily on Java applets.
This stuff is still available as a separate tarball picoJavaGUI.tgz.
It also contains a file lib/java.l which provides an interface to a
separately running JVM. This was never really used, though.
In one project we communicated with C++ programs via a PLIO library.
PLIO is the format used by PicoLisp internally for marshalling database
objects, for the 'pr' and 'rd' functions, and family IPC ('tell' etc.).
A skeleton for PLIO can be fetched from software-lab.de/plio.tgz.
I'll gladly explain more details if you have specific questions :)
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Scaling
Hi Thorsten, in addition to what Henrik wrote: So dividing a database in several smaller files and accessing them with something like id or ext gives a distributed faster database, and when doing Dividing the database into multiple files is the normal approach to design a DB application in PicoLisp, so this is not what I would call distributed. Distribution involves separate machines, connected via TCP. On each machine, typically several PicoLisp database processes are running, and they exchange objects via 'id' or 'ext', but - more importantly - can do remote calls (via 'pr', 'rd' etc., i.e. the PLIO protocol mentioned in the other mail) and remote queries (see doc/refR.html#remote/2). Direct remote DB operations involve only read accesses (queries). Changes to the individual DBs have to be done the normal way (e.g. the 'put' family of methods), where each application (PicoLisp process family) is maintaining its own DB. Hmm, that's all rather hard to explain, and unfortunately not formally documented yet (except for Henrik's great descriptions). so ie in an Amazon EC2 account the database might (automagically) end up on different servers, thus becoming faster and (almost endlessly) scalable. Yes, though the current system doesn't have any mechanisms for dynamically relocation of database processes yet. Actually, I was planning for something along that way, but the project where I would have needed that was terminated :( Is anybody using Emacs/Gnus for this mailing list and can give some advice how to make that work? Yes, our Argentinian frieds. By now, they should be up ;-) Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Inferior processes and access to foreign libraries
Hi Jon, This overview of the different ways of integration with other software is very good! It deserves a place in the wiki. ;-) Good idea! I'm still not sure, however, if it is complete. I'm getting a bit confused about that, it is the only place where pil32 and pil64 are substantially different. What might be a proper title? Foreign Code Integration? Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Inferior processes and access to foreign libraries
Hi Alex, On 24-03-11 16:21 , Alexander Burger wrote: Hi Jon, This overview of the different ways of integration with other software is very good! It deserves a place in the wiki. ;-) Good idea! I'm still not sure, however, if it is complete. I'm getting a bit confused about that, it is the only place where pil32 and pil64 are substantially different. If it's not complete, you can always add more later ... ;-) What might be a proper title? Foreign Code Integration? Cheers, - Alex I think Foreign Code Integration sounds good. /Jon -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Inferior processes and access to foreign libraries
Hi Alexander,
thanks for all that information, definitely worth a wiki entry.
Due to limited skills number 1. and number 4./5. seem to be the most
realistic options for me, but I will need some time to digest so much new
info.
My plans are to read and learn more about picolisp the next weeks, while
terminating some other stuff, and then build a small but real webapp in
picolisp - I'm already looking forward for that.
Picolisp seems so easy and so powerful at the same time.
Cheers
Thorsten
2011/3/24 Alexander Burger a...@software-lab.de
Hi Thorsten,
I wonder if one could build a picolisp app that fully integrates with R
(statistics software, http://www.r-project.org/) and GRASS GIS (a
command
line GIS that uses a superset of shell commands, http://grass.fbk.eu/)
on
its linux host machine?
There are basically five ways a PicoLisp program can be integrated with
other software:
1. Call existing library functions
This is available only in the 64-bit version with the 'native'
function. C functions can be directly called from Lisp code. For an
extensive example, look at lib/openGl.l.
2. Write a shared object file (DLL)
This is analog in the 32-bit and 64-bit versions, but due to the
underlying implementation such shared object files are written in C
for the 32-bit version, and in assembly for the 64-bit version.
Examples are the 'ext' and 'ht' libraries, built in the Makefile's
(src/ext.c, src/ht.c, src64/ext.l and src64/ht.l).
3. Write inline C code
The 32-bit and 64-bit versions of PicoLisp behave a little different
here. In the 32-bit version, these functions must contain glue code
to convert between C and Lisp data. The file misc/crc.l contains an
example for both versions.
4. Start other programs as sub-processes, and communicate with their
standard I/O via pipes. This can be done with the 'in', 'out' and
'pipe' functions.
5. Use sockets or some other type of IPC (e.g. named pipes) to
communicate with another program.
Did I forget some other way?
It's always claimed that clojure is great because it has access to
countless
java libraries on the jvm. But clojure is all about functional
programming,
BTW, there is also ErsatzLisp, a subset of PicoLisp written in Java. It
provides for some ways to call the underlying JVM.
concurrency and avoiding mutable state, while java is all about objects
with
mutable state. So it would only make sense for a clojure program to call
java libraries like pure functions without side effects and use the
return
value, otherwise the clojure clean and scalable programming model would
be
messed up.
Yep. The same problems apply to ErsatzLisp.
Can't picolisp do this too? Call Java (and C, C++, Python ...) functions
Several years ago, the PicoLisp GUI depended heavily on Java applets.
This stuff is still available as a separate tarball picoJavaGUI.tgz.
It also contains a file lib/java.l which provides an interface to a
separately running JVM. This was never really used, though.
In one project we communicated with C++ programs via a PLIO library.
PLIO is the format used by PicoLisp internally for marshalling database
objects, for the 'pr' and 'rd' functions, and family IPC ('tell' etc.).
A skeleton for PLIO can be fetched from software-lab.de/plio.tgz.
I'll gladly explain more details if you have specific questions :)
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Inferior processes and access to foreign libraries
Hi Henrik, that's what I thought - moving lists back and forth between two variants of lisp should somehow be manageable. Thanks for the tip, I wil have a closer look. Thorsten BTW a lot of nice articles on your blog. 2011/3/24 Henrik Sarvell hsarv...@gmail.com Hi Again :) Yes I've done some experimenting in this direction here: http://www.prodevtips.com/2010/05/30/clojure-with-a-picolisp-database-via-clojure-http-client/ On Thu, Mar 24, 2011 at 7:03 PM, Thorsten gruenderteam.ber...@googlemail.com wrote: Hello, one thing I like very much about emacs is the ability to run foreign programs (like R and picolisp) as inferior process and communicate with them as if they were part of emacs. I wonder if one could build a picolisp app that fully integrates with R (statistics software, http://www.r-project.org/) and GRASS GIS (a command line GIS that uses a superset of shell commands, http://grass.fbk.eu/) on its linux host machine? Emacs obviously can use foreign programs, like ie ledger mode ( http://wwwemacswiki.org/emacs/LedgerModehttp://www.emacswiki.org/emacs/LedgerMode) which uses a fast program written in C++. I wonder if picolisp can access libraries written in other languages too? It's always claimed that clojure is great because it has access to countless java libraries on the jvm. But clojure is all about functional programming, concurrency and avoiding mutable state, while java is all about objects with mutable state. So it would only make sense for a clojure program to call java libraries like pure functions without side effects and use the return value, otherwise the clojure clean and scalable programming model would be messed up. Can't picolisp do this too? Call Java (and C, C++, Python ...) functions (with a list of data, maybe) and use the return value? Maybe using clojure as man in the middle between java and picolisp who takes care of converting lists in other datastructures and vice versa? Thanks any help on the road to picolisp enlightment Thorsten
Re: Scaling
Hi Alexander, Distribution involves separate machines, connected via TCP. On each machine, typically several PicoLisp database processes are running, Changes to the individual DBs have to be done the normal way (e.g. the 'put' family of methods), where each application (PicoLisp process family) is maintaining its own DB. Is my interpretation right, that the ' several PicoLisp database processes' running on one machine form a 'PicoLisp process family' that is considered as one application with one database? So it is one database per machine, using several processes on that machine, that has to be changed individually, but can be queried as part of a distributed red of databases on several machines connected via TCP? How do you split up the databases? Rather by rows or rather by columns (I know they are not 2D tables in picolisp, what I mean is: does every DB cover the whole class hierarchy, but only a fraction of the objects, or does each DB cover a fraction of the class hierarchy, but all objects belonging to these classes? Cheers Thorsten
Re: Inferior processes and access to foreign libraries
Hi Alexander, Yeah, these are the easiest. If you have a 64-bit system, however, 3 is as simple as 1. Unfortunately I installed 32-bit Ubuntu on my machine, only to find out later that it has 64-bit architecture ... If you like, you might join the #picolisp IRC channel. Gives more interactive feedback. And, it looks we are in the same time zone :) yes, I live in Berlin. The #picolisp IRC channel is another thing I have to check out (I never used IRC ;). But I already found an emacs mode for IRC, so lets see ... Cheers Thorsten
Re: Inferior processes and access to foreign libraries
On Thu, Mar 24, 2011 at 9:54 AM, Alexander Burger a...@software-lab.dewrote: Hi Thorsten, If you like, you might join the #picolisp IRC channel. Gives more interactive feedback. And, it looks we are in the same time zone :) irc.freenode.net /join #picolisp I highly recommend it Thorsten.
Re: Scaling
Hi Thorsten, Distribution involves separate machines, connected via TCP. On each machine, typically several PicoLisp database processes are running, Is my interpretation right, that the ' several PicoLisp database processes' running on one machine form a 'PicoLisp process family' that is considered as one application with one database? Yes, but there may be also several such families on a single machine. A single application, operating on a single database, consists of a parent process with an arbitrary number of child processes. This structure is necessary because synchronization of all processes that access a given database must go via a common parent (family IPC uses simple pipes). A single database means usually a single directory, containing all files of that database. Theoretically, a database may consist of maximally 65536 files, but this dosn't make sense in a typical Unix environment, because of too many file descriptors and other resource problems. A single file can contain maximally 4 Tera objects (42 bit object ID). It makes well sense to run several applications (= databases) on a single machine, to get a better load distribution. I have no general rule, for opimal tuning some experimentation is required. It depends mostly on the number of CPU cores and the amount of available RAM (file buffer cache). For the program logic (how those applications communicate with each other), it doesn't matter which application is running on which machine, as long as all is properly configered. I had an admin application for connecting/starting/stopping the individual apps. How do you split up the databases? Rather by rows or rather by columns (I Not on that level, but on a functional level. For example, we had many databases (about 70) collecting data from filer volumes, sending some of their data to a second layer (also 70) which in turn sent some boiled-up stuff to a single dedicated database containing some global data. Another front-end application queried all the lower levels to generate statistics and user reports, and contained a rule database (in Pilog) about what to do on the lower levels. know they are not 2D tables in picolisp, what I mean is: does every DB cover Right. the whole class hierarchy, but only a fraction of the objects, or does each Yes, this was the case for the first and second layer described above. In each layer all databases had the same model (E/R definitions, in fact the same program code). DB cover a fraction of the class hierarchy, but all objects belonging to these classes? So each application is a complete class hierarchy in itself, independent from (but knowing about) the other DBs. But what I described was for that concrete use case. I had only a single project with such large DBs until now. Probably many other designs are possible. As Henrik said, stress is on ease of designing such structures, not on a given framework. The philosophy of PicoLisp was always to go for a vertical approach, with easy access from the lowest to the highest levels. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Pico Lisp and Emacs Lisp
On 03/24/2011 08:39 AM, Thorsten wrote: Hallo, it seems to me that elisp and picolisp are close relatives in the lisp familiy, Yeah... they both use parens and dynamic binding... and I wonder if it would be possible to convert elisp code to picolisp code - and how difficult this would be? No way, raw translation won't do any good. Porting is required. There have been apparently successful attempts to convert elisp to scheme (http://www-pu.informatik.uni-tuebingen.de/users/knauel/selc-ifl.pdf), and scheme is very different from elisp. Not so different if you leave aside the #t #f '() and dynamic binding. (it would be more troublesome to port stuff from scheme to elisp) I was thinking about `refactoring` the text in *.el files to a syntax that picolisp can understand, but that might be too naive. All the (1800 ?) primitive C functions in elisp were problematic when porting elisp to scheme, but maybe thats not the case with picolisp. Of course the thousands of buffer functions etc are meaningless in picolisp, Deppends on what you want to do, but a LOT of elisp code (I'd say most of it) deppends on buffers (as in data type/structure). If you mean buffers as in frames/windows, yeah.. but there's very little of it AFAIK. since there are no buffers in the gui framework. But maybe one could connect the conkeror webbrowser (http://conkeror.org/), a fine javascript browser modeled closely after emacs, to the gui framework of picolisp and map the emacs buffer commands etc to the related conkeror concepts (it has buffers, keymaps ...). Then suddenly many of those emacs modes and libraries would make sense in picolisp, and with more than 1 million lines of elisp code available the claims that 'picolisp has no libraries' would stop. They wouldn't be modeled the picolisp way, nor they'd be specific to it. Besides... most of emacs is about modes. Most about modes is parsing, highlighting and indenting (maybe some smart stuff like applying overlays to hide stuff like in html-mode) and a few macros and word-delimiters definitions. So.. what's there to port? Regarding conkeror.. I've used it for a year, but then moved to vimperator which I find to be quite superior in most aspects. All this would make sense if there was a picolisp-based-editor, and even if that were the case, it's not a good idea to mass-rip stuff from emacs (since emacs is full of contradictions and different criteria) By the way, these kind of discussions are better in IRC (#picol...@irc.freenode.net) - Arm -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Pico Lisp and Emacs Lisp
Did someone just suggest: All this would make sense if there was a picolisp-based-editor, and even if that were the case, it's not a good idea to mass-rip stuff from emacs (since emacs is full of contradictions and different criteria) By the way, these kind of discussions are better in IRC (# picol...@irc.freenode.net) - Arm Yes. I'm learning a lot from them.
