Re: Problems with +Blob
Hi Henrik Randall, I just went and read that too - links per inode would include both directories or files or a mix of both. I checked the filesystem code for this, the ext3 inode is limited by a 16bit links_count. Under linux (UNIX) a directory simply contains a list of names and their inodes (a link) - there is no difference at that level between a file and a sub-directory. I just checked the ext3 limits on wikipedia, apparently it can't handle more than roughly 32000 sub folders inside a folder, it's unclear whether that applies to files too or not. Anyway, how does it work, could this level be reached or is there a stopper somewhere? I don't think the number of files is limited this way. I can definitely create a directory with over 100k files directly in it. Inodes seem to form a linked list so it just gets slow to work with when there are lots of files. Cheers, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Henrik, It seems to me that the 64bit version will be completely incompatible with the old 32bit databases/blobs. Yes. Though they are completely compatible on the Lisp source level, a database file itself cannot be exchanged between the two versions (and a 32-bit app and a 64-bit app cannot communicate via PLIO when external symbols are involved). Alex how will you go about migrating all the old databases? I believe it is straightforward to write a conversion program. But on the other hand, I will probably not migrate any existing application, as I see no technical reason for it. Perhaps apps running in 32 bits are even more efficient? Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
On Mon, Aug 17, 2009 at 10:45:08AM +0200, Henrik Sarvell wrote: But for new projects you will use the 64bit version, I mean if not then why bother with creating it? I'm not sure yet. Let's see how it works out, and wait until it is all bug-free ;-) In fact, in the long range I believe that I'll concentrate on the 64-bit version. It may may be a little bit too early to decide at the moment, but certain aspects are surely more efficient in 64-bits. Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Works, but I can't pretend to understand how the folder and file naming wor= ks. I just checked the ext3 limits on wikipedia, apparently it can't handle more than roughly 32000 sub folders inside a folder, it's unclear whether that applies to files too or not. Anyway, how does it work, could this level be reached or is there a stopper somewhere? /Henrik On Sun, Aug 16, 2009 at 7:17 AM, Alexander Burgera...@software-lab.de wrot= e: Sorry to all. My mistake! I did not look closely enough :-( The two operations must be in opposite order, first the 'put' and then the 'out'. Otherwise the blob file might not exist yet. So the correct procdeure is: =A0 (put! Article 'body T) =A0 (out (blob Article 'body) =A0 =A0 =A0(prinl a really long text) ) Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=3dunsubscribe -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Henrik, I just went and read that too - links per inode would include both directories or files or a mix of both. I checked the filesystem code for this, the ext3 inode is limited by a 16bit links_count. Under linux (UNIX) a directory simply contains a list of names and their inodes (a link) - there is no difference at that level between a file and a sub-directory. So, yes, that applies to files, too. Alex, can that number of db files be reached (likely, not probable, impossible)??? Rand On Sun, Aug 16, 2009 at 7:11 PM, Henrik Sarvell hsarv...@gmail.com wrote: Works, but I can't pretend to understand how the folder and file naming wor= ks. I just checked the ext3 limits on wikipedia, apparently it can't handle more than roughly 32000 sub folders inside a folder, it's unclear whether that applies to files too or not. Anyway, how does it work, could this level be reached or is there a stopper somewhere? /Henrik On Sun, Aug 16, 2009 at 7:17 AM, Alexander Burgera...@software-lab.de wrot= e: Sorry to all. My mistake! I did not look closely enough :-( The two operations must be in opposite order, first the 'put' and then the 'out'. Otherwise the blob file might not exist yet. So the correct procdeure is: =A0 (put! Article 'body T) =A0 (out (blob Article 'body) =A0 =A0 =A0(prinl a really long text) ) Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=3dunsubscribe -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
On Sun, Aug 16, 2009 at 07:11:43PM +0200, Henrik Sarvell wrote:
I just checked the ext3 limits on wikipedia, apparently it can't
handle more than roughly 32000 sub folders inside a folder, it's
No problem. There are never more than 64 subdirectories or files in a
blob-directory.
As always in picoLisp, is all very simple: The name of the external
symbol is 'chop'ped, and 'glue'd with the slash (lib/db.l:394):
(de blob (Obj Var)
(pack *Blob (glue / (chop Obj)) . Var) )
Then 'Var' is the relation's name, and gets appended as the file's
extension.
For example, the '+Item' class in the demo application has two blobs
(rel txt (+Blob)) # Memo
(rel jpg (+Blob)) # Picture
If you have an '+Item' '{3-1a7B}' with both a memo and a picture, then
these blobs will be the files
blob/app/3/-/1/a/7/B.txt
blob/app/3/-/1/a/7/B.jpg
On Sun, Aug 16, 2009 at 7:17 AM, Alexander Burgera...@software-lab.de wrote:
So the correct procedure is:
(put! Article 'body T)
(out (blob Article 'body)
(prinl a really long text) )
The 'put' method of the '+Blob' relation class takes care of creating
the path of directories. So it must be _before_ the 'out' statement.
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Henrik Randal, Alex, can that number of db files be reached (likely, not probable, impossible)??? Works, but I can't pretend to understand how the folder and file naming wor= ks. Anyway, how does it work, could this level be reached or is there a stopper somewhere? as far as I can tell, it is almost impossible because the directory structure for blobs is not flat but it is a tree instead. It looks like the names of persistent symbols are represented as 'fmt64' or so, and the (sub)directories and each letter of the name (except the last one) is a subdirectory of the blob directory. Thus, there can be up to 56 (sub)directories (plus maximum about the same number of files multiplied by the number of blob rel attributes: well, if you had lots of those, there could be problems but you would have to try hard to break it in this way). I am interested in the details too though;-) Cheers, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Randall,
Alex, can that number of db files be reached (likely, not probable,
impossible)???
No need to worry (I just wrote another mail, answering Henrik's
question).
In addition to those explanations, let me shortly show how this will
look in 64-bit picoLisp.
The internal representation of database symbols is quite different in
the new version.
In the 32-bit version, object names were something like '{34-a;7X}'.
That is, it uses a base64 representation of both the database file
number and the ID of the object within the file. The dash '-' separates
the file number from the object ID, and both of them consist of
characters 0-9, :, ;, A-Z and a-z.
In the 64-bit version, names are packed more efficiently internally, so
that they completely fit into a 62 bit word (pico/doc64/structures).
This results also in a different printed representation. Now the
database file number is encoded in a kind of hexadecimal number, which
uses @, A, B .. O instead of 0, 1, 2, .. F. The object ID is encoded in
octal. This has the advantage that the dash '-' as a separator is not
needed any more, because the file number digits 0 .. 7 are different
from the letters A .. O. With that, an object might look like {1},
{123}, {A45}, {CDE7654321} and so on.
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Tomas, bingo! As ever, you are very right :-) as far as I can tell, it is almost impossible because the directory structure for blobs is not flat but it is a tree instead. It looks like the names of persistent symbols are represented as 'fmt64' or so, and the (sub)directories and each letter of the name (except the last one) is a subdirectory of the blob directory. Thus, there can be up to 56 (sub)directories (plus maximum about the same number of files Why do you think 56 istead of 64? multiplied by the number of blob rel attributes: well, if you had lots Yep. Good definition. of those, there could be problems but you would have to try hard to break it in this way). Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
On Sun, Aug 16, 2009 at 08:20:31PM +0200, Alexander Burger wrote:
from the letters A .. O. With that, an object might look like {1},
{123}, {A45}, {CDE7654321} and so on.
Oops, I forgot the point I actually wanted to explain ;-)
With the above naming scheme, the splitting in directories for blobs
will be handled differently:
If we would split a symbol like {CDE7654321} in the same way as on the
32-bit version, we would get rather long path names
blob/app/C/D/E/7/6/5/4/3/2/1.ext
and directories with a low branching factor (16 or 8).
Instead, I will keep the complete file number specification (here CDE)
as a single top-level directory, and then always group two octal numbers
together to get a branching factor of 64 again:
blob/app/CDE/76/54/32/1.ext
This seems more reasonable to me. Praxis has shown that only a few
database files per application actually contain blobs, so that the top
level directories (here CDE) would not be so very many. In total, the
pathes will be more compact.
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Alex, bingo! As ever, you are very right :-) not always, I am a human too, see bellow:-D Why do you think 56 istead of 64? 64 makes sense! Surprisingly in my database which is a few levels deep, has 56 directories and some top level subdirectories are still missing. Not sure why. Cheers, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Alex,
database file number is encoded in a kind of hexadecimal number, which
uses @, A, B .. O instead of 0, 1, 2, .. F. The object ID is encoded in
octal. This has the advantage that the dash '-' as a separator is not
needed any more, because the file number digits 0 .. 7 are different
from the letters A .. O. With that, an object might look like {1},
{123}, {A45}, {CDE7654321} and so on.
why do you need to mix hexadecimal and octal? I imagine having the
second part in hex would be possible too. E.g. using 0-F for the
second part and G.. for the first would work too and the names would
be shorter?
Cheers,
Tomas
--
UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
It seems to me that the 64bit version will be completely incompatible
with the old 32bit databases/blobs. Alex how will you go about
migrating all the old databases?
/Henrik
On Sun, Aug 16, 2009 at 9:10 PM, Tomas Hlavatyt...@logand.com wrote:
Hi Alex,
database file number is encoded in a kind of hexadecimal number, which
uses @, A, B .. O instead of 0, 1, 2, .. F. The object ID is encoded in
octal. This has the advantage that the dash '-' as a separator is not
needed any more, because the file number digits 0 .. 7 are different
from the letters A .. O. With that, an object might look like {1},
{123}, {A45}, {CDE7654321} and so on.
why do you need to mix hexadecimal and octal? =A0I imagine having the
second part in hex would be possible too. =A0E.g. using 0-F for the
second part and G.. for the first would work too and the names would
be shorter?
Cheers,
Tomas
--
UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=3dunsubscribe
--
UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Henrik, To me it seems you need to pass some kind of file name, for instance in the app/item.l form, the name of the tempfile. And then the file is opened and copied to the blobfile. yes, a blob is a file. I think it is stored as a flag indicating it's presence. If it is present, the function 'blob' will return a filename where you can find its content. However I simply want to do like this: (put Article 'body a really long text here, not a file...) and have the text end up in an automatically generated blob file. As far as I can see the current put and/or blob! functions working with +Blob can't be made to work like this? Well, if the text is really really long then it probably should not be represented as a string in the first place. If it is not that long, I would either use +String instead of +Blob or use +Blob the way Alex uses it. I haven't looked at the sources, but maybe there is a way adding such abstraction/method as you aim for. Cheers, Tomas -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Henrik, In the top section of my main I have this: (setq *Blob (pack *BP blob/)) where *BP is my base path. OK. I've seen so far. Code which is using +BlobField or blob! To me it seems you need to pass some kind of file name, for instance in the ... However I simply want to do like this: (put Article 'body a really long text here, not a file...) and have the text end up in an automatically generated blob file. Yes, that's right. These are pre-cooked classes and functions. To do it manually, you need two steps: (out (blob Article 'body) (prinl a really long text) ) (put! Article 'body T) If you put 'NIL' instead of 'T', the blob is deleted. It will be removed the next time by 'dbgc'. Alternatively you could remove it immediately: (call 'rm (blob Article 'body)) (put! Article 'body NIL) Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
Re: Problems with +Blob
Hi Tomas, using temporary files is fine. If you operate on the lower level of 'put!', you could even write the blob file directly. I think the initial worry of Henrik was how to avoid writing a temp file first, and then using 'blob!'. As I wrote in my answer, the correct and simple way is (out (blob Article 'body) (prinl a really long text) ) (put! Article 'body T) There is not more to blobs. It is just that the typical cases of blobs, like images or PDFs, start out with a file. Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe
