Dictionary files are not like regular files.
They're a special binary format, and from the file system standpoint, the
size of the dictionary file remains constant - which is unlike regular
files. With regular files, every time you open it with some
specified minimumSize, Blackbook will double check that the space
on flash reserved for that file is *at least* the minimumSize you specified, and
adjust. So, for example, if you opened a regular file for the first
time with 0x100 bytes minimumSize, and then open it a second time
requesting 0x1000 bytes minimumSize, then the file will have grown to be at
least 0x1000 bytes long. Dictionary files don't do
this.
The
Dictionary file parser (BDictionaryM.nc) knows better than to rely on what the
file system says when it comes to the data length of a dictionary file.
The parser reads the binary file up to the fill bytes (0xFF), and therefore
knows exactly how much real data is actually contained within
the Dictionary file at the time the file is opened.
"once the file is out of Blackbook,
there's no way to determine the end of "real" data and start of alien
data."
Right
- because Dictionary files need to be parsed to extract the useful
information. It should be possible to copy a dictionary file completely
over to another mote in a straight binary read/write fashion
(BFileRead/BFileWrite) and then use BDictionary to open the file and parse it
out. The only other method is to copy the key-values out key by key,
while traversing the Dictionary file.
"..
is totalSize the
ceil-to-page-multiple value of reservedLength or it's the same as minimumSize at
creation (and correspondingly for remainingBytes)?"
Looking back at my code, the totalSize
parameter in BDictionary.opened(...) ==
BFileDir.getReservedLength(<dictionary file>), which is the amount of
data in the file you have available to write to.
The size you have available in your dictionary file is
equal to the minimumSize value you passed in to begin with, extending to the
next page boundary if necessary.
Flash
pages on the AT45DB and STM25P80 flash types are in increments of 0x100
bytes. For every file you open in Blackbook, there's 10 bytes of nodemeta
(per node - regular files can extend multiple nodes on flash, and a
Dictionary file can only extend a single node on flash) and 14 bytes
of filemeta that gets written at the start of the file. This metadata is
stuff the file system doesn't let you see. That means if you request
a minimum of 0x100 bytes to write to, the file system will tack on 24 bytes of
metadata, bringing the total file size to 0x118 bytes. This does not land
on a page boundary, so the file system will make the file size up to the next
page boundary, which happens to be 0x200. Now you have a 0x200 byte
footprint on flash for your file, and 24 bytes of that is metadata, leaving you
with 488 bytes (0x1E8) to write to. Since 0x1E8 is greater than the 0x100
bytes minimum size you requested, everyone is happy.
Dictionary files also happen to have a 0xD1C7 magic
number at the beginning of writable data which is included as part of the
"totalSize" of the file. Each insert takes up 10 bytes in keymeta
storage, and the value you store can of course be variable
size.
If
you're interested in parsing a Dictionary file, I can explain the layout of the
data if you'd like. Let me know if you need further
clarifications,
-David
-----Original Message-----Dear Mr. Moss,
From: Manu Bansal [mailto:[EMAIL PROTECTED]
Sent: Tuesday, September 19, 2006 9:07 AM
To: David Moss
Cc: [email protected]
Subject: Re: [Tinyos-help] RE: Blackbook5: dataLength, minimumSize for dictionary files
Please read inline.
On 9/18/06, David Moss <[EMAIL PROTECTED]> wrote:Hi Manu,
> In Blackbook5, what does BDictionary.getDataLength return?
> .. what should minimumSize include?
First, are you referring to the BFileDir.getDataLength(char *filename)
command, being run on a Dictionary file?
Yes. Excuse the slip please.
Dictionary files have special properties to them that allow them to be
updated indefinitely. The way it works is when you first open a Dictionary
file, you must specify the minimum size. On the flash, Blackbook will
reserve *at least* that minimum size to store data - but if you want the
background information on that, the actual size on flash will extend to the
next page boundary. This can make your actual file size up to 255 bytes
larger than you requested. Better a file be made a few bytes too big than
too small.
After you have opened a Dictionary file for the first time, its footprint on
flash will remain constant throughout its lifetime. The dictionary file
itself will physically move around to various locations on flash as it fills
up and removes invalid keys throughout its lifetime, but the valid
(undeletable) footprint on flash will always stay the same.
This is unexpected - I understood that the footprint could actually be made to grow/shrink on each open call by passing an appropriate mimimumSize. Does that hold for (only) regular files?
Because flash can't actually "delete" data very easily, is better to leave
old data written to flash and just ignore it. That means that as you delete
or update existing keys in your Dictionary file, the amount of data in that
Dictionary file is growing, not shrinking or changing. There comes a point
where the amount of data (invalid + valid data) in the Dictionary file fills
up to the size you requested when you first created the file. At that
point, the Dictionary file will physically move to another location on flash
and copy all the previous valid key-values over, leaving all the old
key-value pairs behind (to be deleted by the garbage collector when the time
is right).
So... if you call getDataLength(..) on your dictionary file, you'll see the
size is changing all over the place as you add or update keys - or maybe it
changes only after you've closed and opened the file again. However it does
That explains. I was expecting the size of only valid values/key-values, since that is actually the logical written data length, whereas what's being returned is actually the physical written data length. I understand that the physical written data included exactly keys and values - no meta information is written to that reserved space, and is therefore not included in the return value of getDataLength. Please correct me if I'm wrong.
it, a better indicator of Dictionary file size is getReservedLength(..),
which tells you the dictionary file's footprint on flash. If you were to
copy a complete dictionary file from flash to your computer, you'd want to
copy its entire footprint, which is getReservedLength(..). The reserved
Not necessarily. The reserved locations not yet written to contain alien data (or maybe FF's), and once the file is out of Blackbook, there's no way to determine the end of "real" data and start of alien data. This may be a trivial issue for ASCII files being manually interpreted back at the computer, but I'm using the file to form network packets.
length should remain constant throughout the file's lifetime, as I said.
But it could be much bigger than the amount of data actually written to the
dictionary file.
Also, when you call Bdictionary.open, you get an opened() event with
uint32_t totalSize, uint32_t remainingBytes. These two variables give you a
snapshot of the state of the Dictionary file, but it's only valid at the
time you open the Dictionary file. The amount of actual data in the
Dictionary file should be (totalSize - remainingBytes).
This seems an accurate value. For now, I can work with the limitation that it's accurate only at the time of open call. I have another question here - is totalSize the ceil-to-page-multiple value of reservedLength or it's the same as minimumSize at creation (and correspondingly for remainingBytes)? The difference will be same either way, so I can still use it.
Back to the minimumSize parameter on open... Because the Dictionary file's
footprint remains constant throughout its lifetime, that means that the
minimumSize parameter in Bdictionary.open(..) gets completely ignored if
it's not the first time you've opened that specific Dictionary file. The
minimumSize parameter is only accessed the first time you open a Dictionary
file.
Repeating my question, is this the behavior for non-dictionary files also?
The larger you make the minimumSize value when you call Bdictionary.open for
the first time, then A) you'll be able to store more and larger key-value
pairs in your file, and B) it's going to take more time and energy to insert
and retrieve key-value pairs as your Dictionary file fills up.
And yes, the only real way to count the number of keys is to actually
traverse the key list like you mentioned. If Blackbook provided that
functionality itself, that's exactly what it would do. And since that's not
needed in all applications, it's better to save ROM space for the apps that
don't need it and only implement it at a higher level above Blackbook for
the apps that do need it.
If the open-time (totalSize - remainingBytes) doesn't work good enough, I'll implement this. A more efficient scheme that occurs to me is to keep a RAM counter throughout the open-time of the file that keeps updating, and is initialized at open either through the traversal or through it's last value stored in flash metadata. The latter would be messy at application level, but easier (my guess) at Blackbook level.
I hope that clears things up and I also hope Blackbook is working out for
you. Let me know if you have any more questions
Alot! And Blackbook is working, yeah. It's taken off a lot of development time, thanks! And thanks for your detailed reply.
Manu.
-David
-----Original Message-----
From: Manu Bansal [mailto: [EMAIL PROTECTED]]
Sent: Monday, September 18, 2006 11:50 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Blackbook5: dataLength, minimumSize for dictionary files
Dear Mr. Moss,
I am an undergraduate student at Indian Institute of Technology Kanpur (IIT
Kanpur), working on a sensor network project and using Blackbook5. There's a
question to which I couldn't find an answer in the included pdf or the
interface inline comments. I couldn't get any help from the tinyos-help list
either, hence a personal mail.
In Blackbook5, what does BDictionary.getDataLength return? Is it the length
of
data inserted as values, or keys and values both, or other? And similarly,
what should minimumSize include? For regular files, the semantics is clear,
but I'm not sure about dictionary files, and getDataLength doesn't return
the
expected value. For 5 key-value pairs each having value 12B long, the
returned dataLength is 1D/1E/14 on different occasions. Being able to obtain
the no. of entries in a dictionary file will also solve my problem. The only
way I can think to do this is to traverse the key-link-list.
With warm regards and thanks,
Manu Bansal.
--
Manu Kumar Bansal
Fourth Year Undergraduate
B.Tech-M.Tech Dual Degree
Computer Science and Engineering
IIT Kanpur, Kanpur, India - 208016
http://home.iitk.ac.in/~manub
_______________________________________________
Tinyos-help mailing list
[email protected]
https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
_______________________________________________ Tinyos-help mailing list [email protected] https://mail.millennium.berkeley.edu/cgi-bin/mailman/listinfo/tinyos-help
