Hi everyone,
as some of you may know, I'm currently writing my thesis on "designing a
storage abstraction layer for the IoT" which includes writing an implementation
of the concept for RIOT. Up until now, I've been working mostly on my own, and
while I think I have a good understanding of what I'm trying to achieve, I
don't have a lot of experience designing (storage) APIs for embedded devices. I
thought that some of you could provide helpful input, or point me towards
flaws/bad pracices in my concept that I'm not able to see. As such, I would
like to present my train of thought for my concept as well as an outline of my
current ideas regarding API design, and I would welcome your thoughts and input!
I'm not planning to go into much detail into how things will be implemented
here, but please let me know if you have any questions or doubts about the
feasibility/usefulness of anything!
The initial idea [^1] was to start from the POSIX file system API and go from
there. A common file system API provides you with an unmanaged sequence of
bytes (files), will let you create directories, control accessibility of a file
depending on user privileges, create hard/soft links and more. My research
suggests that a large portion of that functionality should not be the focus
when implementing a data storage system for RIOT.
From the sensor network storage systems I've studied so far (Matchbox, ELF,
Coffee/Antelope, Capsule), the consensus seems to be the focus on storing
streams of immutable data (sensor values), possibly indexed by time.Mutability
is also supported in more recent systems, but always under the assumptions that
mutating already written data will consume significantly more memory due to the
nature of flash memory (i.e. write once/block erase).
In my opinion, a solution that provides a higher-level abstraction compared to
"files" should be developed for RIOT. My plan is to follow a similar approach
to Capsule [^2] and implement an object-based storage systems. The general idea
is that the system provides a number of different storage objects, e.g. an
indexed stream, an array or a queue. For example (prefixed `wip` because it
doesn't have a name yet):
wip_fd sensordata = wip_stream("sensor:humidity", sizeof(uint64_t));
wip_stream_append(&sensordata, 12345);
// Averaging a stream
uint64_t sum = 0;
wip_iter iter = wip_iterator(&sensordata);
while(uint64_t value = WIP_NEXT(uint64_t, iter)) {
sum += value;
}
...
This would allow a more declarative usage of storage when developing an
application for RIOT. Processing a large sequence of data without loading all
of it into memory would be supported via iterators. My initial idea was to use
callbacks and functions like "fold" and "foreach" instead, but this turned out
to be pretty clumsy in C. Note that, while my concept does not involve the
concept of "directories", one could implement storage object namespaces based a
separator in the object name, e.g. `sensor` in the above example.
Swapping out large data structures also comes to mind (e.g. a RPL routing table
in storing mode). ICNs may also profit from this. Storing mutable, unmanaged
byte sequences (i.e. files) would also be supported.
wip_fd arr = wip_array("core:routingtable", sizeof(rt_entry));
rt_entry x = WIP_ARRAY_GET(arr, rt_entry, 42);
x.metric += 1
WIP_ARRAY_SET(arr, x, 42);
I'm aware that it'd be possible to implement all this on top of the POSIX file
system API, but I personally think that having a higher level of abstraction
that covers the common use cases is preferable to placing the burden on the
application developer. In any case, the storage system I'm envisioning will be
detached from the lower level flash storage abstraction, so it'd be possible to
develop other storage layers on top of it. I've already extracted Coffee (from
Contiki) from their soruce tree [^3], but may not have the time to port it to
RIOT.
Looking forward to your feedback!
Cheers,
Lucas
[^1]: https://github.com/RIOT-OS/RIOT/pull/2474
[^2]: http://dl.acm.org/citation.cfm?id=1182827&CFID=566423331&CFTOKEN=15751545
[^3]: https://github.com/x3ro/coffee-fs
_______________________________________________
devel mailing list
devel@riot-os.org
https://lists.riot-os.org/mailman/listinfo/devel
