Hello Tim!
Thank you for your interest in this.
(I am replying in the list for the moment, in case anyone else is
interested...)
As far as I know, no effort has been taken on this.
I will be glad to help you porting the settings storage to NuttX.
The published repo is not maintained, so I will update it later today with
all the latest
changes/fixes in this specific app you are interested in.
---
License-wise, this is 100% original work by me, so I guess there is nothing
to concert us.
It can be committed directly to the apps repo under Apache license.
---
Regarding the parsers.
Currently it supports two parsers ("storages" as they are refered in code):
binary and text.
The binary storage, directly dumps the RAM contents in a file.
This is speed-efficient, but may waste a bit of space, and it is not very
portable.
For example, a save made in a little-endian machine may not work in a
big-endian one.
Similarly, the read and write implementations need to have the exact same
cofniguration (e.g. key size).
I typically use this to store settings directly in the MCU's Flash memory.
So, none of the above limitations matter.
The text storage solves all of the above and makes the data human-readable,
in expense of some
more CPU time.
I typically use the text storage in an SD card file.
More specifically, I usually use **both** storages at the same time.
The text one is human-readable and easy to update (e.g. get the SD on your
computer, and change anything you want),
while the binary one is there for speed and reliability (SD card
failed/removed, the app can still access the settings).
The settings library can reliably syncrhonize multiple storages together,
and make them act as one.
---
My intention was to also add XML and JSON (and I have provided two parser
implementations), but I never found the time.
I guess adding YAML will be fairly trivial.
I am willing to help add any new parsers that may be needed.
---
So, shall we create an issue in apps repo, to cooperate development?
On Mon, Oct 30, 2023 at 10:10 AM alin.jerpe...@sony.com <
alin.jerpe...@sony.com> wrote:
> Hi Tim,
>
> I am sorry but it slipped by agenda.
> Thanks for looking at this topic. Please take your time and submit when
> you are ready.
>
> Best regards
> Alin
>
>
> From: Tim Hardisty <timhardist...@gmail.com>
> Sent: den 28 oktober 2023 12:10
> To: dev@nuttx.apache.org
> Subject: Re: Code donation
>
> Hi, Did anyone else make any progress with porting your settings code to
> NuttX Apps? It's taken a while but I am at long last looking to add a
> settings function to my new app and this still looks like it'll fit the
> bill nicely. If not, I will
> ZjQcmQRYFpfptBannerStart
> This message is from an unknown sender
> You have not previously corresponded with this sender and they are
> external to Sony
> ZjQcmQRYFpfptBannerEnd
>
> Hi,
>
>
>
> Did anyone else make any progress with porting your settings code to
>
> NuttX Apps? It's taken a while but I am at long last looking to add a
>
> settings function to my new app and this still looks like it'll fit the
>
> bill nicely.
>
>
>
> If not, I will set it up as an app, make sure it works, and submit it
>
> for review hopefully in the next few weeks. I won't add YAML as yet,
>
> just binary - my current think is to use a YAML parser between the user
>
> and this function rather than storing YAML directly, but not 100%
>
> decided yet.
>
>
>
> On 05/12/2022 11:22, Fotis Panagiotopoulos wrote:
>
> > Hello!
>
> >
>
> > Thank you for your interest in this!
>
> >
>
> > As requested, I created a temporal repository and uploaded all the code
>
> > there.
>
> >
>
> > https://github.com/fjpanag/code_for_nuttx<
> https://github.com/fjpanag/code_for_nuttx>
>
> >
>
> > I have added a NOTICE file in each subdir, with some basic information.
>
> > Please contact me before starting any work, to provide you with help and
>
> > coordinate the effort.
>
> > Better yet, you may open a Github issue for the piece of software that
> you
>
> > would like to help porting.
>
> >
>
> >
>
> > Alan,
>
> > I would love to cooperate with you in this effort.
>
> > Pick your target!
>
> >
>
> >
>
> > Alin,
>
> > I would really like to see my MQTT broker become part of NuttX.
>
> >
>
> > Alternatively, you can have a look at the XML or JSON parsers, which are
> of
>
> > general usefulness,
>
> > and the porting effort would be very very minimal.
>
> >
>
> >
>
> > Tim,
>
> > Thank you for your interest in the settings storage.
>
> > It is a very useful and versatile piece of software. In fact, I have used
>
> > it in all my projects over the last 5-6 years.
>
> >
>
> > I also thought about YAML before, but never got to actually implementing
>
> > this.
>
> >
>
> > I suggest you first add a YAML parser to NuttX (maybe similarly to JSON
> and
>
> > XML, if they get accepted),
>
> > and then use it in the settings storage. It would be best not to couple
>
> > these two softwares so someone
>
> > may use one without the other. But, nevertheless, it is your call...
>
> >
>
> > If you provide me with a YAML parser, I believe that I can develop for
> you
>
> > a new settings storage that uses this parser.
>
> >
>
> > Fotis
>
> >
>
> > On Mon, Dec 5, 2022 at 12:12 AM Tim Hardisty<t...@jti.uk.com
> .invalid<mailto:t...@jti.uk.com.invalid>>
>
> > wrote:
>
> >
>
> >> I have interest in your settings storage, with the probability of adding
>
> >> yaml output since I was going to do something similar for my current
>
> >> project (once I get over the pain of getting NuttX fixed for the arch
> I'm
>
> >> using).
>
> >>
>
> >> As have been suggested, maybe push it to github and I/we can clone
> what's
>
> >> of interest, see if it makes sense, then get it working right for NuttX
>
> >> and
>
> >> do a PR...in the fullness of time (i.e. I'm a slow worker!)
>
> >>
>
> >> On 04/12/2022, 16:55, "Fotis Panagiotopoulos"<f.j.pa...@gmail.com
> <mailto:f.j.pa...@gmail.com>>
>
> >> wrote:
>
> >>
>
> >> Hello everyone!
>
> >>
>
> >> Christmas arrived a bit earlier for NuttX as I would like to donate
>
> >> some of
>
> >> my personal code to the community!
>
> >>
>
> >> A bit of context.
>
> >> Over the years that I am working on embedded systems, I have
> developed
>
> >> lots
>
> >> of software that I use in my projects.
>
> >> Some of it is quite general-purpose, or useful for other
> applications,
>
> >> and
>
> >> I have found my self reusing it
>
> >> quite often. In fact, there are some things that I use in
> practically
>
> >> all
>
> >> firmwares that I have developed over
>
> >> the last years.
>
> >>
>
> >> I always wanted to open-source this software so other people can
>
> >> benefit
>
> >> from it.
>
> >> But I never managed to do so. Open-sourcing needs some effort, the
>
> >> software
>
> >> needs maintenance, documentation
>
> >> and support, and most importantly in most cases a "porting layer"
>
> >> needs to
>
> >> be developed.
>
> >> Last but not least, every project needs a bit of "marketing" and
>
> >> "advertising" so others can learn about
>
> >> your work and use it.
>
> >>
>
> >> For the last couple of years I have been using NuttX a lot, and I
> have
>
> >> ported most of the aforementioned software
>
> >> to NuttX. I believe that NuttX and its community are perfect for
> me to
>
> >> publish my code, instead of creating
>
> >> a ton of small repos, of questionable usefulness and increasing my
>
> >> workload
>
> >> considerably.
>
> >>
>
> >> It is very important that I can get immediate feedback from the
>
> >> community,
>
> >> learn what people are actually
>
> >> interested in (instead of investing on software that no one needs),
>
> >> and
>
> >> provide actual and *working*
>
> >> samples of the code (as NuttX already supports a ton of different
>
> >> boards
>
> >> and arches).
>
> >>
>
> >> Using POSIX as the porting layer is also awesome.
>
> >>
>
> >> That being said, my free time is still exceptionally limited and I
>
> >> cannot
>
> >> do this myself.
>
> >> I still need the help of the community, and most importantly I
> need to
>
> >> see
>
> >> interest in a piece of
>
> >> software before putting any work on it.
>
> >>
>
> >> So, what I offer:
>
> >> * I offer various codes, fully featured, production ready and
> tested.
>
> >> * All code will be offered for free (of course) and under Apache
>
> >> licensing.
>
> >> * I will provide support to those working on these codes, to my
> best
>
> >> ability.
>
> >> * I will contribute to testing everything integrated to NuttX, as
>
> >> hardware
>
> >> availability allows me.
>
> >> * I will do some licensing check, to ensure code is 100% original
> and
>
> >> mine,
>
> >> or state the licenses of the projects I borrowed code from.
>
> >>
>
> >> What I ask for:
>
> >> I need people that are interested in each of these codes to
> integrate
>
> >> them
>
> >> into NuttX apps.
>
> >> You just have to pick what it is interesting to you, contact me to
>
> >> provide
>
> >> you with the code,
>
> >> and integrate it to NuttX. You will need to:
>
> >> * Add the code into the NuttX apps repo, and ensure Kconfig and the
>
> >> build
>
> >> system use the code properly (should be trivial).
>
> >> * Adapt the file format and the coding style to the NuttX one (this
>
> >> may
>
> >> need some work, but it can also be automated).
>
> >> * Provide an example app, something that someone can run to use or
>
> >> demo the
>
> >> new code.
>
> >> * Test and verify the example app on actual hardware (I may be
> able to
>
> >> cross-check it on my hardware too).
>
> >>
>
> >> The code that I offer (for the moment):
>
> >>
>
> >>
>
> >> *** Lua v5.2.4 ***
>
> >> I know that there is already a Lua app for NuttX.
>
> >> But for anyone using it, it may be beneficial to use my work.
>
> >>
>
> >> First and foremost, I have ported the eLua LTR patch to Lua 5.2.
> This
>
> >> patch
>
> >> dramatically reduces the memory usage of Lua.
>
> >> In fact, I found out that it is crucial to have this patch enabled
> for
>
> >> any
>
> >> actual real-life usage of Lua on any "normal" MCU.
>
> >>
>
> >> I have created a Kconfig for all Lua configurations, so it can
>
> >> integrate
>
> >> with NuttX better.
>
> >>
>
> >> I have also made some other minor changes to the code that might be
>
> >> interesting for you.
>
> >> For example there is a simplistic sandboxing option.
>
> >>
>
> >>
>
> >> *** MQTT Broker ***
>
> >> Yes, a full-blown, spec-compliant MQTT Broker!
>
> >> To my knowledge there is no other open-source and portable MQTT
> broker
>
> >> for
>
> >> embedded systems.
>
> >>
>
> >> It follows the MQTT v3.1.1 specification as closely as possible. I
>
> >> think
>
> >> there is only one violation, needed due to its embedded nature,
>
> >> but in all practical cases you may consider it fully compliant.
>
> >>
>
> >> It has been tested with dozens of devices, and it performs greatly.
>
> >> There are a couple of things that may need to be improved, but are
>
> >> trivial,
>
> >> and will not affect the normal use of the software.
>
> >>
>
> >> I know that such a broker may not be your best option for a proper
> and
>
> >> large installation of IoT devices, but it is exceptionally useful
>
> >> for at least the following cases:
>
> >> * You have only a few devices, isolated (no internet), that you
> need
>
> >> to
>
> >> connect, and you want to avoid the cost (and maintenance) of a
> proper
>
> >> broker (e.g. Raspberry Pi).
>
> >> * You need to directly communicate with a device that only supports
>
> >> MQTT.
>
> >> Instead of going through an external server, you run the server
>
> >> locally,
>
> >> and communicate with the device directly.
>
> >>
>
> >>
>
> >> *** MQTT Client ***
>
> >> A production-tested, robust and quite flexible MQTT client.
>
> >>
>
> >> I know that there are plenty of such clients available out there,
> but
>
> >> here
>
> >> is another one.
>
> >>
>
> >> Back in the day I tried to use the Eclipse Paho library. I found
> it to
>
> >> be a
>
> >> horrible piece of software. Crashes, buffer overflows, spec
>
> >> violations,
>
> >> missing functionality and more.
>
> >> I don't know whether it has improved now, but back then I
> rolled-out
>
> >> my own
>
> >> implementation. There were not many other alternatives available,
> and
>
> >> Paho
>
> >> did not worth contributing to it.
>
> >> It needed so much work, that starting from scratch was much easier.
>
> >>
>
> >> Then, when I started using NuttX, I saw support for MQTT-C. Well, I
>
> >> tried
>
> >> it and I wasn't greatly satisfied (I don't remember the reasons).
>
> >> So I decided to keep using my own (and well tested) implementation.
>
> >>
>
> >> So NuttX, instead of relying on an external project, now can have
> its
>
> >> own
>
> >> client.
>
> >>
>
> >>
>
> >> *** Settings Storage ***
>
> >> This is probably my favorite.
>
> >> A key-value pairs storage for non-volatile settings or
> configurations.
>
> >> Very needed when you need to adjust the system's operation in
>
> >> run-time,
>
> >> download parameters etc.
>
> >>
>
> >> It acts in two layers: the settings API and the actual storages
> used.
>
> >>
>
> >> The settings can store or retrieve key-value pairs.
>
> >> Type casts are supported and are performed automatically. For
> example
>
> >> you
>
> >> can store the string "true" and then read it as the int 1 etc.
>
> >> There is support for caching writes (minimizing overhead and
> reducing
>
> >> wear
>
> >> on the actual storage medium).
>
> >> There are signal notifications when a setting value is changed.
>
> >>
>
> >> The bottom layer, the storages, is responsible for actually reading
>
> >> and
>
> >> writing the settings map to the physical mediums.
>
> >> Every storage can also format the data according to its type.
>
> >> For the moment there are only ASCII and binary storages, but it
> would
>
> >> be
>
> >> very easy to expand this to JSON, XML, YAML and more.
>
> >> There is support for multiple storages used simultaneously, and
>
> >> synchronization between them (for example I usually use both the
> MCU's
>
> >> Flash and the SD card).
>
> >>
>
> >>
>
> >> *** FTP Client ***
>
> >> This is a client that I had developed before I learned about NuttX.
>
> >> I also tried NuttX FTPc, but I experienced may failures that I
> never
>
> >> got
>
> >> the time to troubleshoot. I just went back to the tried solution.
>
> >> One thing that I didn't really like in NuttX ftpc, it was that it
> is
>
> >> very
>
> >> taciturn. When something went wrong, you couldn't understand what
> the
>
> >> issue
>
> >> is.
>
> >> The error information of this library is a bit better.
>
> >>
>
> >> If an alternative ftpc implementation is of interest to anyone, the
>
> >> code is
>
> >> available.
>
> >>
>
> >>
>
> >> *** JSON Parser ***
>
> >> This is a very minimal parse-only JSON implementation.
>
> >>
>
> >> The motivation behind developing this was memory consumption.
>
> >> This parser can open an arbitrarily large JSON file (even MBs),
> using
>
> >> only
>
> >> a few bytes of RAM!
>
> >> (If I recall correctly only 20 bytes!)
>
> >>
>
> >> Although there are other options there for you, if you need the
>
> >> smallest
>
> >> possible memory footprint, this is your parser.
>
> >>
>
> >>
>
> >> *** XML Parser ***
>
> >> Exactly as above with JSON.
>
> >> A minimal parser, that can open any file using just a few bytes of
>
> >> RAM.
>
> >>
>
> >> If you need something simple or you have memory constraints, use
> this.
>
> >>
>
> >>
>
> >> *** XTEA ***
>
> >> Simple XTEA encryption and decryption.
>
> >> Very useful when you need a quite fast way to encrypt (or obscure)
>
> >> some
>
> >> data.
>
> >> (Need to check licensing!)
>
> >>
>
> >>
>
> >> *** Sun Calculations ***
>
> >> A small library that can do astronomical calculations for the sun.
>
> >> That is sunrise time, sunset, day duration and more.
>
> >>
>
> >>
>
> >> *** Geolocation ***
>
> >> I used the ipgeolocation.io API and NuttX wget to get geolocation
>
> >> information about the device.
>
> >> Nothing special, but it may serve as a working example, or even
>
> >> inspiration
>
> >> for others...
>
> >>
>
> >>
>
> >> If everyone is interested in any of the above please contact me.
>
> >>
>
> >> *PLEASE be responsible, and respectful.* (I am sure you will! :D )
>
> >> If you request for a piece of software* you will have to help to
> get
>
> >> this
>
> >> into NuttX, and not only use it for your own personal gain*.
>
> >>
>
> >>
>
> >>
>
