Re: Seperating firmware-utils into seperate repo

2022-04-12 Thread Joseph Mullally
> Most in-house OpenWrt packages are actually stored in their
> own git repo, see https://git.openwrt.org/ down in the
> /project/something_something.git
>
>  -Alberto

Thanks for the good suggestions Alberto. I still think it will be a
messy process for most new device contributors (where a lot of these
PRs will come from) and reviewers alike, as Adrian points out in the
"post-merge" peer review discussion above.

Overall I can't see what the real benefits of this change are. Unless
it really is official OpenWRT policy to release these as generic
firmware tools for use by other projects, and it absolutely has to be
a separate git repo. On many levels that's a good idea but also
nebulous. E.g. for the xiaomifw vacuum example, why not put it in its
own seperate upstream repo.

Anyway, maybe people disagree and are OK with this.

At a minimum to finish out this package split, someone needs to do this?
* Create a GitHub mirror for https://git.openwrt.org/project/firmware-utils.git

To avoid chaos in the future (or even now with 22.03, bugfixes, backports etc):
* Release branches for "firmware-utils.git" matching "openwrt.git"
(unless some versioning strategy like SEMVER is planned, but the
OpenWRT release branches seem the simplest)

Thanks,
- Joe

___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel


Re: Seperating firmware-utils into seperate repo

2022-03-15 Thread Alberto Bursi




On 14/03/22 01:56, Joseph Mullally wrote:

Hi,

firmware-utils was separated from openwrt.git into its own repository
a few months ago:
https://git.openwrt.org/?p=project/firmware-utils.git
https://github.com/openwrt/openwrt/commit/8cc9a74a3f6bf363645efda6db417f8dadd3d844

If it's going to stay separate, it looks like these changes are still needed?
- Release branches matching the main openwrt.git repo. (E.g. To
facilitate firmware bugfixes on older OpenWRT release after any tool
APIs in /master have changed)
- A GitHub mirror (like everything here https://openwrt.org/submitting-patches )
- Update the developer guide with a workflow for adding and testing a
new device involving changes to firmware-utils.git and openwrt.git.
(This is tricky. When recently adding a new device, I built the
changes in openwrt.git once, then manually rebuilt an updated
firmware-util binary, then rebuilt the openwrt.git firmware image. Is
there a proper building the main package and changes to package repos
at the same time? There is also the can of worms w.r.t. people
submitting changes and testing, likely it will need to be handled with
2 concurrent Pull Requests kept open until everything is 100% ready,
in addition to the committers bump of the firmware-utils dependency).


Most in-house OpenWrt packages are actually stored in their own git 
repo, see https://git.openwrt.org/ down in the 
/project/something_something.git


Afaik a viable strategy for doing development on those packages is doing 
your development as patches that exist in the build system only, which 
is mostly automated with the quilt tool, as explained here 
https://openwrt.org/docs/guide-developer/toolchain/use-patches-with-buildsystem


Then when you are ready you need to copy the changes to a clone of the 
tool's git repo and then make proper commits and send them with 
git-email from mailing list.


If you really need to do A LOT of development on those packages you 
probably want to clone the git repo from openwrt server and put it on 
another git server (or in github, which is also a git server) and just 
edit package makefile in the local openwrt build system to use your 
development repo to get that package's source instead.



-Alberto

___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel


Seperating firmware-utils into seperate repo

2022-03-13 Thread Joseph Mullally
Hi,

firmware-utils was separated from openwrt.git into its own repository
a few months ago:
https://git.openwrt.org/?p=project/firmware-utils.git
https://github.com/openwrt/openwrt/commit/8cc9a74a3f6bf363645efda6db417f8dadd3d844

If it's going to stay separate, it looks like these changes are still needed?
- Release branches matching the main openwrt.git repo. (E.g. To
facilitate firmware bugfixes on older OpenWRT release after any tool
APIs in /master have changed)
- A GitHub mirror (like everything here https://openwrt.org/submitting-patches )
- Update the developer guide with a workflow for adding and testing a
new device involving changes to firmware-utils.git and openwrt.git.
(This is tricky. When recently adding a new device, I built the
changes in openwrt.git once, then manually rebuilt an updated
firmware-util binary, then rebuilt the openwrt.git firmware image. Is
there a proper building the main package and changes to package repos
at the same time? There is also the can of worms w.r.t. people
submitting changes and testing, likely it will need to be handled with
2 concurrent Pull Requests kept open until everything is 100% ready,
in addition to the committers bump of the firmware-utils dependency).

Possibly, but involves a lot of work and complexity:
- Separating the per-device configuration from the C code into CLI
arguments for the image Makefiles (or json files?), so that most new
devices involving them can go back to being added with one atomic
commit to openwrt.git. (see list below, and caveat)

Re: Splitting firmware-utils out so it can be consumed by non-OpenWRT projects
* This seems like the only practical benefit? Specifically, that this
folder can now be consumed directly as a git submodule or package by
another project, instead of copying from
openwrt.git/tools/firmware-utils like in this existing example:
https://www.freshports.org/devel/firmware-utils/
https://www.transit.hanse.de/mirror/svn.openwrt.org/firmware-utils/
* Vending to 3rd parties implies having a versioning policy to keep
the API stable for all tools after branching from /master to release
branches (i.e. API changes only allowed in /master, and the repo
release branches would be equivalent to Semantic Versioning major
releases). (The planned image tests would also cover this)
* Also, will it decrease the benefit to 3rd parties if all the
model-specific image configuration was separated from these tools and
put back into openwrt.git, and how much of that config should stay in
one repo or the other?

Re: Splitting firmware-utils out so it can be covered under CI/CD tests
* Is there a technical reason for this? (e.g. why not re-run tests on
every openwrt.git commit, or just on changes to
openwrt.git/tools/firmware-utils ?)

Re: Seperating the per-device parameters (openwrt.git) from the base
firmware tool (firmware-utils.git)
* These are the affected files:
  addpattern.c
  avm-wasp-checksum.c
  iptime-crc32.c
  iptime-naspkg.c
  mkcasfw.c
  mkcsysimg.c
  mkfwimage.c
  mkheader_gemtek.c
  mkmerakifw.c
  mkmerakifw-old.c
  mkmylofw.c
  mkplanexfw.c
  mkporayfw.c
  mktplinkfw2.c
  mkzcfw.c
  mkzynfw.c
  motorola-bin.c
  tplink-safeloader.c
  xiaomifw.c
  zynos.h

Overall, it seems debatable as to whether this particular package
should be split out, or stay in the main repo like before with the
rest of the per-device configuration.

Cheers,
- Joe

___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel