date:20211008

Re: [OE-core] [PATCH] ffmpeg: fix CVE-2021-38114

2021-10-08 Thread Kiran Surendran

Ah yes sorry, tried to use the same patch I made for hardknott, I'll 
send the correct one for 4.4


Regards,

Kiran

On 10/8/21 5:12 PM, Richard Purdie wrote:

[Please note: This e-mail is from an EXTERNAL e-mail address]

On Fri, 2021-10-08 at 09:48 -0700, Kiran Surendran wrote:

backport from upstream

Signed-off-by: Kiran Surendran 
---
  .../ffmpeg/ffmpeg/fix-CVE-2021-38114.patch| 67 +++
  .../recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb |  3 +-
  2 files changed, 69 insertions(+), 1 deletion(-)
  create mode 100644 
meta/recipes-multimedia/ffmpeg/ffmpeg/fix-CVE-2021-38114.patch

Master has 4.4 so this doesnt apply? Which release was this targeted at?

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156787): 
https://lists.openembedded.org/g/openembedded-core/message/156787
Mute This Topic: https://lists.openembedded.org/mt/86175257/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] ptest-runner: install -collect-system-data script

2021-10-08 Thread Tim Orling

When ptest-runner times out or otherwise fails, it tries to
call ptest-runner-collect-system-data, so install the script.

The script currently calls dmesg, df, free (which are provided
by busybox, etc.) and pstree (which is a sub-package of psmisc).

Add pstree as an RDEPENDS.

Signed-off-by: Tim Orling 
---
 meta/recipes-support/ptest-runner/ptest-runner_2.4.2.bb | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-support/ptest-runner/ptest-runner_2.4.2.bb 
b/meta/recipes-support/ptest-runner/ptest-runner_2.4.2.bb
index 1d3c24a177..23ab48ba2b 100644
--- a/meta/recipes-support/ptest-runner/ptest-runner_2.4.2.bb
+++ b/meta/recipes-support/ptest-runner/ptest-runner_2.4.2.bb
@@ -15,7 +15,7 @@ SRC_URI = "git://git.yoctoproject.org/ptest-runner2 \
 
 S = "${WORKDIR}/git"
 
-FILES:${PN} = "${bindir}/ptest-runner"
+FILES:${PN} = "${bindir}/ptest-runner 
${bindir}/ptest-runner-collect-system-data"
 
 EXTRA_OEMAKE = "-e MAKEFLAGS= CFLAGS="${CFLAGS} 
-DDEFAULT_DIRECTORY=\\\"${libdir}\\\"""
 
@@ -25,6 +25,10 @@ do_compile () {
 
 do_install () {
install -D -m 0755 ${S}/ptest-runner ${D}${bindir}/ptest-runner
+   install -D -m 0755 ${S}/ptest-runner-collect-system-data 
${D}${bindir}/ptest-runner-collect-system-data
 }
 
 RDEPENDS:${PN}:append:libc-glibc = " libgcc"
+
+# pstree is called by ptest-runner-collect-system-data
+RDEPENDS:${PN}:append = " pstree"
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156786): 
https://lists.openembedded.org/g/openembedded-core/message/156786
Mute This Topic: https://lists.openembedded.org/mt/86182192/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] ffmpeg: fix CVE-2021-38114

2021-10-08 Thread Richard Purdie

On Fri, 2021-10-08 at 09:48 -0700, Kiran Surendran wrote:
> backport from upstream
> 
> Signed-off-by: Kiran Surendran 
> ---
>  .../ffmpeg/ffmpeg/fix-CVE-2021-38114.patch| 67 +++
>  .../recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb |  3 +-
>  2 files changed, 69 insertions(+), 1 deletion(-)
>  create mode 100644 
> meta/recipes-multimedia/ffmpeg/ffmpeg/fix-CVE-2021-38114.patch

Master has 4.4 so this doesnt apply? Which release was this targeted at?

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156785): 
https://lists.openembedded.org/g/openembedded-core/message/156785
Mute This Topic: https://lists.openembedded.org/mt/86175257/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-Core][dunfell][PATCH] binutils: Fix a missing break in case statement

2021-10-08 Thread Christian Eggers

This was missed during patch forward porting
its only effective when printing options

Signed-off-by: Khem Raj 
Signed-off-by: Richard Purdie 
---
 ...-system-directories-when-cross-linki.patch | 26 +--
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git 
a/meta/recipes-devtools/binutils/binutils/0009-warn-for-uses-of-system-directories-when-cross-linki.patch
 
b/meta/recipes-devtools/binutils/binutils/0009-warn-for-uses-of-system-directories-when-cross-linki.patch
index 11a8110d40..88cce49e46 100644
--- 
a/meta/recipes-devtools/binutils/binutils/0009-warn-for-uses-of-system-directories-when-cross-linki.patch
+++ 
b/meta/recipes-devtools/binutils/binutils/0009-warn-for-uses-of-system-directories-when-cross-linki.patch
@@ -1,4 +1,4 @@
-From 7b24f81e04c9d00d96de7dbd250beade6d2c6e44 Mon Sep 17 00:00:00 2001
+From 12b658c0fe5771d16067baef933b7f34ed455def Mon Sep 17 00:00:00 2001
 From: Khem Raj 
 Date: Fri, 15 Jan 2016 06:31:09 +
 Subject: [PATCH] warn for uses of system directories when cross linking
@@ -59,8 +59,8 @@ Signed-off-by: Khem Raj 
  ld/ldfile.c | 17 +
  ld/ldlex.h  |  2 ++
  ld/ldmain.c |  2 ++
- ld/lexsup.c | 15 +++
- 9 files changed, 85 insertions(+)
+ ld/lexsup.c | 16 
+ 9 files changed, 86 insertions(+)
 
 diff --git a/ld/config.in b/ld/config.in
 index d93c9b0830..5da2742bea 100644
@@ -77,10 +77,10 @@ index d93c9b0830..5da2742bea 100644
  #undef EXTRA_SHLIB_EXTENSION
  
 diff --git a/ld/configure b/ld/configure
-index 811134a503..f8c17c19ae 100755
+index f432f4637d..a9da3c115e 100755
 --- a/ld/configure
 +++ b/ld/configure
-@@ -826,6 +826,7 @@ with_lib_path
+@@ -830,6 +830,7 @@ with_lib_path
  enable_targets
  enable_64_bit_bfd
  with_sysroot
@@ -88,7 +88,7 @@ index 811134a503..f8c17c19ae 100755
  enable_gold
  enable_got
  enable_compressed_debug_sections
-@@ -1491,6 +1492,8 @@ Optional Features:
+@@ -1495,6 +1496,8 @@ Optional Features:
--disable-largefile omit support for large files
--enable-targetsalternative target configurations
--enable-64-bit-bfd 64-bit support (on hosts with narrower word sizes)
@@ -97,7 +97,7 @@ index 811134a503..f8c17c19ae 100755
--enable-gold[=ARG] build gold [ARG={default,yes,no}]
--enable-got= GOT handling scheme (target, single, negative,
multigot)
-@@ -15788,6 +15791,19 @@ fi
+@@ -16624,6 +16627,19 @@ fi
  
  
  
@@ -222,10 +222,10 @@ index 5287f19a7f..55096e4fc9 100644
  
  /* The initial parser states.  */
 diff --git a/ld/ldmain.c b/ld/ldmain.c
-index da1ad17763..12d0b07d8a 100644
+index c4af10f4e9..95b56b2d2d 100644
 --- a/ld/ldmain.c
 +++ b/ld/ldmain.c
-@@ -274,6 +274,8 @@ main (int argc, char **argv)
+@@ -273,6 +273,8 @@ main (int argc, char **argv)
command_line.warn_mismatch = TRUE;
command_line.warn_search_mismatch = TRUE;
command_line.check_section_addresses = -1;
@@ -235,7 +235,7 @@ index da1ad17763..12d0b07d8a 100644
/* We initialize DEMANGLING based on the environment variable
   COLLECT_NO_DEMANGLE.  The gcc collect2 program will demangle the
 diff --git a/ld/lexsup.c b/ld/lexsup.c
-index 3d15cc491d..0e8b4f2b7a 100644
+index 3d15cc491d..6478821443 100644
 --- a/ld/lexsup.c
 +++ b/ld/lexsup.c
 @@ -550,6 +550,14 @@ static const struct ld_option ld_options[] =
@@ -253,10 +253,10 @@ index 3d15cc491d..0e8b4f2b7a 100644
  };
  
  #define OPTION_COUNT ARRAY_SIZE (ld_options)
-@@ -1603,6 +1611,13 @@ parse_args (unsigned argc, char **argv)
- 
+@@ -1604,6 +1612,14 @@ parse_args (unsigned argc, char **argv)
case OPTION_PRINT_MAP_DISCARDED:
  config.print_map_discarded = TRUE;
+ break;
 +
 +  case OPTION_NO_POISON_SYSTEM_DIRECTORIES:
 +command_line.poison_system_directories = FALSE;
@@ -264,6 +264,6 @@ index 3d15cc491d..0e8b4f2b7a 100644
 +
 +  case OPTION_ERROR_POISON_SYSTEM_DIRECTORIES:
 +command_line.error_poison_system_directories = TRUE;
- break;
++break;
}
  }
-- 
Christian Eggers
Embedded software developer

Arnold & Richter Cine Technik GmbH & Co. Betriebs KG
Sitz: Muenchen - Registergericht: Amtsgericht Muenchen - Handelsregisternummer: 
HRA 57918
Persoenlich haftender Gesellschafter: Arnold & Richter Cine Technik GmbH
Sitz: Muenchen - Registergericht: Amtsgericht Muenchen - Handelsregisternummer: 
HRB 54477
Geschaeftsfuehrer: Dr. Michael Neuhaeuser; Stephan Schenk; Walter Trauninger; 
Markus Zeiler


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156784): 
https://lists.openembedded.org/g/openembedded-core/message/156784
Mute This Topic: https://lists.openembedded.org/mt/86180878/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] ffmpeg: fix CVE-2021-38114

2021-10-08 Thread Kiran Surendran

backport from upstream

Signed-off-by: Kiran Surendran 
---
 .../ffmpeg/ffmpeg/fix-CVE-2021-38114.patch| 67 +++
 .../recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb |  3 +-
 2 files changed, 69 insertions(+), 1 deletion(-)
 create mode 100644 
meta/recipes-multimedia/ffmpeg/ffmpeg/fix-CVE-2021-38114.patch

diff --git a/meta/recipes-multimedia/ffmpeg/ffmpeg/fix-CVE-2021-38114.patch 
b/meta/recipes-multimedia/ffmpeg/ffmpeg/fix-CVE-2021-38114.patch
new file mode 100644
index 00..3de7cf7e0f
--- /dev/null
+++ b/meta/recipes-multimedia/ffmpeg/ffmpeg/fix-CVE-2021-38114.patch
@@ -0,0 +1,67 @@
+CVE: CVE-2021-38114
+Upstream-Status: Backport
+Signed-off-by: Kiran Surendran 
+
+From 662aef4aacf23b4be4c1cfaebd837e225b357e51 Mon Sep 17 00:00:00 2001
+From: maryam ebr 
+Date: Tue, 3 Aug 2021 01:05:47 -0400
+Subject: [PATCH] avcodec/dnxhddec: check and propagate function return value
+
+Similar to CVE-2013-0868, here return value check for 'init_vlc' is needed.
+crafted DNxHD data can cause unspecified impact.
+
+Reviewed-by: Paul B Mahol 
+Signed-off-by: James Almer 
+---
+ libavcodec/dnxhddec.c | 22 +++---
+ 1 file changed, 15 insertions(+), 7 deletions(-)
+
+diff --git a/libavcodec/dnxhddec.c b/libavcodec/dnxhddec.c
+index e5d01e2e71..54f894f81b 100644
+--- a/libavcodec/dnxhddec.c
 b/libavcodec/dnxhddec.c
+@@ -110,6 +110,7 @@ static av_cold int dnxhd_decode_init(AVCodecContext *avctx)
+ 
+ static int dnxhd_init_vlc(DNXHDContext *ctx, uint32_t cid, int bitdepth)
+ {
++int ret;
+ if (cid != ctx->cid) {
+ int index;
+ 
+@@ -129,19 +130,26 @@ static int dnxhd_init_vlc(DNXHDContext *ctx, uint32_t 
cid, int bitdepth)
+ ff_free_vlc(>dc_vlc);
+ ff_free_vlc(>run_vlc);
+ 
+-init_vlc(>ac_vlc, DNXHD_VLC_BITS, 257,
++if ((ret = init_vlc(>ac_vlc, DNXHD_VLC_BITS, 257,
+  ctx->cid_table->ac_bits, 1, 1,
+- ctx->cid_table->ac_codes, 2, 2, 0);
+-init_vlc(>dc_vlc, DNXHD_DC_VLC_BITS, bitdepth > 8 ? 14 : 12,
++ ctx->cid_table->ac_codes, 2, 2, 0)) < 0)
++goto out;
++if ((ret = init_vlc(>dc_vlc, DNXHD_DC_VLC_BITS, bitdepth > 8 ? 
14 : 12,
+  ctx->cid_table->dc_bits, 1, 1,
+- ctx->cid_table->dc_codes, 1, 1, 0);
+-init_vlc(>run_vlc, DNXHD_VLC_BITS, 62,
++ ctx->cid_table->dc_codes, 1, 1, 0)) < 0)
++goto out;
++if ((ret = init_vlc(>run_vlc, DNXHD_VLC_BITS, 62,
+  ctx->cid_table->run_bits, 1, 1,
+- ctx->cid_table->run_codes, 2, 2, 0);
++ ctx->cid_table->run_codes, 2, 2, 0)) < 0)
++goto out;
+ 
+ ctx->cid = cid;
+ }
+-return 0;
++ret = 0;
++out:
++if (ret < 0)
++av_log(ctx->avctx, AV_LOG_ERROR, "init_vlc failed\n");
++return ret;
+ }
+ 
+ static int dnxhd_get_profile(int cid)
+-- 
+2.31.1
+
diff --git a/meta/recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb 
b/meta/recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb
index 0a49493abd..7df356946b 100644
--- a/meta/recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb
+++ b/meta/recipes-multimedia/ffmpeg/ffmpeg_4.3.2.bb
@@ -31,7 +31,8 @@ SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
file://fix-CVE-2020-22015.patch \
file://fix-CVE-2020-22021.patch \
file://fix-CVE-2020-22033-CVE-2020-22019.patch \
-   "
+   file://fix-CVE-2021-38114.patch \ 
+  "
 SRC_URI[sha256sum] = 
"46e4e64f1dd0233cbc0934b9f1c0da676008cad34725113fb7f802cfa84ccddb"
 
 # Build fails when thumb is enabled: 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7717
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156783): 
https://lists.openembedded.org/g/openembedded-core/message/156783
Mute This Topic: https://lists.openembedded.org/mt/86175257/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] oeqa/selftest/reproducibile: Include rust packages

2021-10-08 Thread Richard Purdie

On Fri, 2021-10-08 at 17:29 +0530, pgowda cve wrote:
> Hi Richard,
> 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/764
> 
> Thanks very much for sharing your inputs and comments on the issue.
> 
> I could see some more builds where rust exclusions were unused:-
> Some of the example builds are:-
> https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/757
> https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/755
> https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/751
> 
> Can you please let me know about this inconsistent behavior?

The reproducible test runs on different hosts and has two builds it compares.
The first build can come from sstate. The second build is built on the specific
worker the builds runs on.

We can see from my link above that the build that shows reproducibility issues
ran on fedora33 so it is probably something specific to that distro.

The most likely way to reproduce the issue is therefore to build on something
like an ubuntu system and compare the packages with those built on a fedora33
system.

> I will try to check the issue based on your inputs.
> 
> I tried to build the reproducible with and without rust exclusions on
> many poky sources.
> Could not see this error on any sources ranging over the past two months.
> More details on the issue are shared at:-
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14525

It sounds like you're only testing on debian whilst the autobuilder has multiple
different hosts which is the difference.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156782): 
https://lists.openembedded.org/g/openembedded-core/message/156782
Mute This Topic: https://lists.openembedded.org/mt/86165875/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 03/12] recipetool: Skip common source files in guess_license

2021-10-08 Thread Richard Purdie

On Fri, 2021-10-08 at 09:45 +0200, Konrad Weihmann wrote:
> 
> On 08.10.21 09:42, Stefan Herbrechtsmeier wrote:
> > From: Stefan Herbrechtsmeier 
> > 
> > Signed-off-by: Stefan Herbrechtsmeier 
> > 
> > ---
> > 
> > (no changes since v1)
> > 
> >   scripts/lib/recipetool/create.py | 2 ++
> >   1 file changed, 2 insertions(+)
> > 
> > diff --git a/scripts/lib/recipetool/create.py 
> > b/scripts/lib/recipetool/create.py
> > index e88a4253da..83cf25d9b7 100644
> > --- a/scripts/lib/recipetool/create.py
> > +++ b/scripts/lib/recipetool/create.py
> > @@ -1114,6 +1114,8 @@ def guess_license(srctree, d):
> >   licfiles = []
> >   for root, dirs, files in os.walk(srctree):
> >   for fn in files:
> > +if fn.endswith(".html") or fn.endswith(".js") or 
> > fn.endswith(".json") or fn.endswith(".svg") or fn.endswith(".ts"):
> 
> Why not
> 
> _, _ext = os.path.splitext(fn)
> if _ext in [".html", ".js", ".json", ".svg", ".ts"]:

you can do fn.endswith((".html", ".js", ".json", ".svg", ".ts"))

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156781): 
https://lists.openembedded.org/g/openembedded-core/message/156781
Mute This Topic: https://lists.openembedded.org/mt/86165223/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [docs] [PATCH 8/8] dev-manual: remove errant /

2021-10-08 Thread Michael Opdenacker


On 10/7/21 5:06 PM, Jon Mason wrote:
> Signed-off-by: Jon Mason 
> ---
>  documentation/dev-manual/start.rst | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/documentation/dev-manual/start.rst 
> b/documentation/dev-manual/start.rst
> index fc1b7c30752e..1ca25318cd53 100644
> --- a/documentation/dev-manual/start.rst
> +++ b/documentation/dev-manual/start.rst


Reviewed-by: Michael Opdenacker 
Applied to master-next. Many thanks!
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156780): 
https://lists.openembedded.org/g/openembedded-core/message/156780
Mute This Topic: https://lists.openembedded.org/mt/86168761/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [docs] [PATCH 7/8] documentation: use YOCTO_DL_URL

2021-10-08 Thread Michael Opdenacker

Hi Jon

On 10/7/21 5:06 PM, Jon Mason wrote:
> Signed-off-by: Jon Mason 
> ---
>  documentation/profile-manual/usage.rst | 28 +-
>  documentation/ref-manual/faq.rst   | 16 +++
>  documentation/ref-manual/variables.rst |  8 
>  3 files changed, 26 insertions(+), 26 deletions(-)


Reviewed-by: Michael Opdenacker 
Merged in "master-next".
Many thanks!
Michael.
-- 

Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156779): 
https://lists.openembedded.org/g/openembedded-core/message/156779
Mute This Topic: https://lists.openembedded.org/mt/86168683/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] oeqa/selftest/reproducibile: Include rust packages

2021-10-08 Thread Pgowda

Hi Richard,

>> https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/764

Thanks very much for sharing your inputs and comments on the issue.

I could see some more builds where rust exclusions were unused:-
Some of the example builds are:-
https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/757
https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/755
https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/751

Can you please let me know about this inconsistent behavior?
I will try to check the issue based on your inputs.

I tried to build the reproducible with and without rust exclusions on
many poky sources.
Could not see this error on any sources ranging over the past two months.
More details on the issue are shared at:-
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14525

Thanks,
pgowda

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156778): 
https://lists.openembedded.org/g/openembedded-core/message/156778
Mute This Topic: https://lists.openembedded.org/mt/86165875/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [docs] [PATCH 5/8] bitbake: replace http with https for URLs

2021-10-08 Thread Richard Purdie

On Thu, 2021-10-07 at 11:06 -0400, Jon Mason wrote:
> https has been the preferred way to access websites for many years now.
> Change all of the URLs with a _working_ https server/certificate to use
> that URL.
> 
> Signed-off-by: Jon Mason 
> ---
>  bitbake/README|  8 ++---
>  bitbake/doc/README|  2 +-
>  .../bitbake-user-manual-fetching.rst  |  2 +-
>  .../bitbake-user-manual-hello.rst |  4 +--
>  .../bitbake-user-manual-intro.rst |  6 ++--
>  .../bitbake-user-manual-ref-variables.rst | 10 +++---
>  bitbake/lib/bb/tests/fetch.py | 36 +--
>  bitbake/lib/layerindexlib/__init__.py |  4 +--
>  bitbake/lib/layerindexlib/restapi.py  |  4 +--
>  bitbake/lib/layerindexlib/tests/restapi.py|  2 +-
>  bitbake/lib/toaster/orm/fixtures/oe-core.xml  | 12 +++
>  bitbake/lib/toaster/orm/fixtures/poky.xml | 24 ++---
>  .../orm/management/commands/lsupdates.py  |  2 +-
>  .../toastergui/templates/layerdetails.html|  2 +-
>  .../templates/package_detail_base.html|  2 +-
>  15 files changed, 60 insertions(+), 60 deletions(-)

There was a small problem with bitbake-selftest now failing due to the changes
to tests/fetch.py.

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2672/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/2648/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/2649/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/2682/steps/12/logs/stdio

I'm isolating the bits that break it out and will drop those from the patch, I
think there are only a couple of bits that break things.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156777): 
https://lists.openembedded.org/g/openembedded-core/message/156777
Mute This Topic: https://lists.openembedded.org/mt/86166459/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [docs] [PATCH 6/8] documentation: replace http with https for URLs

2021-10-08 Thread Michael Opdenacker

Hi Jon,

Thanks for the patch!

On 10/7/21 5:06 PM, Jon Mason wrote:
> https has been the preferred way to access websites for many years now.
> Change all of the URLs with a _working_ https server/certificate to use
> that URL.
>
> Signed-off-by: Jon Mason 
> ---
>  documentation/profile-manual/usage.rst | 28 +++---
>  documentation/ref-manual/faq.rst   | 16 ++---
>  documentation/ref-manual/variables.rst |  8 +++
>  documentation/toaster-manual/reference.rst |  8 +++
>  4 files changed, 30 insertions(+), 30 deletions(-)


Reviewed-by: Michael Opdenacker 
and merged into master-next (I want this to be part of the pull request
I'm sending today).

Thanks again,

Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156776): 
https://lists.openembedded.org/g/openembedded-core/message/156776
Mute This Topic: https://lists.openembedded.org/mt/86166125/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [docs] [PATCH 5/8] bitbake: replace http with https for URLs

2021-10-08 Thread Michael Opdenacker

Hi Jon,

Many thanks for the patches!

On 10/7/21 5:06 PM, Jon Mason wrote:
> --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst
> +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst
> @@ -60,11 +60,11 @@ member Chris Larson split the project into two distinct 
> pieces:
>  -  OpenEmbedded, a metadata set utilized by BitBake
>  
>  Today, BitBake is the primary basis of the
> -`OpenEmbedded `__ project, which is being
> +`OpenEmbedded `__ project, which is being
>  used to build and maintain Linux distributions such as the `Angstrom
>  Distribution `__, and which is
>  also being used as the build tool for Linux projects such as the `Yocto
> -Project `__.
> +Project `__.


Unrelated here, but Angstrom is defunct now and
http://www.angstrom-distribution.org down. I'll address this through a
different patch, though.

Reviewed-by: Michael Opdenacker *
*Thanks again!
Michael
**

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156775): 
https://lists.openembedded.org/g/openembedded-core/message/156775
Mute This Topic: https://lists.openembedded.org/mt/86166015/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] oeqa/selftest/reproducibile: Include rust packages

2021-10-08 Thread Richard Purdie

On Fri, 2021-10-08 at 01:37 -0700, Pgowda wrote:
> rust-llvm-liblto and rust-llvm-staticdev have been tested and generates
> same ipk, rpm and deb files for reproducibleA and reproducibleB in
> debain environment.
> 
> Signed-off-by: Pgowda 
> ---
>  meta/lib/oeqa/selftest/cases/reproducible.py | 4 
>  1 file changed, 4 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py 
> b/meta/lib/oeqa/selftest/cases/reproducible.py
> index e4582cb82a..a147e933cc 100644
> --- a/meta/lib/oeqa/selftest/cases/reproducible.py
> +++ b/meta/lib/oeqa/selftest/cases/reproducible.py
> @@ -27,8 +27,6 @@ import datetime
>  
>  # ruby-ri-docs, meson:
>  
> #https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210215-0_td9la2/packages/diff-html/
> -# rust-llvm:
> -#https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210825-kaihham6/
>  exclude_packages = [
>   'glide',
>   'go-helloworld',
> @@ -36,8 +34,6 @@ exclude_packages = [
>   'go_',
>   'go-',
>   'ruby-ri-docs',
> - 'rust-llvm-liblto',
> - 'rust-llvm-staticdev'
>   ]
>  
>  def is_excluded(package):

Not sure how you checked this but here is a build where they were not:

https://autobuilder.yoctoproject.org/typhoon/#/builders/118/builds/764

Note the lines:


2021-10-08 04:36:27,176 - oe-selftest - INFO - Reproducibility summary for rpm: 
same=8819 different=0 different_excluded=11 missing=0 total=8830
unused_exclusions=['go-runtime', 'go_', 'ruby-ri-docs']

i.e. it used the rust exclusions.

Cheers,

Richard


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156774): 
https://lists.openembedded.org/g/openembedded-core/message/156774
Mute This Topic: https://lists.openembedded.org/mt/86165875/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] license: Allow treating missing license as error

2021-10-08 Thread Mike Crowe via lists.openembedded.org

Use mechanism inspired by insane.bbclass to allow individual recipes or
other configuration to determine whether a missing licence should be
treated as a warning (as it is now) or as an error. This is controlled
by whether the error class is in WARN_LICENSE or ERROR_LICENSE.

Use bb.fatal in the error case to ensure that the task really fails. If
only bb.error is used then do_populate_lic isn't re-run on subsequent
builds which could lead to the error being missed.

Signed-off-by: Mike Crowe 
---
 meta/classes/license.bbclass | 19 ++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 45d912741d..1805f18076 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -12,6 +12,23 @@ LICENSE_CREATE_PACKAGE ??= "0"
 LICENSE_PACKAGE_SUFFIX ??= "-lic"
 LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
 
+# Elect whether a given type of error is a warning or error, they may
+# have been set by other files.
+WARN_LICENSE ?= "no-license"
+ERROR_LICENSE ?= ""
+WARN_LICENSE[doc] = "Space-separated list of license problems that should be 
reported only as warnings"
+ERROR_LICENSE[doc] = "Space-separated list of license problems that should be 
reported as errors"
+
+def package_license_handle_error(error_class, error_msg, d):
+if error_class in (d.getVar("ERROR_LICENSE") or "").split():
+package_qa_write_error(error_class, error_msg, d)
+bb.fatal("License Issue: %s [%s]" % (error_msg, error_class))
+elif error_class in (d.getVar("WARN_LICENSE") or "").split():
+package_qa_write_error(error_class, error_msg, d)
+bb.warn("License Issue: %s [%s]" % (error_msg, error_class))
+else:
+bb.note("QA Issue: %s [%s]" % (error_msg, error_class))
+
 addtask populate_lic after do_patch before do_build
 do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
 do_populate_lic[cleandirs] = "${LICSSTATEDIR}"
@@ -190,7 +207,7 @@ def find_license_files(d):
 # Add explicity avoid of CLOSED license because this isn't generic
 if license_type != 'CLOSED':
 # And here is where we warn people that their licenses are 
lousy
-bb.warn("%s: No generic license file exists for: %s in any 
provider" % (pn, license_type))
+package_license_handle_error("no-license", "%s: No generic 
license file exists for: %s in any provider" % (pn, license_type), d)
 pass
 
 if not generic_directory:
-- 
2.30.2


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156773): 
https://lists.openembedded.org/g/openembedded-core/message/156773
Mute This Topic: https://lists.openembedded.org/mt/86165990/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] oeqa/selftest/reproducibile: Include rust packages

2021-10-08 Thread Pgowda

rust-llvm-liblto and rust-llvm-staticdev have been tested and generates
same ipk, rpm and deb files for reproducibleA and reproducibleB in
debain environment.

Signed-off-by: Pgowda 
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 4 
 1 file changed, 4 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py 
b/meta/lib/oeqa/selftest/cases/reproducible.py
index e4582cb82a..a147e933cc 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -27,8 +27,6 @@ import datetime
 
 # ruby-ri-docs, meson:
 
#https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210215-0_td9la2/packages/diff-html/
-# rust-llvm:
-#https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210825-kaihham6/
 exclude_packages = [
'glide',
'go-helloworld',
@@ -36,8 +34,6 @@ exclude_packages = [
'go_',
'go-',
'ruby-ri-docs',
-   'rust-llvm-liblto',
-   'rust-llvm-staticdev'
]
 
 def is_excluded(package):
-- 
2.31.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156772): 
https://lists.openembedded.org/g/openembedded-core/message/156772
Mute This Topic: https://lists.openembedded.org/mt/86165875/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] lttng-ust: fix do_compile failure for arm32 with DEBUG_BUILD enabled

2021-10-08 Thread Changqing Li

From: Changqing Li 

After upgrade to 2.13.0, for arm32, with DEBUG_BUILD enabled, lttng-ust
build failed with error:
| 
/path/to/tmp/work/cortexa15t2hf-neon-poky-linux-gnueabi/lttng-ust/2_2.13.0-r0/recipe-sysroot-native/usr/bin/arm-poky-linux-gnueabi/../../libexec/arm-poky-linux-gnueabi/gcc/arm-poky-linux-gnueabi/11.2.0/ld:
 ../../../src/lib/lttng-ust/.libs/liblttng-ust.so: undefined reference to 
`_uatomic_link_error'
| collect2: error: ld returned 1 exit status
| Makefile:399: recipe for target 'test_ust_error' failed
| make[3]: *** [test_ust_error] Error 1

The problem has reported to upstream, and upstream suggests to use
-DUATOMIC_NO_LINK_ERROR for the failure case, refer [1].

[1]https://lists.lttng.org/pipermail/lttng-dev/2021-September/030056.html

Signed-off-by: Changqing Li 
---
 meta/recipes-kernel/lttng/lttng-ust_2.13.0.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-kernel/lttng/lttng-ust_2.13.0.bb 
b/meta/recipes-kernel/lttng/lttng-ust_2.13.0.bb
index 4ec92c0683..7c6a0743e1 100644
--- a/meta/recipes-kernel/lttng/lttng-ust_2.13.0.bb
+++ b/meta/recipes-kernel/lttng/lttng-ust_2.13.0.bb
@@ -16,6 +16,7 @@ inherit autotools lib_package manpages python3native pkgconfig
 include lttng-platforms.inc
 
 EXTRA_OECONF = "--disable-numa"
+CPPFLAGS:append:arm = "${@oe.utils.vartrue('DEBUG_BUILD', 
'-DUATOMIC_NO_LINK_ERROR', '', d)}"
 
 DEPENDS = "liburcu util-linux"
 RDEPENDS:${PN}-bin = "python3-core"
-- 
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156771): 
https://lists.openembedded.org/g/openembedded-core/message/156771
Mute This Topic: https://lists.openembedded.org/mt/86165853/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 03/12] recipetool: Skip common source files in guess_license

2021-10-08 Thread Stefan Herbrechtsmeier


Am 08.10.2021 um 09:45 schrieb Konrad Weihmann:



On 08.10.21 09:42, Stefan Herbrechtsmeier wrote:

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 


---

(no changes since v1)

  scripts/lib/recipetool/create.py | 2 ++
  1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py 
b/scripts/lib/recipetool/create.py

index e88a4253da..83cf25d9b7 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1114,6 +1114,8 @@ def guess_license(srctree, d):
  licfiles = []
  for root, dirs, files in os.walk(srctree):
  for fn in files:
+    if fn.endswith(".html") or fn.endswith(".js") or 
fn.endswith(".json") or fn.endswith(".svg") or fn.endswith(".ts"):


Why not

_, _ext = os.path.splitext(fn)
if _ext in [".html", ".js", ".json", ".svg", ".ts"]:

?

to me that would be easier to read and maintain than this ever growing 
or clause construction


Will be changed in the next version.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156770): 
https://lists.openembedded.org/g/openembedded-core/message/156770
Mute This Topic: https://lists.openembedded.org/mt/86165223/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 12/12] recipetool: Rework crunch_license to recognize more variants

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 

---

(no changes since v1)

 scripts/lib/recipetool/create.py | 121 +--
 1 file changed, 97 insertions(+), 24 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 277266be4e..3b58d6fa82 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1048,54 +1048,127 @@ def crunch_license(licfile):
 import oe.utils
 
 # Note: these are carefully constructed!
-license_title_re = re.compile(r'^\(?(#+ *)?(The )?.{1,10} [Ll]icen[sc]e( 
\(.{1,10}\))?\)?:?$')
-license_statement_re = re.compile(r'^(This (project|software) is( free 
software)? (released|licen[sc]ed)|(Released|Licen[cs]ed)) under the .{1,10} 
[Ll]icen[sc]e:?$')
-copyright_re = re.compile('^(#+)? *Copyright .*$')
+license_title_re = re.compile(r'^#*\(? *(This is )?([Tt]he )?.{0,15} 
?[Ll]icen[sc]e( \(.{1,10}\))?\)?[:\.]? ?#*$')
+license_statement_re = re.compile(r'^((This (project|software)|.{1,10}) 
is( free software)? (released|licen[sc]ed)|(Released|Licen[cs]ed)) under the 
.{1,10} [Ll]icen[sc]e:?$')
+copyright_re = re.compile('^ *[#\*]* *(Modified work |MIT LICENSED 
)?Copyright ?(\([cC]\))? .*$')
+disclaimer_re = re.compile('^ *\*? ?All [Rr]ights [Rr]eserved\.$')
+email_re = re.compile('^.*<[\w\.-]*@[\w\.\-]*>$')
+header_re = re.compile('^(\/\**!?)? ?[\-=\*]* ?(\*\/)?$')
+tag_re = re.compile('^ *@?\(?([Ll]icense|MIT)\)?$')
+url_re = re.compile('^ *[#\*]* *https?:\/\/[\w\.\/\-]+$')
 
 crunched_md5sums = {}
+
+# common licenses
+crunched_md5sums['89f3bf322f30a1dcfe952e09945842f0'] = 'Apache-2.0'
+crunched_md5sums['13b6fe3075f8f42f2270a748965bf3a1'] = 'BSD-0-Clause'
+crunched_md5sums['ba87a7d7c20719c8df4b8beed9b78c43'] = 'BSD-2-Clause'
+crunched_md5sums['7f8892c03b72de419c27be4ebfa253f8'] = 'BSD-3-Clause'
+crunched_md5sums['21128c0790b23a8a9f9e260d5f6b3619'] = 'BSL-1.0'
+crunched_md5sums['975742a59ae1b8abdea63a97121f49f4'] = 'EDL-1.0'
+crunched_md5sums['5322cee4433d84fb3aafc9e253116447'] = 'EPL-1.0'
+crunched_md5sums['6922352e87de080f42419bed93063754'] = 'EPL-2.0'
+crunched_md5sums['793475baa22295cae1d3d4046a3a0ceb'] = 'GPL-2.0-only'
+crunched_md5sums['ff9047f969b02c20f0559470df5cb433'] = 'GPL-2.0-or-later'
+crunched_md5sums['ea6de5453fcadf534df246e6cdafadcd'] = 'GPL-3.0-only'
+crunched_md5sums['b419257d4d153a6fde92ddf96acf5b67'] = 'GPL-3.0-or-later'
+crunched_md5sums['228737f4c49d3ee75b8fb3706b090b84'] = 'ISC'
+crunched_md5sums['c6a782e826ca4e85bf7f8b89435a677d'] = 'LGPL-2.0-only'
+crunched_md5sums['32d8f758a066752f0db09bd7624b8090'] = 'LGPL-2.0-or-later'
+crunched_md5sums['4820937eb198b4f84c52217ed230be33'] = 'LGPL-2.1-only'
+crunched_md5sums['db13fe9f3a13af7adab2dc7a76f9e44a'] = 'LGPL-2.1-or-later'
+crunched_md5sums['d7a0f2e4e0950e837ac3eabf5bd1d246'] = 'LGPL-3.0-only'
+crunched_md5sums['abbf328e2b434f9153351f06b9f79d02'] = 'LGPL-3.0-or-later'
+crunched_md5sums['eecf6429523cbc9693547cf2db790b5c'] = 'MIT'
+crunched_md5sums['b218b0e94290b9b818c4be67c8e1cc82'] = 'MIT-0'
+crunched_md5sums['ddc18131d6748374f0f35a621c245b49'] = 'Unlicense'
+crunched_md5sums['51f9570ff32571fc0a443102285c5e33'] = 'WTFPL'
+
 # The following two were gleaned from the "forever" npm package
 crunched_md5sums['0a97f8e4cbaf889d6fa51f84b89a79f6'] = 'ISC'
-crunched_md5sums['eecf6429523cbc9693547cf2db790b5c'] = 'MIT'
-# https://github.com/vasi/pixz/blob/master/LICENSE
-crunched_md5sums['2f03392b40bbe663597b5bd3cc5ebdb9'] = 'BSD-2-Clause'
 # https://github.com/waffle-gl/waffle/blob/master/LICENSE.txt
-crunched_md5sums['e72e5dfef0b1a4ca8a3d26a60587db66'] = 'BSD-2-Clause'
+crunched_md5sums['50fab24ce589d69af8964fdbfe414c60'] = 'BSD-2-Clause'
 # https://github.com/spigwitmer/fakeds1963s/blob/master/LICENSE
-crunched_md5sums['8be76ac6d191671f347ee4916baa637e'] = 'GPLv2'
-# https://github.com/datto/dattobd/blob/master/COPYING
-# 
http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/docs/GPLv2.TXT
-crunched_md5sums['1d65c5ad4bf6489f85f4812bf08ae73d'] = 'GPLv2'
+crunched_md5sums['88a4355858a1433fea99fae34a44da88'] = 'GPLv2'
 # http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt
-# 
http://git.neil.brown.name/?p=mdadm.git;a=blob;f=COPYING;h=d159169d1050894d3ea3b98e1c965c4058208fe1;hb=HEAD
-crunched_md5sums['fb530f66a7a89ce920f0e912b5b66d4b'] = 'GPLv2'
-# https://github.com/gkos/nrf24/blob/master/COPYING
-crunched_md5sums['7b6aaa4daeafdfa6ed5443fd2684581b'] = 'GPLv2'
-# https://github.com/josch09/resetusb/blob/master/COPYING
-crunched_md5sums['8b8ac1d631a4d220342e83bcf1a1fbc3'] = 'GPLv3'
+crunched_md5sums['063b5c3ebb5f3aa4c85a2ed18a31fbe7'] = 'GPLv2'
 # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv2.1
-

[OE-core] [PATCH v2 11/12] npm: Use configs for npm environment and args for npm run command

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Use parameter configs of class NpmEnvironment and parameter args of
function run to support a common npmrc for all run calls of a single
NpmEnvironment.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 meta/classes/npm.bbclass | 14 ++
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 05bb9f578d..547c531883 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -61,8 +61,8 @@ def npm_pack(env, srcdir, workdir):
 """Run 'npm pack' on a specified directory"""
 import shlex
 cmd = "npm pack %s" % shlex.quote(srcdir)
-configs = [("ignore-scripts", "true")]
-tarball = env.run(cmd, configs=configs, workdir=workdir).strip("\n")
+args = [("ignore-scripts", "true")]
+tarball = env.run(cmd, args=args, workdir=workdir).strip("\n")
 return os.path.join(workdir, tarball)
 
 python npm_do_configure() {
@@ -228,15 +228,11 @@ python npm_do_compile() {
 
 bb.utils.remove(d.getVar("NPM_BUILD"), recurse=True)
 
-env = NpmEnvironment(d, configs=npm_global_configs(d))
-
-dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
-
 with tempfile.TemporaryDirectory() as tmpdir:
 args = []
-configs = []
+configs = npm_global_configs(d)
 
-if dev:
+if bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False):
 configs.append(("also", "development"))
 else:
 configs.append(("only", "production"))
@@ -254,6 +250,8 @@ python npm_do_compile() {
 configs.append(("nodedir", d.getVar("NPM_NODEDIR")))
 configs.append(("python", d.getVar("PYTHON")))
 
+env = NpmEnvironment(d, configs)
+
 # Add node-pre-gyp configuration
 args.append(("target_arch", d.getVar("NPM_ARCH")))
 args.append(("build-from-source", "true"))
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156767): 
https://lists.openembedded.org/g/openembedded-core/message/156767
Mute This Topic: https://lists.openembedded.org/mt/86165301/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 08/12] recipetool: npm: Do not add package.json files to LIC_FILES_CHKSUM

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

The package.json files doesn't contain any licenses. The name of the
license doesn't comply the license requirements of most liceneses.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create_npm.py | 10 --
 1 file changed, 10 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 2bcae91dfa..c939780931 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -115,17 +115,9 @@ class NpmRecipeHandler(RecipeHandler):
 
 def _handle_licenses(self, srctree, shrinkwrap_file, dev):
 """Return the extra license files and the list of packages"""
-licfiles = []
 packages = {}
 
-def _licfiles_append(licfile):
-"""Append 'licfile' to the license files list"""
-licfilepath = os.path.join(srctree, licfile)
-licmd5 = bb.utils.md5_file(licfilepath)
-licfiles.append("file://%s;md5=%s" % (licfile, licmd5))
-
 # Handle the parent package
-_licfiles_append("package.json")
 packages["${PN}"] = ""
 
 # Handle the dependencies
@@ -133,7 +125,6 @@ class NpmRecipeHandler(RecipeHandler):
 suffix = "-".join([self._npm_name(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
-_licfiles_append(os.path.join(destdir, "package.json"))
 packages["${PN}-" + suffix] = destdir
 
 with open(shrinkwrap_file, "r") as f:
@@ -246,7 +237,6 @@ class NpmRecipeHandler(RecipeHandler):
 
 bb.note("Handling licences ...")
 (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, 
dev)
-extravalues["LIC_FILES_CHKSUM"] = licfiles
 split_pkg_licenses(guess_license(srctree, d), packages, lines_after, 
[])
 
 classes.append("npm")
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156768): 
https://lists.openembedded.org/g/openembedded-core/message/156768
Mute This Topic: https://lists.openembedded.org/mt/86165302/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 07/12] recipetool: Add support for linenumbers to licenses.csv

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Add support for linenumbers (begin and end lines) to licenses.csv. Add
an optional linenumbers parameter to get_license_md5sums to support
different use cases.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index a8c4cdef4a..277266be4e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1002,11 +1002,11 @@ def handle_license_vars(srctree, lines_before, handled, 
extravalues, d):
 handled.append(('license', licvalues))
 return licvalues
 
-def get_license_md5sums(d, static_only=False):
+def get_license_md5sums(d, static_only=False, linenumbers=False):
 import bb.utils
 import csv
 md5sums = {}
-if not static_only:
+if not static_only and not linenumbers:
 # Gather md5sums of license files in common license dir
 commonlicdir = d.getVar('COMMON_LICENSE_DIR')
 for fn in os.listdir(commonlicdir):
@@ -1024,10 +1024,14 @@ def get_license_md5sums(d, static_only=False):
 csv_path = os.path.join(path, 'lib', 'recipetool', 'licenses.csv')
 if os.path.isfile(csv_path):
 with open(csv_path, newline='') as csv_file:
-fieldnames = ['md5sum', 'license']
+fieldnames = ['md5sum', 'license', 'beginline', 'endline', 
'md5']
 reader = csv.DictReader(csv_file, delimiter=',', 
fieldnames=fieldnames)
 for row in reader:
-md5sums[row['md5sum']] = row['license']
+if linenumbers:
+md5sums[row['md5sum']] = (
+row['license'], row['beginline'], row['endline'], 
row['md5'])
+else:
+md5sums[row['md5sum']] = row['license']
 
 return md5sums
 
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156766): 
https://lists.openembedded.org/g/openembedded-core/message/156766
Mute This Topic: https://lists.openembedded.org/mt/86165300/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 03/12] recipetool: Skip common source files in guess_license

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index e88a4253da..83cf25d9b7 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1114,6 +1114,8 @@ def guess_license(srctree, d):
 licfiles = []
 for root, dirs, files in os.walk(srctree):
 for fn in files:
+if fn.endswith(".html") or fn.endswith(".js") or 
fn.endswith(".json") or fn.endswith(".svg") or fn.endswith(".ts"):
+continue
 for spec in licspecs:
 if fnmatch.fnmatch(fn, spec):
 fullpath = os.path.join(root, fn)
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156763): 
https://lists.openembedded.org/g/openembedded-core/message/156763
Mute This Topic: https://lists.openembedded.org/mt/86165297/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 10/12] npm: Add variable NPM_NODEDIR with default value

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Replace the variable NPM_NODEDIR fallback to a default value inside the
code with a variable NPM_NODEDIR with default value.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 meta/classes/npm.bbclass | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 477b40d921..05bb9f578d 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -26,6 +26,8 @@ EXTRA_OENPM = ""
 
 NPM_INSTALL_DEV ?= "0"
 
+NPM_NODEDIR ?= "${RECIPE_SYSROOT_NATIVE}${prefix_native}"
+
 def npm_target_arch_map(target_arch):
 """Maps arch names to npm arch names"""
 import re
@@ -249,11 +251,7 @@ python npm_do_compile() {
 # Add node-gyp configuration
 configs.append(("arch", d.getVar("NPM_ARCH")))
 configs.append(("release", "true"))
-nodedir = d.getVar("NPM_NODEDIR")
-if not nodedir:
-sysroot = d.getVar("RECIPE_SYSROOT_NATIVE")
-nodedir = os.path.join(sysroot, 
d.getVar("prefix_native").strip("/"))
-configs.append(("nodedir", nodedir))
+configs.append(("nodedir", d.getVar("NPM_NODEDIR")))
 configs.append(("python", d.getVar("PYTHON")))
 
 # Add node-pre-gyp configuration
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156764): 
https://lists.openembedded.org/g/openembedded-core/message/156764
Mute This Topic: https://lists.openembedded.org/mt/86165298/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 06/12] recipetool: Add logger info for missing license entries

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 87b25ebc7e..a8c4cdef4a 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1128,6 +1128,9 @@ def guess_license(srctree, d):
 license, crunched_md5, lictext = crunch_license(licfile)
 if lictext and not license:
 license = 'Unknown'
+logger.info("Please add the following line for '%s' to a 
'lib/recipetool/licenses.csv' " \
+"and replace `Unknown` with the license:\n" \
+"%s,Unknown" % (os.path.relpath(licfile, srctree), 
md5value))
 if license:
 licenses.append((license, os.path.relpath(licfile, srctree), 
md5value))
 
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156765): 
https://lists.openembedded.org/g/openembedded-core/message/156765
Mute This Topic: https://lists.openembedded.org/mt/86165299/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 05/12] recipetool: ignore empty license files

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 893980a5ce..87b25ebc7e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1126,9 +1126,10 @@ def guess_license(srctree, d):
 license = md5sums.get(md5value, None)
 if not license:
 license, crunched_md5, lictext = crunch_license(licfile)
-if not license:
+if lictext and not license:
 license = 'Unknown'
-licenses.append((license, os.path.relpath(licfile, srctree), md5value))
+if license:
+licenses.append((license, os.path.relpath(licfile, srctree), 
md5value))
 
 # FIXME should we grab at least one source file with a license header and 
add that too?
 
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156762): 
https://lists.openembedded.org/g/openembedded-core/message/156762
Mute This Topic: https://lists.openembedded.org/mt/86165296/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 09/12] recipetool: npm: Use README as license fallback

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Use the README as license fallback if a license file is missing. Use the
linenumbers parameter of get_license_md5sums function to determine the
license text inside the README.

Signed-off-by: Stefan Herbrechtsmeier 

---

Changes in v2:
- Rework to add licenses to package LICENSE:${PN} and LIC_FILES_CHKSUM

 scripts/lib/recipetool/create_npm.py | 52 +++-
 1 file changed, 51 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index c939780931..3394a89970 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -6,6 +6,7 @@
 """Recipe creation tool - npm module support plugin"""
 
 import json
+import logging
 import os
 import re
 import sys
@@ -14,8 +15,10 @@ import bb
 from bb.fetch2.npm import NpmEnvironment
 from bb.fetch2.npmsw import foreach_dependencies
 from recipetool.create import RecipeHandler
+from recipetool.create import get_license_md5sums
 from recipetool.create import guess_license
 from recipetool.create import split_pkg_licenses
+logger = logging.getLogger('recipetool')
 
 TINFOIL = None
 
@@ -115,17 +118,36 @@ class NpmRecipeHandler(RecipeHandler):
 
 def _handle_licenses(self, srctree, shrinkwrap_file, dev):
 """Return the extra license files and the list of packages"""
+licfiles = []
 packages = {}
 
 # Handle the parent package
 packages["${PN}"] = ""
 
+def _licfiles_append_fallback_readme_files(destdir):
+"""Append README files as fallback to license files if a license 
files is missing"""
+
+fallback = True
+readmes = []
+basedir = os.path.join(srctree, destdir)
+for fn in os.listdir(basedir):
+upper = fn.upper()
+if upper.startswith("README"):
+fullpath = os.path.join(basedir, fn)
+readmes.append(fullpath)
+if upper.startswith("COPYING") or "LICENCE" in upper or 
"LICENSE" in upper:
+fallback = False
+if fallback:
+for readme in readmes:
+licfiles.append(os.path.relpath(readme, srctree))
+
 # Handle the dependencies
 def _handle_dependency(name, params, deptree):
 suffix = "-".join([self._npm_name(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
 packages["${PN}-" + suffix] = destdir
+_licfiles_append_fallback_readme_files(destdir)
 
 with open(shrinkwrap_file, "r") as f:
 shrinkwrap = json.load(f)
@@ -237,7 +259,35 @@ class NpmRecipeHandler(RecipeHandler):
 
 bb.note("Handling licences ...")
 (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, 
dev)
-split_pkg_licenses(guess_license(srctree, d), packages, lines_after, 
[])
+
+def _guess_odd_license(licfiles):
+import bb
+
+md5sums = get_license_md5sums(d, linenumbers=True)
+
+chksums = []
+licenses = []
+for licfile in licfiles:
+f = os.path.join(srctree, licfile)
+md5value = bb.utils.md5_file(f)
+(license, beginline, endline, md5) = md5sums.get(md5value,
+(None, "", "", ""))
+if not license:
+license = "Unknown"
+logger.info("Please add the following line for '%s' to a "
+"'lib/recipetool/licenses.csv' and replace `Unknown`, "
+"`X`, `Y` and `MD5` with the license, begin line, "
+"end line and partial MD5 checksum:\n" \
+"%s,Unknown,X,Y,MD5" % (licfile, md5value))
+chksums.append("file://%s%s%s;md5=%s" % (licfile,
+";beginline=%s" % (beginline) if beginline else "",
+";endline=%s" % (endline) if endline else "",
+md5 if md5 else md5value))
+licenses.append((license, licfile, md5value))
+return (licenses, chksums)
+
+(licenses, extravalues["LIC_FILES_CHKSUM"]) = 
_guess_odd_license(licfiles)
+split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, 
lines_after)
 
 classes.append("npm")
 handled.append("buildsystem")
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156761): 
https://lists.openembedded.org/g/openembedded-core/message/156761
Mute This Topic: https://lists.openembedded.org/mt/86165294/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 04/12] recipetool: Change default paramter fallback_licenses of function split_pkg_licenses from None to []

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 83cf25d9b7..893980a5ce 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1134,7 +1134,7 @@ def guess_license(srctree, d):
 
 return licenses
 
-def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, 
pn='${PN}'):
+def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=[], 
pn='${PN}'):
 """
 Given a list of (license, path, md5sum) as returned by guess_license(),
 a dict of package name to path mappings, write out a set of
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156760): 
https://lists.openembedded.org/g/openembedded-core/message/156760
Mute This Topic: https://lists.openembedded.org/mt/86165293/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH v2 02/12] recipetool: Move license md5sums into CSV files

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py| 52 +
 scripts/lib/recipetool/licenses.csv | 37 
 2 files changed, 52 insertions(+), 37 deletions(-)
 create mode 100644 scripts/lib/recipetool/licenses.csv

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5cd72ea0a7..e88a4253da 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1004,6 +1004,7 @@ def handle_license_vars(srctree, lines_before, handled, 
extravalues, d):
 
 def get_license_md5sums(d, static_only=False):
 import bb.utils
+import csv
 md5sums = {}
 if not static_only:
 # Gather md5sums of license files in common license dir
@@ -1011,46 +1012,23 @@ def get_license_md5sums(d, static_only=False):
 for fn in os.listdir(commonlicdir):
 md5value = bb.utils.md5_file(os.path.join(commonlicdir, fn))
 md5sums[md5value] = fn
+
 # The following were extracted from common values in various recipes
 # (double checking the license against the license file itself, not just
 # the LICENSE value in the recipe)
-md5sums['94d55d512a9ba36caa9b7df079bae19f'] = 'GPLv2'
-md5sums['b234ee4d69f5fce4486a80fdaf4a4263'] = 'GPLv2'
-md5sums['59530bdf33659b29e73d4adb9f9f6552'] = 'GPLv2'
-md5sums['0636e73ff0215e8d672dc4c32c317bb3'] = 'GPLv2'
-md5sums['eb723b61539feef013de476e68b5c50a'] = 'GPLv2'
-md5sums['751419260aa954499f7abaabaa882bbe'] = 'GPLv2'
-md5sums['393a5ca445f6965873eca0259a17f833'] = 'GPLv2'
-md5sums['12f884d2ae1ff87c09e5b7ccc2c4ca7e'] = 'GPLv2'
-md5sums['8ca43cbc842c2336e835926c2166c28b'] = 'GPLv2'
-md5sums['ebb5c50ab7cab4baeffba14977030c07'] = 'GPLv2'
-md5sums['c93c0550bd3173f4504b2cbd8991e50b'] = 'GPLv2'
-md5sums['9ac2e7cff1ddaf48b6eab6028f23ef88'] = 'GPLv2'
-md5sums['4325afd396febcb659c36b49533135d4'] = 'GPLv2'
-md5sums['18810669f13b87348459e611d31ab760'] = 'GPLv2'
-md5sums['d7810fab7487fb0aad327b76f1be7cd7'] = 'GPLv2' # the Linux kernel's 
COPYING file
-md5sums['bbb461211a33b134d42ed5ee802b37ff'] = 'LGPLv2.1'
-md5sums['7fbc338309ac38fefcd64b04bb903e34'] = 'LGPLv2.1'
-md5sums['4fbd65380cdd255951079008b364516c'] = 'LGPLv2.1'
-md5sums['2d5025d4aa3495befef8f17206a5b0a1'] = 'LGPLv2.1'
-md5sums['fbc093901857fcd118f065f900982c24'] = 'LGPLv2.1'
-md5sums['a6f89e2100d9b6cdffcea4f398e37343'] = 'LGPLv2.1'
-md5sums['d8045f3b8f929c1cb29a1e3fd737b499'] = 'LGPLv2.1'
-md5sums['fad9b3332be894bab9bc501572864b29'] = 'LGPLv2.1'
-md5sums['3bf50002aefd002f49e7bb854063f7e7'] = 'LGPLv2'
-md5sums['9f604d8a4f8e74f4f5140845a21b6674'] = 'LGPLv2'
-md5sums['5f30f0716dfdd0d91eb439ebec522ec2'] = 'LGPLv2'
-md5sums['55ca817ccb7d5b5b66355690e9abc605'] = 'LGPLv2'
-md5sums['252890d9eee26aab7b432e8b8a616475'] = 'LGPLv2'
-md5sums['3214f080875748938ba060314b4f727d'] = 'LGPLv2'
-md5sums['db979804f025cf55aabec7129cb671ed'] = 'LGPLv2'
-md5sums['d32239bcb673463ab874e80d47fae504'] = 'GPLv3'
-md5sums['f27defe1e96c2e1ecd4e0c9be8967949'] = 'GPLv3'
-md5sums['6a6a8e020838b23406c81b19c1d46df6'] = 'LGPLv3'
-md5sums['3b83ef96387f14655fc854ddc3c6bd57'] = 'Apache-2.0'
-md5sums['385c55653886acac3821999a3ccd17b3'] = 'Artistic-1.0 | GPL-2.0' # 
some perl modules
-md5sums['54c7042be62e169199200bc6477f04d1'] = 'BSD-3-Clause'
-md5sums['bfe1f75d606912a4111c90743d6c7325'] = 'MPL-1.1'
+
+# Read license md5sums from csv file
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+for path in (d.getVar('BBPATH').split(':')
++ [os.path.join(scripts_path, '..', '..')]):
+csv_path = os.path.join(path, 'lib', 'recipetool', 'licenses.csv')
+if os.path.isfile(csv_path):
+with open(csv_path, newline='') as csv_file:
+fieldnames = ['md5sum', 'license']
+reader = csv.DictReader(csv_file, delimiter=',', 
fieldnames=fieldnames)
+for row in reader:
+md5sums[row['md5sum']] = row['license']
+
 return md5sums
 
 def crunch_license(licfile):
diff --git a/scripts/lib/recipetool/licenses.csv 
b/scripts/lib/recipetool/licenses.csv
new file mode 100644
index 00..0d3fb0607b
--- /dev/null
+++ b/scripts/lib/recipetool/licenses.csv
@@ -0,0 +1,37 @@
+0636e73ff0215e8d672dc4c32c317bb3,GPLv2
+12f884d2ae1ff87c09e5b7ccc2c4ca7e,GPLv2
+18810669f13b87348459e611d31ab760,GPLv2
+252890d9eee26aab7b432e8b8a616475,LGPLv2
+2d5025d4aa3495befef8f17206a5b0a1,LGPLv2.1
+3214f080875748938ba060314b4f727d,LGPLv2
+385c55653886acac3821999a3ccd17b3,Artistic-1.0 | GPL-2.0
+393a5ca445f6965873eca0259a17f833,GPLv2
+3b83ef96387f14655fc854ddc3c6bd57,Apache-2.0
+3bf50002aefd002f49e7bb854063f7e7,LGPLv2
+4325afd396febcb659c36b49533135d4,GPLv2
+4fbd65380cdd255951079008b364516c,LGPLv2.1

[OE-core] [PATCH v2 01/12] npm: Add support for EXTRA_OENPM arguments

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Add support for EXTRA_OENPM arguments to set node-gyp variables for
example. This allows use of shared librariess, avoid download
of external sources or build from source.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 meta/classes/npm.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 91a2321116..477b40d921 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -22,6 +22,8 @@ inherit python3native
 DEPENDS:prepend = "nodejs-native "
 RDEPENDS:${PN}:append:class-target = " nodejs"
 
+EXTRA_OENPM = ""
+
 NPM_INSTALL_DEV ?= "0"
 
 def npm_target_arch_map(target_arch):
@@ -260,7 +262,8 @@ python npm_do_compile() {
 
 # Pack and install the main package
 tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
-env.run("npm install %s" % shlex.quote(tarball), args=args, 
configs=configs)
+cmd = "npm install %s %s" % (shlex.quote(tarball), 
d.getVar("EXTRA_OENPM"))
+env.run(cmd, args=args, configs=configs)
 }
 
 npm_do_install() {
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156758): 
https://lists.openembedded.org/g/openembedded-core/message/156758
Mute This Topic: https://lists.openembedded.org/mt/86165291/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH 03/12] recipetool: Skip common source files in guess_license

2021-10-08 Thread Konrad Weihmann




On 08.10.21 09:42, Stefan Herbrechtsmeier wrote:

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

  scripts/lib/recipetool/create.py | 2 ++
  1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index e88a4253da..83cf25d9b7 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1114,6 +1114,8 @@ def guess_license(srctree, d):
  licfiles = []
  for root, dirs, files in os.walk(srctree):
  for fn in files:
+if fn.endswith(".html") or fn.endswith(".js") or fn.endswith(".json") or 
fn.endswith(".svg") or fn.endswith(".ts"):


Why not

_, _ext = os.path.splitext(fn)
if _ext in [".html", ".js", ".json", ".svg", ".ts"]:

?

to me that would be easier to read and maintain than this ever growing 
or clause construction



+continue
  for spec in licspecs:
  if fnmatch.fnmatch(fn, spec):
  fullpath = os.path.join(root, fn)






-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156757): 
https://lists.openembedded.org/g/openembedded-core/message/156757
Mute This Topic: https://lists.openembedded.org/mt/86165223/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 11/12] npm: Use configs for npm environment and args for npm run command

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Use parameter configs of class NpmEnvironment and parameter args of
function run to support a common npmrc for all run calls of a single
NpmEnvironment.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 meta/classes/npm.bbclass | 14 ++
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 05bb9f578d..547c531883 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -61,8 +61,8 @@ def npm_pack(env, srcdir, workdir):
 """Run 'npm pack' on a specified directory"""
 import shlex
 cmd = "npm pack %s" % shlex.quote(srcdir)
-configs = [("ignore-scripts", "true")]
-tarball = env.run(cmd, configs=configs, workdir=workdir).strip("\n")
+args = [("ignore-scripts", "true")]
+tarball = env.run(cmd, args=args, workdir=workdir).strip("\n")
 return os.path.join(workdir, tarball)
 
 python npm_do_configure() {
@@ -228,15 +228,11 @@ python npm_do_compile() {
 
 bb.utils.remove(d.getVar("NPM_BUILD"), recurse=True)
 
-env = NpmEnvironment(d, configs=npm_global_configs(d))
-
-dev = bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False)
-
 with tempfile.TemporaryDirectory() as tmpdir:
 args = []
-configs = []
+configs = npm_global_configs(d)
 
-if dev:
+if bb.utils.to_boolean(d.getVar("NPM_INSTALL_DEV"), False):
 configs.append(("also", "development"))
 else:
 configs.append(("only", "production"))
@@ -254,6 +250,8 @@ python npm_do_compile() {
 configs.append(("nodedir", d.getVar("NPM_NODEDIR")))
 configs.append(("python", d.getVar("PYTHON")))
 
+env = NpmEnvironment(d, configs)
+
 # Add node-pre-gyp configuration
 args.append(("target_arch", d.getVar("NPM_ARCH")))
 args.append(("build-from-source", "true"))
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156756): 
https://lists.openembedded.org/g/openembedded-core/message/156756
Mute This Topic: https://lists.openembedded.org/mt/86165234/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 12/12] recipetool: Rework crunch_license to recognize more variants

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 121 +--
 1 file changed, 97 insertions(+), 24 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 277266be4e..3b58d6fa82 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1048,54 +1048,127 @@ def crunch_license(licfile):
 import oe.utils
 
 # Note: these are carefully constructed!
-license_title_re = re.compile(r'^\(?(#+ *)?(The )?.{1,10} [Ll]icen[sc]e( 
\(.{1,10}\))?\)?:?$')
-license_statement_re = re.compile(r'^(This (project|software) is( free 
software)? (released|licen[sc]ed)|(Released|Licen[cs]ed)) under the .{1,10} 
[Ll]icen[sc]e:?$')
-copyright_re = re.compile('^(#+)? *Copyright .*$')
+license_title_re = re.compile(r'^#*\(? *(This is )?([Tt]he )?.{0,15} 
?[Ll]icen[sc]e( \(.{1,10}\))?\)?[:\.]? ?#*$')
+license_statement_re = re.compile(r'^((This (project|software)|.{1,10}) 
is( free software)? (released|licen[sc]ed)|(Released|Licen[cs]ed)) under the 
.{1,10} [Ll]icen[sc]e:?$')
+copyright_re = re.compile('^ *[#\*]* *(Modified work |MIT LICENSED 
)?Copyright ?(\([cC]\))? .*$')
+disclaimer_re = re.compile('^ *\*? ?All [Rr]ights [Rr]eserved\.$')
+email_re = re.compile('^.*<[\w\.-]*@[\w\.\-]*>$')
+header_re = re.compile('^(\/\**!?)? ?[\-=\*]* ?(\*\/)?$')
+tag_re = re.compile('^ *@?\(?([Ll]icense|MIT)\)?$')
+url_re = re.compile('^ *[#\*]* *https?:\/\/[\w\.\/\-]+$')
 
 crunched_md5sums = {}
+
+# common licenses
+crunched_md5sums['89f3bf322f30a1dcfe952e09945842f0'] = 'Apache-2.0'
+crunched_md5sums['13b6fe3075f8f42f2270a748965bf3a1'] = 'BSD-0-Clause'
+crunched_md5sums['ba87a7d7c20719c8df4b8beed9b78c43'] = 'BSD-2-Clause'
+crunched_md5sums['7f8892c03b72de419c27be4ebfa253f8'] = 'BSD-3-Clause'
+crunched_md5sums['21128c0790b23a8a9f9e260d5f6b3619'] = 'BSL-1.0'
+crunched_md5sums['975742a59ae1b8abdea63a97121f49f4'] = 'EDL-1.0'
+crunched_md5sums['5322cee4433d84fb3aafc9e253116447'] = 'EPL-1.0'
+crunched_md5sums['6922352e87de080f42419bed93063754'] = 'EPL-2.0'
+crunched_md5sums['793475baa22295cae1d3d4046a3a0ceb'] = 'GPL-2.0-only'
+crunched_md5sums['ff9047f969b02c20f0559470df5cb433'] = 'GPL-2.0-or-later'
+crunched_md5sums['ea6de5453fcadf534df246e6cdafadcd'] = 'GPL-3.0-only'
+crunched_md5sums['b419257d4d153a6fde92ddf96acf5b67'] = 'GPL-3.0-or-later'
+crunched_md5sums['228737f4c49d3ee75b8fb3706b090b84'] = 'ISC'
+crunched_md5sums['c6a782e826ca4e85bf7f8b89435a677d'] = 'LGPL-2.0-only'
+crunched_md5sums['32d8f758a066752f0db09bd7624b8090'] = 'LGPL-2.0-or-later'
+crunched_md5sums['4820937eb198b4f84c52217ed230be33'] = 'LGPL-2.1-only'
+crunched_md5sums['db13fe9f3a13af7adab2dc7a76f9e44a'] = 'LGPL-2.1-or-later'
+crunched_md5sums['d7a0f2e4e0950e837ac3eabf5bd1d246'] = 'LGPL-3.0-only'
+crunched_md5sums['abbf328e2b434f9153351f06b9f79d02'] = 'LGPL-3.0-or-later'
+crunched_md5sums['eecf6429523cbc9693547cf2db790b5c'] = 'MIT'
+crunched_md5sums['b218b0e94290b9b818c4be67c8e1cc82'] = 'MIT-0'
+crunched_md5sums['ddc18131d6748374f0f35a621c245b49'] = 'Unlicense'
+crunched_md5sums['51f9570ff32571fc0a443102285c5e33'] = 'WTFPL'
+
 # The following two were gleaned from the "forever" npm package
 crunched_md5sums['0a97f8e4cbaf889d6fa51f84b89a79f6'] = 'ISC'
-crunched_md5sums['eecf6429523cbc9693547cf2db790b5c'] = 'MIT'
-# https://github.com/vasi/pixz/blob/master/LICENSE
-crunched_md5sums['2f03392b40bbe663597b5bd3cc5ebdb9'] = 'BSD-2-Clause'
 # https://github.com/waffle-gl/waffle/blob/master/LICENSE.txt
-crunched_md5sums['e72e5dfef0b1a4ca8a3d26a60587db66'] = 'BSD-2-Clause'
+crunched_md5sums['50fab24ce589d69af8964fdbfe414c60'] = 'BSD-2-Clause'
 # https://github.com/spigwitmer/fakeds1963s/blob/master/LICENSE
-crunched_md5sums['8be76ac6d191671f347ee4916baa637e'] = 'GPLv2'
-# https://github.com/datto/dattobd/blob/master/COPYING
-# 
http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/docs/GPLv2.TXT
-crunched_md5sums['1d65c5ad4bf6489f85f4812bf08ae73d'] = 'GPLv2'
+crunched_md5sums['88a4355858a1433fea99fae34a44da88'] = 'GPLv2'
 # http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt
-# 
http://git.neil.brown.name/?p=mdadm.git;a=blob;f=COPYING;h=d159169d1050894d3ea3b98e1c965c4058208fe1;hb=HEAD
-crunched_md5sums['fb530f66a7a89ce920f0e912b5b66d4b'] = 'GPLv2'
-# https://github.com/gkos/nrf24/blob/master/COPYING
-crunched_md5sums['7b6aaa4daeafdfa6ed5443fd2684581b'] = 'GPLv2'
-# https://github.com/josch09/resetusb/blob/master/COPYING
-crunched_md5sums['8b8ac1d631a4d220342e83bcf1a1fbc3'] = 'GPLv3'
+crunched_md5sums['063b5c3ebb5f3aa4c85a2ed18a31fbe7'] = 'GPLv2'
 # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv2.1
-crunched_md5sums['2ea316ed973ae176e502e2297b574bb3']

[OE-core] [PATCH 10/12] npm: Add variable NPM_NODEDIR with default value

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Replace the variable NPM_NODEDIR fallback to a default value inside the
code with a variable NPM_NODEDIR with default value.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 meta/classes/npm.bbclass | 8 +++-
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 477b40d921..05bb9f578d 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -26,6 +26,8 @@ EXTRA_OENPM = ""
 
 NPM_INSTALL_DEV ?= "0"
 
+NPM_NODEDIR ?= "${RECIPE_SYSROOT_NATIVE}${prefix_native}"
+
 def npm_target_arch_map(target_arch):
 """Maps arch names to npm arch names"""
 import re
@@ -249,11 +251,7 @@ python npm_do_compile() {
 # Add node-gyp configuration
 configs.append(("arch", d.getVar("NPM_ARCH")))
 configs.append(("release", "true"))
-nodedir = d.getVar("NPM_NODEDIR")
-if not nodedir:
-sysroot = d.getVar("RECIPE_SYSROOT_NATIVE")
-nodedir = os.path.join(sysroot, 
d.getVar("prefix_native").strip("/"))
-configs.append(("nodedir", nodedir))
+configs.append(("nodedir", d.getVar("NPM_NODEDIR")))
 configs.append(("python", d.getVar("PYTHON")))
 
 # Add node-pre-gyp configuration
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156753): 
https://lists.openembedded.org/g/openembedded-core/message/156753
Mute This Topic: https://lists.openembedded.org/mt/86165231/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 09/12] recipetool: npm: Use README as license fallback

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Use the README as license fallback if a license file is missing. Use the
linenumbers parameter of get_license_md5sums function to determine the
license text inside the README.

Signed-off-by: Stefan Herbrechtsmeier 

---

Changes in v2:
- Rework to add licenses to package LICENSE:${PN} and LIC_FILES_CHKSUM

 scripts/lib/recipetool/create_npm.py | 52 +++-
 1 file changed, 51 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index c939780931..3394a89970 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -6,6 +6,7 @@
 """Recipe creation tool - npm module support plugin"""
 
 import json
+import logging
 import os
 import re
 import sys
@@ -14,8 +15,10 @@ import bb
 from bb.fetch2.npm import NpmEnvironment
 from bb.fetch2.npmsw import foreach_dependencies
 from recipetool.create import RecipeHandler
+from recipetool.create import get_license_md5sums
 from recipetool.create import guess_license
 from recipetool.create import split_pkg_licenses
+logger = logging.getLogger('recipetool')
 
 TINFOIL = None
 
@@ -115,17 +118,36 @@ class NpmRecipeHandler(RecipeHandler):
 
 def _handle_licenses(self, srctree, shrinkwrap_file, dev):
 """Return the extra license files and the list of packages"""
+licfiles = []
 packages = {}
 
 # Handle the parent package
 packages["${PN}"] = ""
 
+def _licfiles_append_fallback_readme_files(destdir):
+"""Append README files as fallback to license files if a license 
files is missing"""
+
+fallback = True
+readmes = []
+basedir = os.path.join(srctree, destdir)
+for fn in os.listdir(basedir):
+upper = fn.upper()
+if upper.startswith("README"):
+fullpath = os.path.join(basedir, fn)
+readmes.append(fullpath)
+if upper.startswith("COPYING") or "LICENCE" in upper or 
"LICENSE" in upper:
+fallback = False
+if fallback:
+for readme in readmes:
+licfiles.append(os.path.relpath(readme, srctree))
+
 # Handle the dependencies
 def _handle_dependency(name, params, deptree):
 suffix = "-".join([self._npm_name(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
 packages["${PN}-" + suffix] = destdir
+_licfiles_append_fallback_readme_files(destdir)
 
 with open(shrinkwrap_file, "r") as f:
 shrinkwrap = json.load(f)
@@ -237,7 +259,35 @@ class NpmRecipeHandler(RecipeHandler):
 
 bb.note("Handling licences ...")
 (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, 
dev)
-split_pkg_licenses(guess_license(srctree, d), packages, lines_after, 
[])
+
+def _guess_odd_license(licfiles):
+import bb
+
+md5sums = get_license_md5sums(d, linenumbers=True)
+
+chksums = []
+licenses = []
+for licfile in licfiles:
+f = os.path.join(srctree, licfile)
+md5value = bb.utils.md5_file(f)
+(license, beginline, endline, md5) = md5sums.get(md5value,
+(None, "", "", ""))
+if not license:
+license = "Unknown"
+logger.info("Please add the following line for '%s' to a "
+"'lib/recipetool/licenses.csv' and replace `Unknown`, "
+"`X`, `Y` and `MD5` with the license, begin line, "
+"end line and partial MD5 checksum:\n" \
+"%s,Unknown,X,Y,MD5" % (licfile, md5value))
+chksums.append("file://%s%s%s;md5=%s" % (licfile,
+";beginline=%s" % (beginline) if beginline else "",
+";endline=%s" % (endline) if endline else "",
+md5 if md5 else md5value))
+licenses.append((license, licfile, md5value))
+return (licenses, chksums)
+
+(licenses, extravalues["LIC_FILES_CHKSUM"]) = 
_guess_odd_license(licfiles)
+split_pkg_licenses([*licenses, *guess_license(srctree, d)], packages, 
lines_after)
 
 classes.append("npm")
 handled.append("buildsystem")
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156754): 
https://lists.openembedded.org/g/openembedded-core/message/156754
Mute This Topic: https://lists.openembedded.org/mt/86165232/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 07/12] recipetool: Add support for linenumbers to licenses.csv

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Add support for linenumbers (begin and end lines) to licenses.csv. Add
an optional linenumbers parameter to get_license_md5sums to support
different use cases.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 12 
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index a8c4cdef4a..277266be4e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1002,11 +1002,11 @@ def handle_license_vars(srctree, lines_before, handled, 
extravalues, d):
 handled.append(('license', licvalues))
 return licvalues
 
-def get_license_md5sums(d, static_only=False):
+def get_license_md5sums(d, static_only=False, linenumbers=False):
 import bb.utils
 import csv
 md5sums = {}
-if not static_only:
+if not static_only and not linenumbers:
 # Gather md5sums of license files in common license dir
 commonlicdir = d.getVar('COMMON_LICENSE_DIR')
 for fn in os.listdir(commonlicdir):
@@ -1024,10 +1024,14 @@ def get_license_md5sums(d, static_only=False):
 csv_path = os.path.join(path, 'lib', 'recipetool', 'licenses.csv')
 if os.path.isfile(csv_path):
 with open(csv_path, newline='') as csv_file:
-fieldnames = ['md5sum', 'license']
+fieldnames = ['md5sum', 'license', 'beginline', 'endline', 
'md5']
 reader = csv.DictReader(csv_file, delimiter=',', 
fieldnames=fieldnames)
 for row in reader:
-md5sums[row['md5sum']] = row['license']
+if linenumbers:
+md5sums[row['md5sum']] = (
+row['license'], row['beginline'], row['endline'], 
row['md5'])
+else:
+md5sums[row['md5sum']] = row['license']
 
 return md5sums
 
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156752): 
https://lists.openembedded.org/g/openembedded-core/message/156752
Mute This Topic: https://lists.openembedded.org/mt/86165229/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 08/12] recipetool: npm: Do not add package.json files to LIC_FILES_CHKSUM

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

The package.json files doesn't contain any licenses. The name of the
license doesn't comply the license requirements of most liceneses.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create_npm.py | 10 --
 1 file changed, 10 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py 
b/scripts/lib/recipetool/create_npm.py
index 2bcae91dfa..c939780931 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -115,17 +115,9 @@ class NpmRecipeHandler(RecipeHandler):
 
 def _handle_licenses(self, srctree, shrinkwrap_file, dev):
 """Return the extra license files and the list of packages"""
-licfiles = []
 packages = {}
 
-def _licfiles_append(licfile):
-"""Append 'licfile' to the license files list"""
-licfilepath = os.path.join(srctree, licfile)
-licmd5 = bb.utils.md5_file(licfilepath)
-licfiles.append("file://%s;md5=%s" % (licfile, licmd5))
-
 # Handle the parent package
-_licfiles_append("package.json")
 packages["${PN}"] = ""
 
 # Handle the dependencies
@@ -133,7 +125,6 @@ class NpmRecipeHandler(RecipeHandler):
 suffix = "-".join([self._npm_name(dep) for dep in deptree])
 destdirs = [os.path.join("node_modules", dep) for dep in deptree]
 destdir = os.path.join(*destdirs)
-_licfiles_append(os.path.join(destdir, "package.json"))
 packages["${PN}-" + suffix] = destdir
 
 with open(shrinkwrap_file, "r") as f:
@@ -246,7 +237,6 @@ class NpmRecipeHandler(RecipeHandler):
 
 bb.note("Handling licences ...")
 (licfiles, packages) = self._handle_licenses(srctree, shrinkwrap_file, 
dev)
-extravalues["LIC_FILES_CHKSUM"] = licfiles
 split_pkg_licenses(guess_license(srctree, d), packages, lines_after, 
[])
 
 classes.append("npm")
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156751): 
https://lists.openembedded.org/g/openembedded-core/message/156751
Mute This Topic: https://lists.openembedded.org/mt/86165228/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 06/12] recipetool: Add logger info for missing license entries

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 87b25ebc7e..a8c4cdef4a 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1128,6 +1128,9 @@ def guess_license(srctree, d):
 license, crunched_md5, lictext = crunch_license(licfile)
 if lictext and not license:
 license = 'Unknown'
+logger.info("Please add the following line for '%s' to a 
'lib/recipetool/licenses.csv' " \
+"and replace `Unknown` with the license:\n" \
+"%s,Unknown" % (os.path.relpath(licfile, srctree), 
md5value))
 if license:
 licenses.append((license, os.path.relpath(licfile, srctree), 
md5value))
 
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156749): 
https://lists.openembedded.org/g/openembedded-core/message/156749
Mute This Topic: https://lists.openembedded.org/mt/86165226/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 02/12] recipetool: Move license md5sums into CSV files

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py| 52 +
 scripts/lib/recipetool/licenses.csv | 37 
 2 files changed, 52 insertions(+), 37 deletions(-)
 create mode 100644 scripts/lib/recipetool/licenses.csv

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5cd72ea0a7..e88a4253da 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1004,6 +1004,7 @@ def handle_license_vars(srctree, lines_before, handled, 
extravalues, d):
 
 def get_license_md5sums(d, static_only=False):
 import bb.utils
+import csv
 md5sums = {}
 if not static_only:
 # Gather md5sums of license files in common license dir
@@ -1011,46 +1012,23 @@ def get_license_md5sums(d, static_only=False):
 for fn in os.listdir(commonlicdir):
 md5value = bb.utils.md5_file(os.path.join(commonlicdir, fn))
 md5sums[md5value] = fn
+
 # The following were extracted from common values in various recipes
 # (double checking the license against the license file itself, not just
 # the LICENSE value in the recipe)
-md5sums['94d55d512a9ba36caa9b7df079bae19f'] = 'GPLv2'
-md5sums['b234ee4d69f5fce4486a80fdaf4a4263'] = 'GPLv2'
-md5sums['59530bdf33659b29e73d4adb9f9f6552'] = 'GPLv2'
-md5sums['0636e73ff0215e8d672dc4c32c317bb3'] = 'GPLv2'
-md5sums['eb723b61539feef013de476e68b5c50a'] = 'GPLv2'
-md5sums['751419260aa954499f7abaabaa882bbe'] = 'GPLv2'
-md5sums['393a5ca445f6965873eca0259a17f833'] = 'GPLv2'
-md5sums['12f884d2ae1ff87c09e5b7ccc2c4ca7e'] = 'GPLv2'
-md5sums['8ca43cbc842c2336e835926c2166c28b'] = 'GPLv2'
-md5sums['ebb5c50ab7cab4baeffba14977030c07'] = 'GPLv2'
-md5sums['c93c0550bd3173f4504b2cbd8991e50b'] = 'GPLv2'
-md5sums['9ac2e7cff1ddaf48b6eab6028f23ef88'] = 'GPLv2'
-md5sums['4325afd396febcb659c36b49533135d4'] = 'GPLv2'
-md5sums['18810669f13b87348459e611d31ab760'] = 'GPLv2'
-md5sums['d7810fab7487fb0aad327b76f1be7cd7'] = 'GPLv2' # the Linux kernel's 
COPYING file
-md5sums['bbb461211a33b134d42ed5ee802b37ff'] = 'LGPLv2.1'
-md5sums['7fbc338309ac38fefcd64b04bb903e34'] = 'LGPLv2.1'
-md5sums['4fbd65380cdd255951079008b364516c'] = 'LGPLv2.1'
-md5sums['2d5025d4aa3495befef8f17206a5b0a1'] = 'LGPLv2.1'
-md5sums['fbc093901857fcd118f065f900982c24'] = 'LGPLv2.1'
-md5sums['a6f89e2100d9b6cdffcea4f398e37343'] = 'LGPLv2.1'
-md5sums['d8045f3b8f929c1cb29a1e3fd737b499'] = 'LGPLv2.1'
-md5sums['fad9b3332be894bab9bc501572864b29'] = 'LGPLv2.1'
-md5sums['3bf50002aefd002f49e7bb854063f7e7'] = 'LGPLv2'
-md5sums['9f604d8a4f8e74f4f5140845a21b6674'] = 'LGPLv2'
-md5sums['5f30f0716dfdd0d91eb439ebec522ec2'] = 'LGPLv2'
-md5sums['55ca817ccb7d5b5b66355690e9abc605'] = 'LGPLv2'
-md5sums['252890d9eee26aab7b432e8b8a616475'] = 'LGPLv2'
-md5sums['3214f080875748938ba060314b4f727d'] = 'LGPLv2'
-md5sums['db979804f025cf55aabec7129cb671ed'] = 'LGPLv2'
-md5sums['d32239bcb673463ab874e80d47fae504'] = 'GPLv3'
-md5sums['f27defe1e96c2e1ecd4e0c9be8967949'] = 'GPLv3'
-md5sums['6a6a8e020838b23406c81b19c1d46df6'] = 'LGPLv3'
-md5sums['3b83ef96387f14655fc854ddc3c6bd57'] = 'Apache-2.0'
-md5sums['385c55653886acac3821999a3ccd17b3'] = 'Artistic-1.0 | GPL-2.0' # 
some perl modules
-md5sums['54c7042be62e169199200bc6477f04d1'] = 'BSD-3-Clause'
-md5sums['bfe1f75d606912a4111c90743d6c7325'] = 'MPL-1.1'
+
+# Read license md5sums from csv file
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+for path in (d.getVar('BBPATH').split(':')
++ [os.path.join(scripts_path, '..', '..')]):
+csv_path = os.path.join(path, 'lib', 'recipetool', 'licenses.csv')
+if os.path.isfile(csv_path):
+with open(csv_path, newline='') as csv_file:
+fieldnames = ['md5sum', 'license']
+reader = csv.DictReader(csv_file, delimiter=',', 
fieldnames=fieldnames)
+for row in reader:
+md5sums[row['md5sum']] = row['license']
+
 return md5sums
 
 def crunch_license(licfile):
diff --git a/scripts/lib/recipetool/licenses.csv 
b/scripts/lib/recipetool/licenses.csv
new file mode 100644
index 00..0d3fb0607b
--- /dev/null
+++ b/scripts/lib/recipetool/licenses.csv
@@ -0,0 +1,37 @@
+0636e73ff0215e8d672dc4c32c317bb3,GPLv2
+12f884d2ae1ff87c09e5b7ccc2c4ca7e,GPLv2
+18810669f13b87348459e611d31ab760,GPLv2
+252890d9eee26aab7b432e8b8a616475,LGPLv2
+2d5025d4aa3495befef8f17206a5b0a1,LGPLv2.1
+3214f080875748938ba060314b4f727d,LGPLv2
+385c55653886acac3821999a3ccd17b3,Artistic-1.0 | GPL-2.0
+393a5ca445f6965873eca0259a17f833,GPLv2
+3b83ef96387f14655fc854ddc3c6bd57,Apache-2.0
+3bf50002aefd002f49e7bb854063f7e7,LGPLv2
+4325afd396febcb659c36b49533135d4,GPLv2
+4fbd65380cdd255951079008b364516c,LGPLv2.1

[OE-core] [PATCH 05/12] recipetool: ignore empty license files

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 893980a5ce..87b25ebc7e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1126,9 +1126,10 @@ def guess_license(srctree, d):
 license = md5sums.get(md5value, None)
 if not license:
 license, crunched_md5, lictext = crunch_license(licfile)
-if not license:
+if lictext and not license:
 license = 'Unknown'
-licenses.append((license, os.path.relpath(licfile, srctree), md5value))
+if license:
+licenses.append((license, os.path.relpath(licfile, srctree), 
md5value))
 
 # FIXME should we grab at least one source file with a license header and 
add that too?
 
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156747): 
https://lists.openembedded.org/g/openembedded-core/message/156747
Mute This Topic: https://lists.openembedded.org/mt/86165225/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 01/12] npm: Add support for EXTRA_OENPM arguments

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Add support for EXTRA_OENPM arguments to set node-gyp variables for
example. This allows use of shared librariess, avoid download
of external sources or build from source.

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 meta/classes/npm.bbclass | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 91a2321116..477b40d921 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -22,6 +22,8 @@ inherit python3native
 DEPENDS:prepend = "nodejs-native "
 RDEPENDS:${PN}:append:class-target = " nodejs"
 
+EXTRA_OENPM = ""
+
 NPM_INSTALL_DEV ?= "0"
 
 def npm_target_arch_map(target_arch):
@@ -260,7 +262,8 @@ python npm_do_compile() {
 
 # Pack and install the main package
 tarball = npm_pack(env, d.getVar("NPM_PACKAGE"), tmpdir)
-env.run("npm install %s" % shlex.quote(tarball), args=args, 
configs=configs)
+cmd = "npm install %s %s" % (shlex.quote(tarball), 
d.getVar("EXTRA_OENPM"))
+env.run(cmd, args=args, configs=configs)
 }
 
 npm_do_install() {
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156748): 
https://lists.openembedded.org/g/openembedded-core/message/156748
Mute This Topic: https://lists.openembedded.org/mt/86141407/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 04/12] recipetool: Change default paramter fallback_licenses of function split_pkg_licenses from None to []

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 83cf25d9b7..893980a5ce 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1134,7 +1134,7 @@ def guess_license(srctree, d):
 
 return licenses
 
-def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, 
pn='${PN}'):
+def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=[], 
pn='${PN}'):
 """
 Given a list of (license, path, md5sum) as returned by guess_license(),
 a dict of package name to path mappings, write out a set of
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156746): 
https://lists.openembedded.org/g/openembedded-core/message/156746
Mute This Topic: https://lists.openembedded.org/mt/86165224/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH 03/12] recipetool: Skip common source files in guess_license

2021-10-08 Thread Stefan Herbrechtsmeier

From: Stefan Herbrechtsmeier 

Signed-off-by: Stefan Herbrechtsmeier 
---

(no changes since v1)

 scripts/lib/recipetool/create.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index e88a4253da..83cf25d9b7 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1114,6 +1114,8 @@ def guess_license(srctree, d):
 licfiles = []
 for root, dirs, files in os.walk(srctree):
 for fn in files:
+if fn.endswith(".html") or fn.endswith(".js") or 
fn.endswith(".json") or fn.endswith(".svg") or fn.endswith(".ts"):
+continue
 for spec in licspecs:
 if fnmatch.fnmatch(fn, spec):
 fullpath = os.path.join(root, fn)
-- 
2.20.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156745): 
https://lists.openembedded.org/g/openembedded-core/message/156745
Mute This Topic: https://lists.openembedded.org/mt/86165223/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] Improve npm support to run build scripts

2021-10-08 Thread Stefan Herbrechtsmeier


Hi Tim,

Am 08.10.2021 um 07:27 schrieb Tim Orling:
We have discussed the severe impact on parsing time with 100k recipes. I 
personally worked on a node.js project with that many dependencies (but 
not an OE/Yocto deployment). Parsing time is already a pain point for 
1000s of recipes.


How do you track the version updates, check the CVE status and track the 
licenses?


How do you update a package inside the dependency tree?

Regards
  Stefan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156744): 
https://lists.openembedded.org/g/openembedded-core/message/156744
Mute This Topic: https://lists.openembedded.org/mt/86089523/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

44 matches

Mail list logo