Re: [OE-core] [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer

[Richard Purdie]

On Fri, 2017-09-01 at 16:04 +0300, Alexander Kanavin wrote:
> On 08/31/2017 09:46 PM, Haris Okanovic wrote:
> > 
> > Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist
> > on
> > `bitbake package-index`, which is index of files provided by each
> > IPK package in the feed. It's useful for figuring out which package
> > provides a particular file/program/library/etc.
> > 
> > Disabled by default since generating a filelist involves reading
> > the
> > payload of every package in the feed, a time and IO intensive
> > operation
> > many users won't want to run. Those who do may flip this switch.
> > 
> > Testing:
> >   * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and
> > verified
> > no Packages.filelist are generated.
> >   * Built with PACKAGE_ENABLE_FILELIST="1" and verified each
> > subfeed
> > has Packages.filelist; took ~3min longer for 8,200 IPKs.
> 
>
> The only way to discover that this option exists is to read the code
> for OpkgIndexer. And it's specific to opkg, which is not reflected in
> the option name at all.
> 
> Can you instead just place the index generation into a custom recipe 
> that depends on package-index recipe?

I'm going to disagree with that, I don't want a separate recipe for
this and I'm not sure that makes sense.

I'm ok with adding an option but we need to come up with a better
namespace. How about OPKG_PACKAGE_INDEX_FILELIST_GEN?

(says its package index and opkg specific)

Cheers,

Richard


-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer

[Alexander Kanavin]

On 08/31/2017 09:46 PM, Haris Okanovic wrote:

Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
`bitbake package-index`, which is index of files provided by each
IPK package in the feed. It's useful for figuring out which package
provides a particular file/program/library/etc.

Disabled by default since generating a filelist involves reading the
payload of every package in the feed, a time and IO intensive operation
many users won't want to run. Those who do may flip this switch.

Testing:
  * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
no Packages.filelist are generated.
  * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
has Packages.filelist; took ~3min longer for 8,200 IPKs.


The only way to discover that this option exists is to read the code for 
OpkgIndexer. And it's specific to opkg, which is not reflected in the 
option name at all.


Can you instead just place the index generation into a custom recipe 
that depends on package-index recipe?



Alex
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer

[Haris Okanovic]

Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
`bitbake package-index`, which is index of files provided by each
IPK package in the feed. It's useful for figuring out which package
provides a particular file/program/library/etc.

Disabled by default since generating a filelist involves reading the
payload of every package in the feed, a time and IO intensive operation
many users won't want to run. Those who do may flip this switch.

Testing:
 * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
   no Packages.filelist are generated.
 * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
   has Packages.filelist; took ~3min longer for 8,200 IPKs.

Signed-off-by: Haris Okanovic 
---
 meta/lib/oe/package_manager.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b4b359a8c6..5e1fc48500 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
 else:
 signer = None
 
+enable_filelist = 
bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
+
 if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
 open(os.path.join(self.deploy_dir, "Packages"), "w").close()
 
@@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
 pkgs_dir = os.path.join(self.deploy_dir, arch)
 pkgs_file = os.path.join(pkgs_dir, "Packages")
 
+filelist_cmd = ""
+if enable_filelist:
+filelist_cmd = '-l %s.filelist' % (pkgs_file)
+
 if not os.path.isdir(pkgs_dir):
 continue
 
 if not os.path.exists(pkgs_file):
 open(pkgs_file, "w").close()
 
-index_cmds.add('%s -r %s -p %s -m %s' %
-  (opkg_index_cmd, pkgs_file, pkgs_file, 
pkgs_dir))
+index_cmds.add('%s -r %s -p %s -m %s %s' %
+  (opkg_index_cmd, pkgs_file, pkgs_file, 
filelist_cmd, pkgs_dir))
 
 index_sign_files.add(pkgs_file)
 
-- 
2.13.2

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer

[Haris Okanovic]

Why not use files-in-package.txt reports in buildhistory?
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5870


Practical reason: files-in-package.txt is an OE-ism. Our opkg-based 
distribution provides software feeds built outside of OE. We use 
opkg-utils to generate feed metadata like OE, so it makes sense to reuse 
logic instead of rewriting it. Others may be in the same position.


Philosophical reason: I prefer to build the file list (and index) from 
actual packages (I.e. exactly what we're shipping) instead of OE's 
environment. This makes the file list and index more easily reproducible 
and insulates that process from OE bugs.


-- Haris


On 05/19/2017 12:29 PM, Martin Jansa wrote:

On Fri, May 19, 2017 at 10:01:21AM -0500, Haris Okanovic wrote:

Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
`bitbake package-index`, which is index of files provided by each
IPK package in the feed. It's useful for figuring out which package
provides a particular file/program/library/etc.

Disabled by default since generating a filelist involves reading the
payload of every package in the feed, a time and IO intensive operation
many users won't want to run. Those who do may flip this switch.

Testing:
 * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
   no Packages.filelist are generated.
 * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
   has Packages.filelist; took ~3min longer for 8,200 IPKs.

Signed-off-by: Haris Okanovic 


Why not use files-in-package.txt reports in buildhistory?
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5870

That should work for any package backend and better to generate it once
in one place.


---
 meta/lib/oe/package_manager.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b4b359a8c6..5e1fc48500 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
 else:
 signer = None

+enable_filelist = bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', 
True) or "False")
+
 if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
 open(os.path.join(self.deploy_dir, "Packages"), "w").close()

@@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
 pkgs_dir = os.path.join(self.deploy_dir, arch)
 pkgs_file = os.path.join(pkgs_dir, "Packages")

+filelist_cmd = ""
+if enable_filelist:
+filelist_cmd = '-l %s.filelist' % (pkgs_file)
+
 if not os.path.isdir(pkgs_dir):
 continue

 if not os.path.exists(pkgs_file):
 open(pkgs_file, "w").close()

-index_cmds.add('%s -r %s -p %s -m %s' %
-  (opkg_index_cmd, pkgs_file, pkgs_file, 
pkgs_dir))
+index_cmds.add('%s -r %s -p %s -m %s %s' %
+  (opkg_index_cmd, pkgs_file, pkgs_file, 
filelist_cmd, pkgs_dir))

 index_sign_files.add(pkgs_file)

--
2.12.1

--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core





--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer

[Martin Jansa]

On Fri, May 19, 2017 at 10:01:21AM -0500, Haris Okanovic wrote:
> Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
> `bitbake package-index`, which is index of files provided by each
> IPK package in the feed. It's useful for figuring out which package
> provides a particular file/program/library/etc.
> 
> Disabled by default since generating a filelist involves reading the
> payload of every package in the feed, a time and IO intensive operation
> many users won't want to run. Those who do may flip this switch.
> 
> Testing:
>  * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
>no Packages.filelist are generated.
>  * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
>has Packages.filelist; took ~3min longer for 8,200 IPKs.
> 
> Signed-off-by: Haris Okanovic 

Why not use files-in-package.txt reports in buildhistory?
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5870

That should work for any package backend and better to generate it once
in one place.

> ---
>  meta/lib/oe/package_manager.py | 10 --
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index b4b359a8c6..5e1fc48500 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
>  else:
>  signer = None
>  
> +enable_filelist = 
> bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
> +
>  if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
>  open(os.path.join(self.deploy_dir, "Packages"), "w").close()
>  
> @@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
>  pkgs_dir = os.path.join(self.deploy_dir, arch)
>  pkgs_file = os.path.join(pkgs_dir, "Packages")
>  
> +filelist_cmd = ""
> +if enable_filelist:
> +filelist_cmd = '-l %s.filelist' % (pkgs_file)
> +
>  if not os.path.isdir(pkgs_dir):
>  continue
>  
>  if not os.path.exists(pkgs_file):
>  open(pkgs_file, "w").close()
>  
> -index_cmds.add('%s -r %s -p %s -m %s' %
> -  (opkg_index_cmd, pkgs_file, pkgs_file, 
> pkgs_dir))
> +index_cmds.add('%s -r %s -p %s -m %s %s' %
> +  (opkg_index_cmd, pkgs_file, pkgs_file, 
> filelist_cmd, pkgs_dir))
>  
>  index_sign_files.add(pkgs_file)
>  
> -- 
> 2.12.1
> 
> -- 
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com


signature.asc
Description: Digital signature
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer

[Haris Okanovic]

Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
`bitbake package-index`, which is index of files provided by each
IPK package in the feed. It's useful for figuring out which package
provides a particular file/program/library/etc.

Disabled by default since generating a filelist involves reading the
payload of every package in the feed, a time and IO intensive operation
many users won't want to run. Those who do may flip this switch.

Testing:
 * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
   no Packages.filelist are generated.
 * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
   has Packages.filelist; took ~3min longer for 8,200 IPKs.

Signed-off-by: Haris Okanovic 
---
 meta/lib/oe/package_manager.py | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b4b359a8c6..5e1fc48500 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
 else:
 signer = None
 
+enable_filelist = 
bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
+
 if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
 open(os.path.join(self.deploy_dir, "Packages"), "w").close()
 
@@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
 pkgs_dir = os.path.join(self.deploy_dir, arch)
 pkgs_file = os.path.join(pkgs_dir, "Packages")
 
+filelist_cmd = ""
+if enable_filelist:
+filelist_cmd = '-l %s.filelist' % (pkgs_file)
+
 if not os.path.isdir(pkgs_dir):
 continue
 
 if not os.path.exists(pkgs_file):
 open(pkgs_file, "w").close()
 
-index_cmds.add('%s -r %s -p %s -m %s' %
-  (opkg_index_cmd, pkgs_file, pkgs_file, 
pkgs_dir))
+index_cmds.add('%s -r %s -p %s -m %s %s' %
+  (opkg_index_cmd, pkgs_file, pkgs_file, 
filelist_cmd, pkgs_dir))
 
 index_sign_files.add(pkgs_file)
 
-- 
2.12.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core