Willem-- I ran into this issue, too. The long and the short of it is that the HP packages are screwed up; for instance, the cpqacuxe package has a filename that indicates that it's noarch, but in fact it's a i386 package, so grinder does the wrong thing (or maybe the too-right thing) when trying to build a list of packages that are orphaned.
There are three options: 1. Get HP to fix their packages. And a pony, and the moon on a stick. 2. Grinder has an option that turns off orphaned package cleanup, but it's not currently exposed to the Pulp layer. That could be done. 3. A far more complex approach that deals with inconsistent/broken metadata and packages in a blindingly clever way. I believe there's a ticket open and Pradeep is looking at something along the lines of option 3. We currently only have one HP box looking at Pulp, so this isn't a huge problem for us; if it becomes more of an issue before there's a proper fix, I'll probably contribute a patch to do option 2. Option 1 is obviously the best solution in the same way that riding a unicorn to work is the best solution for commuting. TL;DR: There is no workaround currently, but the Pulp devs are aware of the issue. On Mon, Nov 21, 2011 at 1:16 PM, Willem Bos <[email protected]> wrote: > Hi all, > > I'm trying to sync a HP hosted repository (downloads.linux.hp.com). > Most of the packages are processed fine but a few are constantly > orphaned by Grinder after being downloaded. A few of them do show up > in /var/lib/pulp/packages/ but are not linked to a repo… I tried > creating the repo with '--feed=http://...' and also with an added > '--preserve_metada'. > > Then I downloaded the packages with reposync, created metadata with > createrepo and placed this directory on a internal web server. With > the same result as mentioned above. > > Finally, I tried '--feed=file:///...' (essentially bypassing Grinder) > and all packages were assigned correctly. > > See below for detailed actions. > > Any suggestions? > > Regards, > Willem. > > ## FIRST ATTEMPT: ## > > # pulp-admin repo create \ > --id=hp-test \ > --preserve_metadata \ > --feed=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/ > Successfully created repository [ hp-test ] > > # pulp-admin repo sync --id=hp-test --threads=1 --foreground > Sync for repository hp-test started > Sync: Finished > 46/46 new items downloaded > 0/46 existing items processed > Item Details: > RPMs: 46/46 > > # pulp-admin package search -f "^hp-.*" > +------------------------------------------+ > Package Information > +------------------------------------------+ > Name EVRA Filename > Repositories > hp-tg3 0:3.116j-3.src hp-tg3-3.116j-3.src.rpm > hp-test > hp-smh-templates 0:8.7.0.10-10.noarch > hp-smh-templates-8.7.0.10-10.noarch.rpm hp-test > hp-smh-templates 0:8.6.0.14-16.noarch > hp-smh-templates-8.6.0.14-16.noarch.rpm hp-test > ... > > # pulp-admin package search -f "^hp-snmp.*" > error: No packages found. > > # ls -1d /var/lib/pulp/packages/hp-snmp* > /var/lib/pulp/packages/hp-snmp-agents > > # cat /var/log/pulp/grinder.log > ... > 2011-11-20 21:13:35,155 13269:139726048700160: grinder.YumInfo:INFO: > activeobject:160 46 packages have been marked to be fetched > ... > 2011-11-20 21:13:37,708 13269:139726048700160: > grinder.ParallelFetch:INFO: ParallelFetch:177 46 items are marked to > be fetched > ... > 2011-11-20 21:14:53,340 13269:139725956445952: grinder.BaseFetch:INFO: > activeobject:160 Symlink missing in repo directory. Creating link > /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64//hp-snmp-agents-8.6.2.3-9.x86_64.rpm > to > ../../../../../../../../packages/hp-snmp-agents/8.6.2.3/9/x86_64/e96/hp-snmp-agents-8.6.2.3-9.x86_64.rpm > ... > 2011-11-20 21:16:07,138 13269:139726048700160: > grinder.ParallelFetch:INFO: ParallelFetch:238 ParallelFetch: 46 items > successfully processed, 33 downloaded, 0 items had errors > ... > 2011-11-20 21:16:07,140 13269:139726048700160: grinder.RepoFetch:INFO: > RepoFetch:154 Cleaning any orphaned packages.. > ... > 2011-11-20 21:16:07,143 13269:139726048700160: grinder.RepoFetch:INFO: > RepoFetch:190 Removing orphan package: > /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/hp-snmp-agents-8.6.2.3-9.x86_64.rpm > ... > 2011-11-20 21:16:07,144 13269:139726048700160: grinder.RepoFetch:INFO: > RepoFetch:159 Processed <46> items in [154] seconds. Report: 46 > successes, 33 downloads, 0 errors > 2011-11-20 21:16: > ... > > # pulp-admin repo delete --id=hp-test > Successful deleted repository [ hp-test ] > > # rm -r > /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpvca,hpdiags,hp-health,hp-snmp-agents} > > ## SECOND ATTEMPT: ## > > # cat > /etc/yum.repos.d/hp-test.repo << EOF > [hp-test] > name=hp-test > baseurl=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/i686/ > enabled=1 > gpgcheck=0 > EOF > > # cd /var/www/html > > # reposync --repoid=hp-test . > ... > [hp-test: 11 of 31 ] Downloading hp-smh-templates-8.6.0.14-16.noarch.rpm > hp-smh-templates-8.6.0.14-16.noarch.rpm | 3.4 > MB 00:02 > [hp-test: 12 of 31 ] Downloading hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm > hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm | 1.7 > MB 00:00 > [hp-test: 13 of 31 ] Downloading > hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm > hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm | 1.7 > MB 00:00 > ... > > # cd hp-test > > # createrepo . > Spawning worker 0 with 31 pkgs > ... > > # zcat repodata/*-primary.xml.gz | grep "<name>hp-snmp" > <name>hp-snmp-agents</name> > <name>hp-snmp-agents</name> > > # pulp-admin repo create \ > --id=hp-test \ > --feed=http://n004.localdomain/hp-test/ > > # pulp-admin repo sync --id=hp-test --threads=1 --foreground > > # pulp-admin package search -f "^hp-snmp.*" > <output identical to first attempt> > > # cat /var/log/pulp/grinder.log > <output identical to first attempt> > > # ls -1d /var/lib/pulp/packages/hp-* > <output identical to first attempt> > > # pulp-admin repo delete --id=hp-test > Successful deleted repository [ hp-test ] > > # rm -r > /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpdiags,hp-health,hp-snmp-agents} > > ## THIRD ATTEMPT: ## > > # pulp-admin repo create \ > --id=hp-test \ > --feed=file:///var/www/html/hp-test/ > > # pulp-admin repo sync --id=hp-test --threads=1 --foreground > pulp-admin repo sync --id=hp-test --threads=1 --foreground > Sync for repository hp-test started > Sync: Finished > 31/31 new items downloaded > 0/31 existing items processed > Item Details: > RPMs: 31/31 > > # pulp-admin package search -f "^hp-.*" > +------------------------------------------+ > Package Information > +------------------------------------------+ > Name EVRA Filename > Repositories > hp-snmp-agents 0:8.7.0.23-16.i386 > hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm hp-test > hp-snmp-agents 0:8.6.2.3-8.i386 > hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm hp-test > ... > > _______________________________________________ > Pulp-list mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/pulp-list > -- Chris St. Pierre _______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
