Hi Chris, Thanks for your quick reply. I can't wait to see the solution for option three, would love to see the .diff for option 2 and... well I don't care much for ponies :-)
Regards, Willem. On Mon, Nov 21, 2011 at 8:41 PM, Chris St. Pierre <[email protected]> wrote: > Willem-- > > I ran into this issue, too. The long and the short of it is that the > HP packages are screwed up; for instance, the cpqacuxe package has a > filename that indicates that it's noarch, but in fact it's a i386 > package, so grinder does the wrong thing (or maybe the too-right > thing) when trying to build a list of packages that are orphaned. > > There are three options: > > 1. Get HP to fix their packages. And a pony, and the moon on a stick. > > 2. Grinder has an option that turns off orphaned package cleanup, but > it's not currently exposed to the Pulp layer. That could be done. > > 3. A far more complex approach that deals with inconsistent/broken > metadata and packages in a blindingly clever way. > > I believe there's a ticket open and Pradeep is looking at something > along the lines of option 3. We currently only have one HP box > looking at Pulp, so this isn't a huge problem for us; if it becomes > more of an issue before there's a proper fix, I'll probably contribute > a patch to do option 2. Option 1 is obviously the best solution in > the same way that riding a unicorn to work is the best solution for > commuting. > > TL;DR: There is no workaround currently, but the Pulp devs are aware > of the issue. > > On Mon, Nov 21, 2011 at 1:16 PM, Willem Bos <[email protected]> wrote: >> Hi all, >> >> I'm trying to sync a HP hosted repository (downloads.linux.hp.com). >> Most of the packages are processed fine but a few are constantly >> orphaned by Grinder after being downloaded. A few of them do show up >> in /var/lib/pulp/packages/ but are not linked to a repo… I tried >> creating the repo with '--feed=http://...' and also with an added >> '--preserve_metada'. >> >> Then I downloaded the packages with reposync, created metadata with >> createrepo and placed this directory on a internal web server. With >> the same result as mentioned above. >> >> Finally, I tried '--feed=file:///...' (essentially bypassing Grinder) >> and all packages were assigned correctly. >> >> See below for detailed actions. >> >> Any suggestions? >> >> Regards, >> Willem. >> >> ## FIRST ATTEMPT: ## >> >> # pulp-admin repo create \ >> --id=hp-test \ >> --preserve_metadata \ >> --feed=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/ >> Successfully created repository [ hp-test ] >> >> # pulp-admin repo sync --id=hp-test --threads=1 --foreground >> Sync for repository hp-test started >> Sync: Finished >> 46/46 new items downloaded >> 0/46 existing items processed >> Item Details: >> RPMs: 46/46 >> >> # pulp-admin package search -f "^hp-.*" >> +------------------------------------------+ >> Package Information >> +------------------------------------------+ >> Name EVRA Filename >> Repositories >> hp-tg3 0:3.116j-3.src hp-tg3-3.116j-3.src.rpm >> hp-test >> hp-smh-templates 0:8.7.0.10-10.noarch >> hp-smh-templates-8.7.0.10-10.noarch.rpm hp-test >> hp-smh-templates 0:8.6.0.14-16.noarch >> hp-smh-templates-8.6.0.14-16.noarch.rpm hp-test >> ... >> >> # pulp-admin package search -f "^hp-snmp.*" >> error: No packages found. >> >> # ls -1d /var/lib/pulp/packages/hp-snmp* >> /var/lib/pulp/packages/hp-snmp-agents >> >> # cat /var/log/pulp/grinder.log >> ... >> 2011-11-20 21:13:35,155 13269:139726048700160: grinder.YumInfo:INFO: >> activeobject:160 46 packages have been marked to be fetched >> ... >> 2011-11-20 21:13:37,708 13269:139726048700160: >> grinder.ParallelFetch:INFO: ParallelFetch:177 46 items are marked to >> be fetched >> ... >> 2011-11-20 21:14:53,340 13269:139725956445952: grinder.BaseFetch:INFO: >> activeobject:160 Symlink missing in repo directory. Creating link >> /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64//hp-snmp-agents-8.6.2.3-9.x86_64.rpm >> to >> ../../../../../../../../packages/hp-snmp-agents/8.6.2.3/9/x86_64/e96/hp-snmp-agents-8.6.2.3-9.x86_64.rpm >> ... >> 2011-11-20 21:16:07,138 13269:139726048700160: >> grinder.ParallelFetch:INFO: ParallelFetch:238 ParallelFetch: 46 items >> successfully processed, 33 downloaded, 0 items had errors >> ... >> 2011-11-20 21:16:07,140 13269:139726048700160: grinder.RepoFetch:INFO: >> RepoFetch:154 Cleaning any orphaned packages.. >> ... >> 2011-11-20 21:16:07,143 13269:139726048700160: grinder.RepoFetch:INFO: >> RepoFetch:190 Removing orphan package: >> /var/lib/pulp//repos/SDR/downloads/proliantsupportpack/redhat/6/packages/x86_64/hp-snmp-agents-8.6.2.3-9.x86_64.rpm >> ... >> 2011-11-20 21:16:07,144 13269:139726048700160: grinder.RepoFetch:INFO: >> RepoFetch:159 Processed <46> items in [154] seconds. Report: 46 >> successes, 33 downloads, 0 errors >> 2011-11-20 21:16: >> ... >> >> # pulp-admin repo delete --id=hp-test >> Successful deleted repository [ hp-test ] >> >> # rm -r >> /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpvca,hpdiags,hp-health,hp-snmp-agents} >> >> ## SECOND ATTEMPT: ## >> >> # cat > /etc/yum.repos.d/hp-test.repo << EOF >> [hp-test] >> name=hp-test >> baseurl=http://downloads.linux.hp.com/SDR/downloads/proliantsupportpack/redhat/6/packages/i686/ >> enabled=1 >> gpgcheck=0 >> EOF >> >> # cd /var/www/html >> >> # reposync --repoid=hp-test . >> ... >> [hp-test: 11 of 31 ] Downloading >> hp-smh-templates-8.6.0.14-16.noarch.rpm >> hp-smh-templates-8.6.0.14-16.noarch.rpm | 3.4 >> MB 00:02 >> [hp-test: 12 of 31 ] Downloading >> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm >> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm | 1.7 >> MB 00:00 >> [hp-test: 13 of 31 ] Downloading >> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm >> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm | 1.7 >> MB 00:00 >> ... >> >> # cd hp-test >> >> # createrepo . >> Spawning worker 0 with 31 pkgs >> ... >> >> # zcat repodata/*-primary.xml.gz | grep "<name>hp-snmp" >> <name>hp-snmp-agents</name> >> <name>hp-snmp-agents</name> >> >> # pulp-admin repo create \ >> --id=hp-test \ >> --feed=http://n004.localdomain/hp-test/ >> >> # pulp-admin repo sync --id=hp-test --threads=1 --foreground >> >> # pulp-admin package search -f "^hp-snmp.*" >> <output identical to first attempt> >> >> # cat /var/log/pulp/grinder.log >> <output identical to first attempt> >> >> # ls -1d /var/lib/pulp/packages/hp-* >> <output identical to first attempt> >> >> # pulp-admin repo delete --id=hp-test >> Successful deleted repository [ hp-test ] >> >> # rm -r >> /var/lib/pulp/packages/{cpqacuxe,hpacucli,hpdiags,hp-health,hp-snmp-agents} >> >> ## THIRD ATTEMPT: ## >> >> # pulp-admin repo create \ >> --id=hp-test \ >> --feed=file:///var/www/html/hp-test/ >> >> # pulp-admin repo sync --id=hp-test --threads=1 --foreground >> pulp-admin repo sync --id=hp-test --threads=1 --foreground >> Sync for repository hp-test started >> Sync: Finished >> 31/31 new items downloaded >> 0/31 existing items processed >> Item Details: >> RPMs: 31/31 >> >> # pulp-admin package search -f "^hp-.*" >> +------------------------------------------+ >> Package Information >> +------------------------------------------+ >> Name EVRA Filename >> Repositories >> hp-snmp-agents 0:8.7.0.23-16.i386 >> hp-snmp-agents-8.7.0.23-16.rhel6.i386.rpm hp-test >> hp-snmp-agents 0:8.6.2.3-8.i386 >> hp-snmp-agents-8.6.2.3-8.rhel6.i386.rpm hp-test >> ... >> >> _______________________________________________ >> Pulp-list mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/pulp-list >> > > > > -- > Chris St. Pierre > _______________________________________________ Pulp-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-list
