Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 28, 2016, at 4:59 PM, Mark Hatle  wrote:
> 
> 
> AFAIK, only openssl 1.0.x supports the FIPS module.  There are a few folks
> looking at private implementations of the older module with OpenSSL 1.1.x, but
> definitely not official.
> 

Yes, only 1.0.x atm.

The development timeline from openssl-fops 3.0 is here:
https://www.safelogic.com/category/fips-140/

The path forward with openssl in RPM is murky: LibreSSL? BoringSSL? NanoSSL?

-ENOCLUE

73 de Jeff

Re: Making rpm5 work with dnf

[Mark Hatle]

On 12/28/16 8:25 AM, Jeffrey Johnson wrote:
> 
>> On Dec 28, 2016, at 7:02 AM, Alexander Kanavin 
>>  wrote:
>>
>> On 12/27/2016 08:00 PM, Jeffrey Johnson wrote:
>>
>>> FYI: most of the openssl-1.1.0 port in rpm is now done.
>>>
>>> I’ve done “Do no harm testing.” with openssl-1.0.2j, will get to
>>> detailed openssl-1.1.0 testing as soon as I see a platform that
>>> distributes with openssl-1.1.0 (likely Fedora 26, not yet Fedora 25).
>>
>> Thanks! If you need a platform for testing, then debian testing (stretch) 
>> does ship with both openssl 1.1 and 1.0. You can have both libraries 
>> installed at the same time, but development packages mutually exclude each 
>> other. Despite the name, debian testing is fairly stable.
>>
> 
> Two versions of openssl installed isn’t what is hard, setting up a platform
> with one version of openssl well integrated is what is hard.
> 
> RPM links many libraries, some of which link openssl, and recompiling
> all RPM prerequisites to use a single version of openssl is very time 
> consuming.
> 
> Then there are issues of how openssl is installed: e.g. testing ECDSA
> usually requires rebuilding openssl and reading removed curves.
> 
> Then there is openssl-fips which RPM5 uses (or used, I’ve not checked 
> recently).
>  openssl-fips-2.0.13 functions with openssl-1.0.x, but not openssl-1.1.x. 
> FIPS 140-2
> seems to be quite a mess these days, sigh. Getting all the HMAC’s installed 
> and
> verified for FIPS 140-2 is always a chore.

AFAIK, only openssl 1.0.x supports the FIPS module.  There are a few folks
looking at private implementations of the older module with OpenSSL 1.1.x, but
definitely not official.

--Mark

> Meanwhile I have checked that RPM builds/links against openssl-1.1.0, and
> the code in ramie/rpmssl.c has rather simple usages of openssl.
> 
> hth
> 
> 73 de Jeff
> 
> __
> RPM Package Managerhttp://rpm5.org
> Developer Communication Listrpm-devel@rpm5.org
> 

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 13, 2016, at 11:09 AM, Jeffrey Johnson  wrote:
> 
>> 
>> On Dec 13, 2016, at 8:34 AM, Alexander Kanavin 
>> > > wrote:
>> 
>> On 12/09/2016 06:11 PM, Jeffrey Johnson wrote:
>>> Good: you appear to have +beecrypt (for digests) and +libtomcrypt
>>> (recommended
>>> because of ECDSA and stable) and +openssl (what Yocto wants last I heard).
>> 
>> Speaking of openssl, it was actually quietly being disabled, because I 
>> didn't have the -dev package installed. After the openssl headers appeared, 
>> there's a bunch of new errors:
>> 
>> rpmssl.c: In function ‘rpmsslGenerate’:
>> rpmssl.c:629:29: error: dereferencing pointer to incomplete type ‘EVP_PKEY 
>> {aka struct evp_pkey_st}’
>>  if (EVP_PKEY_type(ssl->pkey->type) == EVP_PKEY_EC) {
>> ^~
>> What this means is that rpm needs to be ported to openssl 1.1 - they made a 
>> lot of previously open structures opaque in that version, with accessor 
>> functions (and maybe there are other issues).
>> 
> 
> Likely: Fedora/Centos tend to back port fixes rather than upgrade versions:
> 
>   $ rpm -q openssl
>   openssl-1.0.2j-1.fc24.x86_64
> 
> FWIW, the versions of pre-reqs are listed in INSTALL.
> 

FYI: most of the openssl-1.1.0 port in rpm is now done.

I’ve done “Do no harm testing.” with openssl-1.0.2j, will get to
detailed openssl-1.1.0 testing as soon as I see a platform that
distributes with openssl-1.1.0 (likely Fedora 26, not yet Fedora 25).

AFAIK, what remains on my todo++ list is
1) the amalgamated mongo-c-driver needs to be updated
I disabled openssl support with openssl-1.1.0 for now.
2) libtpm needs to be updated
Nothing in rpm needs/uses libtpm atm: libtpm is included with
rpm for scan.coverity.com static analysis and portability 
canary testing.
3_ implicit dependencies in various embeddings like paso-mqtt and ruby 
etc.
None of these are critical to RPM.

73 de Jeff

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 13, 2016, at 8:34 AM, Alexander Kanavin 
>  wrote:
> 
> On 12/09/2016 06:11 PM, Jeffrey Johnson wrote:
>> Good: you appear to have +beecrypt (for digests) and +libtomcrypt
>> (recommended
>> because of ECDSA and stable) and +openssl (what Yocto wants last I heard).
> 
> Speaking of openssl, it was actually quietly being disabled, because I didn't 
> have the -dev package installed. After the openssl headers appeared, there's 
> a bunch of new errors:
> 
> rpmssl.c: In function ‘rpmsslGenerate’:
> rpmssl.c:629:29: error: dereferencing pointer to incomplete type ‘EVP_PKEY 
> {aka struct evp_pkey_st}’
>  if (EVP_PKEY_type(ssl->pkey->type) == EVP_PKEY_EC) {
> ^~
> What this means is that rpm needs to be ported to openssl 1.1 - they made a 
> lot of previously open structures opaque in that version, with accessor 
> functions (and maybe there are other issues).
> 

Likely: Fedora/Centos tend to back port fixes rather than upgrade versions:

$ rpm -q openssl
openssl-1.0.2j-1.fc24.x86_64

FWIW, the versions of pre-reqs are listed in INSTALL.

> I'm not sure why yocto wants openssl support, Mark should know that.
> 

I’d recommend libtomcrypt because both complete and slowly changing. There
are now github repositories as well.

73 de Jeff

> Alex
> 
> __
> RPM Package Managerhttp://rpm5.org
> Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/09/2016 06:11 PM, Jeffrey Johnson wrote:

Good: you appear to have +beecrypt (for digests) and +libtomcrypt
(recommended
because of ECDSA and stable) and +openssl (what Yocto wants last I heard).


Speaking of openssl, it was actually quietly being disabled, because I 
didn't have the -dev package installed. After the openssl headers 
appeared, there's a bunch of new errors:


rpmssl.c: In function ‘rpmsslGenerate’:
rpmssl.c:629:29: error: dereferencing pointer to incomplete type 
‘EVP_PKEY {aka struct evp_pkey_st}’

  if (EVP_PKEY_type(ssl->pkey->type) == EVP_PKEY_EC) {
 ^~
What this means is that rpm needs to be ported to openssl 1.1 - they 
made a lot of previously open structures opaque in that version, with 
accessor functions (and maybe there are other issues).


I'm not sure why yocto wants openssl support, Mark should know that.

Alex

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 9, 2016, at 10:23 AM, Alexander Kanavin 
>  wrote:
> 
> 

…

Good.

>> There may be some issues with bison/flex in rpmdb (its WIP, the targets 
>> aren’t critical),
>> and perl-URPM likely should be nuked (rm -rf perl-URPM).
> 
> This is the latest failure:
> 
> make[4]: Entering directory '/home/ak/development/rpm/rpmio'
> /bin/bash ../libtool  --tag=CC   --mode=link gcc -fopenmp -g -O2 -g 
> -D_GNU_SOURCE -D_REENTRANT   -o tgfs tgfs.o librpmio.la ../misc/librpmmisc.la 
>  -lpcreposix -lpcre -ldl -lm -ltcl8.6 -lpython2.7 -lreadline -lmagic -lexpat 
> -lz -lpopt -lpthread  -lelf -lpopt -lz -lexpat -lpcre
> libtool: link: gcc -fopenmp -g -O2 -g -D_GNU_SOURCE -D_REENTRANT -o 
> .libs/tgfs tgfs.o  ./.libs/librpmio.so ../misc/.libs/librpmmisc.so 
> -lpcreposix -ldl -lm -ltcl8.6 -lpython2.7 -lreadline -lmagic -lpthread -lelf 
> /usr/lib/x86_64-linux-gnu/libpopt.so -lz 
> /usr/lib/x86_64-linux-gnu/libexpat.so -lpcre -fopenmp
> /usr/bin/ld: tgfs.o: undefined reference to symbol 'shm_open@@GLIBC_2.2.5'
> //lib/x86_64-linux-gnu/librt.so.1: error adding symbols: DSO missing from 
> command line
> collect2: error: ld returned 1 exit status
> 

Hmm … tgfs issues can be avoided by changing rpmio/Makefile.am.

Add tgfs to EXTRA_PROGRAMS instead of noinst_PROGRAMS:
EXTRA_PROGRAMS += tifs
and rerun devtool.


> No idea yet what that means, but maybe you've seen this before. Linking 
> errors are notoriously unhelpful :(
> 

I have had issue building tgfs with various compiler/loader combinations. Not 
building tgfs
is acceptable for RPM, its built (like libgit2) to fail early/hard checking 
compiler/loader flags,
and to include development code when I do submissions to scan.coverity.com of
compile with -fsanitize:address.

>> If you send me your devtool.conf stanza, I will check in your build options, 
>> and
>> be able to help more effectively.
> 
> Sure, here it is. I didn't write it out from scratch, I copied the %system 
> (as that seemed a reasonable start), and modified it when build issues 
> happened. Host is Debian testing (stretch).
> 

Thank you! Yes %system is/was the most reasonable starting point.

FYI, I am usually building (on Fedora) with the %hi stanza, and (on MacOSX) 
with the %falmouth
stanza. (The stanzas are named after the machines I am using).

Good: you appear to have +beecrypt (for digests) and +libtomcrypt (recommended
because of ECDSA and stable) and +openssl (what Yocto wants last I heard).

I’ll send along some other suggestions later today.

Once you get through rpmio building, the rest of rpm usually builds easily.

The next class of problems comes with testing:
cd tests
make clean test
which will tell whether the just built rpm is functional, and often
requires some fidgety triage like duktape/tgfs.

73 de Jeff 

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/08/2016 07:53 PM, Jeffrey Johnson wrote:


RPM5 builds against libgit2 tip and libgit2 “breaks” every other month.

The fixes are usually rather easy, but the code is entirely proof-of-concept.

I recommend
rm -rf libgit2
(which is likely enuf to disable) and building —without-libgit2


That's what I did, seems to help.


Issue 2:

*** Warning: Linking the shared library librpmio.la against the non-libtool
*** objects  duktape.o is not portable!
libtool: link: g++  -fPIC -DPIC -shared -nostdlib 
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/crti.o 
/usr/lib/gcc/x86_64-linux-gnu/6/crtbeginS.o  .libs/getdate.o .libs/blake2b.o 
.libs/blake2bp.o .libs/blake2s.o .libs/blake2sp.o .libs/blake2-rpm.o 
.libs/arirang.o .libs/blake.o .libs/bmw.o .libs/chi.o .libs/cubehash.o 
.libs/echo.o .libs/edon-r.o .libs/fugue.o .libs/groestl.o .libs/hamsi.o 
.libs/jh.o .libs/keccak.o .libs/lane.o .libs/luffa.o .libs/md2.o .libs/md6.o 
.libs/radiogatun.o .libs/salsa10.o .libs/salsa20.o .libs/shabal.o 
.libs/shavite3.o .libs/simd.o .libs/skein.o .libs/tib3.o .libs/tiger.o 
.libs/rpmgit.o .libs/rpmio-stub.o .libs/rpmjs.o .libs/rpmjsio.o 
.libs/rpmkeyring.o .libs/rpmnix.o .libs/rpmodbc.o .libs/rpmsql.o .libs/set.o 
.libs/ar.o .libs/argv.o .libs/bcon.o .libs/bson.o .libs/bzdio.o .libs/cipher.o 
.libs/cpio.o .libs/crc.o .libs/digest.o .libs/fts.o .libs/gzdio.o .libs/iosm.o 
.libs/jsmn.o .libs/lsyck.o .libs/macro.o .libs/mire.o .libs/mongoc.o 
.libs/mount.o .libs/pcrs.o .libs/poptIO.o .libs/rpmacl.o .libs/rpmasn.o 
.libs/rpmaug.o .libs/rpmbag.o .libs/rpmbc.o .libs/rpmbf.o .libs/rpmcdsa.o 
.libs/rpmct.o .libs/rpmcudf.o .libs/rpmcvs.o .libs/rpmdate.o .libs/rpmdav.o 
.libs/rpmdir.o .libs/rpmficl.o .libs/rpmgc.o .libs/rpmgfs.o .libs/rpmhash.o 
.libs/rpmhkp.o .libs/rpmhook.o .libs/rpmio.o .libs/rpmiob.o .libs/rpmjni.o 
.libs/rpmku.o .libs/rpmlog.o .libs/rpmltc.o .libs/rpmlua.o .libs/rpmmalloc.o 
.libs/rpmmg.o .libs/rpmmqtt.o .libs/rpmmrb.o .libs/rpmnss.o .libs/rpmperl.o 
.libs/rpmpgp.o .libs/rpmpython.o .libs/rpmrpc.o .libs/rpmruby.o .libs/rpmsed.o 
.libs/rpmsm.o .libs/rpmsp.o .libs/rpmsq.o .libs/rpmsquirrel.o .libs/rpmssl.o 
.libs/rpmsvn.o .libs/rpmsw.o .libs/rpmsx.o .libs/rpmsyck.o .libs/rpmtcl.o 
.libs/rpmtpm.o .libs/rpmuuid.o .libs/rpmxar.o .libs/rpmzlog.o .libs/rpmzq.o 
.libs/strcasecmp.o .libs/strtolocale.o .libs/tar.o .libs/ugid.o .libs/url.o 
.libs/xzdio.o .libs/yajl.o .libs/yarn.o  duktape.o   -Wl,-rpath 
-Wl,/home/ak/development/rpm/misc/.libs ../misc/.libs/librpmmisc.so -lpcreposix 
-ldl -ltcl8.6 -lpython2.7 -lreadline -lmagic -lpthread -lelf 
/usr/lib/x86_64-linux-gnu/libpopt.so -lz /usr/lib/x86_64-linux-gnu/libexpat.so 
-lpcre -L/usr/lib/gcc/x86_64-linux-gnu/6 
-L/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu 
-L/usr/lib/gcc/x86_64-linux-gnu/6/../../../../lib -L/lib/x86_64-linux-gnu 
-L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib 
-L/usr/lib/gcc/x86_64-linux-gnu/6/../../.. -lstdc++ -lm -lc -lgcc_s 
/usr/lib/gcc/x86_64-linux-gnu/6/crtendS.o 
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/crtn.o  -g -O2 
-Wl,--version-script -Wl,../rpmio/librpmio.vers   -fopenmp -Wl,-soname 
-Wl,librpmio-5.4.so -o .libs/librpmio-5.4.so
/usr/bin/ld: duktape.o: relocation R_X86_64_PC32 against symbol 
`stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile 
with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Makefile:2597: recipe for target 'librpmio.la' failed

The duktape PIC issue can be fixed by adding common CFLAGS to duktape's custom 
flags, I guess, which currently isn't the case.


This is not that easy unfortunately. The PIC compiler flags are supplied 
by libtool, and duktape.o is created without using libtool.

So I'm not sure what a real fix would be, can you provide that?


Yup. There’s other duktape.c annoyances encountered if/when switching between 
gcc and clang.

I usually comment out duktape (or make it an optional build) in 
rpmio/Makefile.am when it gets in my way.


I've removed duktape.o from here:

librpmio_la_LIBADD = duktape.o -lm

and that helps.


There may be some issues with bison/flex in rpmdb (its WIP, the targets aren’t 
critical),
 and perl-URPM likely should be nuked (rm -rf perl-URPM).


This is the latest failure:

make[4]: Entering directory '/home/ak/development/rpm/rpmio'
/bin/bash ../libtool  --tag=CC   --mode=link gcc -fopenmp -g -O2 -g 
-D_GNU_SOURCE -D_REENTRANT   -o tgfs tgfs.o librpmio.la 
../misc/librpmmisc.la  -lpcreposix -lpcre -ldl -lm -ltcl8.6 -lpython2.7 
-lreadline -lmagic -lexpat -lz -lpopt -lpthread  -lelf -lpopt -lz 
-lexpat -lpcre
libtool: link: gcc -fopenmp -g -O2 -g -D_GNU_SOURCE -D_REENTRANT -o 
.libs/tgfs tgfs.o  ./.libs/librpmio.so ../misc/.libs/librpmmisc.so 
-lpcreposix -ldl -lm -ltcl8.6 -lpython2.7 -lreadline -lmagic -lpthread 
-lelf /usr/lib/x86_64-linux-gnu/libpopt.so -lz 
/usr/lib/x86_64-linux-gnu/libexpat.so -lpcre -fopenmp

/usr/bin/ld: tgfs.o: undefined 

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 8, 2016, at 8:49 AM, Alexander Kanavin 
>  wrote:
> 
> On 12/08/2016 10:54 AM, Jeff Johnson wrote:
>>> Jeff, can you commit the changes to cvs?
>>> 
>> 
>> I can’t check in code blindly, nor will I check in code that cannot be 
>> tested,
>> nor do I have the time to generate diff’s and examine your port.
>> 
>> There is no way to test (even de facto, by running some other application)
>> rpm-python3 atm.
> 
> FWIW, rpm's python support may not be needed at all to run dnf in the future, 
> if they port dnf on top of libhif as promised, and stop using a combination 
> of hawkey and rpm-python. I didn't study the code in detail to confirm this 
> though. It's all in flux at the moment.
> 

I hope that is true: I have essentially zero interest in rpm-python these days.

My guess is that — as long as yum is around — rpm-python will continue
a zombie existence in “open sores” software.

Meanwhile, spinning rpm-python out of RPM CVS into a separate git project
is likely the best way forward. That was rpm.org’s plan as well way back when.

73 de Jeff

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 8, 2016, at 10:16 AM, Alexander Kanavin 
>  wrote:
> 
> On 12/07/2016 07:21 PM, Jeffrey Johnson wrote:
>> Your next stopping point is going to be Berkeley DB as a pre-requisite.
> 
> I didn't get there yet; I hit a couple different issues, and these do look 
> like bugs. The relevant configure options are
> 
>   --with-libgit2 \
>   --enable-build-pic \
> 

You are doing well ;-)

> Issue 1:
> 
> make[4]: Entering directory '/home/ak/development/rpm/rpmio'
> /bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2 -no-undefined 
> -release 5.4 -Wl,--version-script,../rpmio/librpmio.vers-o librpmio.la 
> -rpath /usr/lib getdate.lo blake2b.lo blake2bp.lo blake2s.lo blake2sp.lo 
> blake2-rpm.lo arirang.lo blake.lo bmw.lo chi.lo cubehash.lo echo.lo edon-r.lo 
> fugue.lo groestl.lo hamsi.lo jh.lo keccak.lo lane.lo luffa.lo md2.lo md6.lo 
> radiogatun.lo salsa10.lo salsa20.lo shabal.lo shavite3.lo simd.lo skein.lo 
> tib3.lo tiger.lo rpmgit.lo rpmio-stub.lo rpmjs.lo rpmjsio.lo rpmkeyring.lo 
> rpmnix.lo rpmodbc.lo rpmsql.lo set.lo ar.lo argv.lo bcon.lo bson.lo bzdio.lo 
> cipher.lo cpio.lo crc.lo digest.lo fts.lo gzdio.lo iosm.lo jsmn.lo lsyck.lo 
> macro.lo mire.lo mongoc.lo mount.lo pcrs.lo poptIO.lo rpmacl.lo rpmasn.lo 
> rpmaug.lo rpmbag.lo rpmbc.lo rpmbf.lo rpmcdsa.lo rpmct.lo rpmcudf.lo 
> rpmcvs.lo rpmdate.lo rpmdav.lo rpmdir.lo rpmficl.lo rpmgc.lo rpmgfs.lo 
> rpmhash.lo rpmhkp.lo rpmhook.lo rpmio.lo rpmiob.lo rpmjni.lo rpmku.lo 
> rpmlog.lo rpmltc.lo rpmlua.lo rpmmalloc.lo rpmmg.lo rpmmqtt.lo rpmmrb.lo 
> rpmnss.lo rpmperl.lo rpmpgp.lo rpmpython.lo rpmrpc.lo rpmruby.lo rpmsed.lo 
> rpmsm.lo rpmsp.lo rpmsq.lo rpmsquirrel.lo rpmssl.lo rpmsvn.lo rpmsw.lo 
> rpmsx.lo rpmsyck.lo rpmtcl.lo rpmtpm.lo rpmuuid.lo rpmxar.lo rpmzlog.lo 
> rpmzq.lo strcasecmp.lo strtolocale.lo tar.lo ugid.lo url.lo xzdio.lo yajl.lo 
> yarn.lo duktape.o -lm ../misc/librpmmisc.la -lpcreposix -lpcre -ldl -lm 
> -ltcl8.6 -lpython2.7 -lreadline -lmagic -lexpat -lz -lpopt -lpthread  -lelf 
> -lpopt -lz -lexpat -lpcre
> depbase=`echo tgit.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
> gcc -DHAVE_CONFIG_H -I. -I..  -I. -I.. -I../build -I../lib -I../lib 
> -I../rpmdb -I../rpmio -I../misc -I../beecrypt/include -I../beecrypt/include 
> -I../beecrypt -I../beecrypt -I../libgit2/include -I../libgit2/include 
> -I../libgit2 -I../libgit2 -I../lua/local -I../lua/local -I../lua -I../lua 
> -I../neon/src -I../neon/src -I../neon -I../neon -I../syck/lib -I../syck/lib 
> -I../syck -I../syck -DRPM_OS_LINUX=040800 -I../gnulib -I../gnulib 
> -I/usr/include/db62 -I/usr/include/python2.7 -I/usr/include/python2.7 
> -fopenmp -g -O2 -g -D_GNU_SOURCE -D_REENTRANT -MT tgit.o -MD -MP -MF 
> $depbase.Tpo -c -o tgit.o tgit.c &&\
> mv -f $depbase.Tpo $depbase.Po
> tgit.c: In function ‘cmd_ls_remote’:
> tgit.c:3755:2: error: too few arguments to function ‘git_remote_connect’
>  git_remote_connect(remote, GIT_DIRECTION_FETCH,
>  ^
> tgit.c:47:25: note: in definition of macro ‘chkgit’
> Xchkgit(_git, _msg, _error, _rpmgit_debug, __FUNCTION__, __FILE__, 
> __LINE__)
> ^~
> In file included from ../libgit2/include/git2/clone.h:14:0,
> from ../libgit2/include/git2.h:19,
> from ./rpmgit.h:32,
> from tgit.c:9:
> ../libgit2/include/git2/remote.h:249:17: note: declared here
> GIT_EXTERN(int) git_remote_connect(git_remote *remote, git_direction 
> direction, const git_remote_callbacks *callbacks, const git_proxy_options 
> *proxy_opts, const git_strarray *custom_headers);
> ^~
> Makefile:3104: recipe for target ‘tgit.o’

RPM5 builds against libgit2 tip and libgit2 “breaks” every other month.

The fixes are usually rather easy, but the code is entirely proof-of-concept.

I recommend
rm -rf libgit2
(which is likely enuf to disable) and building —without-libgit2


> Issue 2:
> 
> *** Warning: Linking the shared library librpmio.la against the non-libtool
> *** objects  duktape.o is not portable!
> libtool: link: g++  -fPIC -DPIC -shared -nostdlib 
> /usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/crti.o 
> /usr/lib/gcc/x86_64-linux-gnu/6/crtbeginS.o  .libs/getdate.o .libs/blake2b.o 
> .libs/blake2bp.o .libs/blake2s.o .libs/blake2sp.o .libs/blake2-rpm.o 
> .libs/arirang.o .libs/blake.o .libs/bmw.o .libs/chi.o .libs/cubehash.o 
> .libs/echo.o .libs/edon-r.o .libs/fugue.o .libs/groestl.o .libs/hamsi.o 
> .libs/jh.o .libs/keccak.o .libs/lane.o .libs/luffa.o .libs/md2.o .libs/md6.o 
> .libs/radiogatun.o .libs/salsa10.o .libs/salsa20.o .libs/shabal.o 
> .libs/shavite3.o .libs/simd.o .libs/skein.o .libs/tib3.o .libs/tiger.o 
> .libs/rpmgit.o .libs/rpmio-stub.o .libs/rpmjs.o .libs/rpmjsio.o 
> .libs/rpmkeyring.o .libs/rpmnix.o .libs/rpmodbc.o .libs/rpmsql.o .libs/set.o 
> .libs/ar.o .libs/argv.o .libs/bcon.o .libs/bson.o .libs/bzdio.o 
> .libs/cipher.o .libs/cpio.o .libs/crc.o .libs/digest.o .libs/fts.o 
> 

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/07/2016 07:21 PM, Jeffrey Johnson wrote:

Your next stopping point is going to be Berkeley DB as a pre-requisite.


I didn't get there yet; I hit a couple different issues, and these do 
look like bugs. The relevant configure options are


   --with-libgit2 \
   --enable-build-pic \

Issue 1:

make[4]: Entering directory '/home/ak/development/rpm/rpmio'
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2 -no-undefined 
-release 5.4 -Wl,--version-script,../rpmio/librpmio.vers-o 
librpmio.la -rpath /usr/lib getdate.lo blake2b.lo blake2bp.lo blake2s.lo 
blake2sp.lo blake2-rpm.lo arirang.lo blake.lo bmw.lo chi.lo cubehash.lo 
echo.lo edon-r.lo fugue.lo groestl.lo hamsi.lo jh.lo keccak.lo lane.lo 
luffa.lo md2.lo md6.lo radiogatun.lo salsa10.lo salsa20.lo shabal.lo 
shavite3.lo simd.lo skein.lo tib3.lo tiger.lo rpmgit.lo rpmio-stub.lo 
rpmjs.lo rpmjsio.lo rpmkeyring.lo rpmnix.lo rpmodbc.lo rpmsql.lo set.lo 
ar.lo argv.lo bcon.lo bson.lo bzdio.lo cipher.lo cpio.lo crc.lo 
digest.lo fts.lo gzdio.lo iosm.lo jsmn.lo lsyck.lo macro.lo mire.lo 
mongoc.lo mount.lo pcrs.lo poptIO.lo rpmacl.lo rpmasn.lo rpmaug.lo 
rpmbag.lo rpmbc.lo rpmbf.lo rpmcdsa.lo rpmct.lo rpmcudf.lo rpmcvs.lo 
rpmdate.lo rpmdav.lo rpmdir.lo rpmficl.lo rpmgc.lo rpmgfs.lo rpmhash.lo 
rpmhkp.lo rpmhook.lo rpmio.lo rpmiob.lo rpmjni.lo rpmku.lo rpmlog.lo 
rpmltc.lo rpmlua.lo rpmmalloc.lo rpmmg.lo rpmmqtt.lo rpmmrb.lo rpmnss.lo 
rpmperl.lo rpmpgp.lo rpmpython.lo rpmrpc.lo rpmruby.lo rpmsed.lo 
rpmsm.lo rpmsp.lo rpmsq.lo rpmsquirrel.lo rpmssl.lo rpmsvn.lo rpmsw.lo 
rpmsx.lo rpmsyck.lo rpmtcl.lo rpmtpm.lo rpmuuid.lo rpmxar.lo rpmzlog.lo 
rpmzq.lo strcasecmp.lo strtolocale.lo tar.lo ugid.lo url.lo xzdio.lo 
yajl.lo yarn.lo duktape.o -lm ../misc/librpmmisc.la -lpcreposix -lpcre 
-ldl -lm -ltcl8.6 -lpython2.7 -lreadline -lmagic -lexpat -lz -lpopt 
-lpthread  -lelf -lpopt -lz -lexpat -lpcre

depbase=`echo tgit.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
gcc -DHAVE_CONFIG_H -I. -I..  -I. -I.. -I../build -I../lib -I../lib 
-I../rpmdb -I../rpmio -I../misc -I../beecrypt/include 
-I../beecrypt/include -I../beecrypt -I../beecrypt -I../libgit2/include 
-I../libgit2/include -I../libgit2 -I../libgit2 -I../lua/local 
-I../lua/local -I../lua -I../lua -I../neon/src -I../neon/src -I../neon 
-I../neon -I../syck/lib -I../syck/lib -I../syck -I../syck 
-DRPM_OS_LINUX=040800 -I../gnulib -I../gnulib -I/usr/include/db62 
-I/usr/include/python2.7 -I/usr/include/python2.7 -fopenmp -g -O2 -g 
-D_GNU_SOURCE -D_REENTRANT -MT tgit.o -MD -MP -MF $depbase.Tpo -c -o 
tgit.o tgit.c &&\

mv -f $depbase.Tpo $depbase.Po
tgit.c: In function ‘cmd_ls_remote’:
tgit.c:3755:2: error: too few arguments to function ‘git_remote_connect’
  git_remote_connect(remote, GIT_DIRECTION_FETCH,
  ^
tgit.c:47:25: note: in definition of macro ‘chkgit’
 Xchkgit(_git, _msg, _error, _rpmgit_debug, __FUNCTION__, __FILE__, 
__LINE__)

 ^~
In file included from ../libgit2/include/git2/clone.h:14:0,
 from ../libgit2/include/git2.h:19,
 from ./rpmgit.h:32,
 from tgit.c:9:
../libgit2/include/git2/remote.h:249:17: note: declared here
 GIT_EXTERN(int) git_remote_connect(git_remote *remote, git_direction 
direction, const git_remote_callbacks *callbacks, const 
git_proxy_options *proxy_opts, const git_strarray *custom_headers);

 ^~
Makefile:3104: recipe for target 'tgit.o' failed

Issue 2:

*** Warning: Linking the shared library librpmio.la against the non-libtool
*** objects  duktape.o is not portable!
libtool: link: g++  -fPIC -DPIC -shared -nostdlib 
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/crti.o 
/usr/lib/gcc/x86_64-linux-gnu/6/crtbeginS.o  .libs/getdate.o 
.libs/blake2b.o .libs/blake2bp.o .libs/blake2s.o .libs/blake2sp.o 
.libs/blake2-rpm.o .libs/arirang.o .libs/blake.o .libs/bmw.o .libs/chi.o 
.libs/cubehash.o .libs/echo.o .libs/edon-r.o .libs/fugue.o 
.libs/groestl.o .libs/hamsi.o .libs/jh.o .libs/keccak.o .libs/lane.o 
.libs/luffa.o .libs/md2.o .libs/md6.o .libs/radiogatun.o .libs/salsa10.o 
.libs/salsa20.o .libs/shabal.o .libs/shavite3.o .libs/simd.o 
.libs/skein.o .libs/tib3.o .libs/tiger.o .libs/rpmgit.o 
.libs/rpmio-stub.o .libs/rpmjs.o .libs/rpmjsio.o .libs/rpmkeyring.o 
.libs/rpmnix.o .libs/rpmodbc.o .libs/rpmsql.o .libs/set.o .libs/ar.o 
.libs/argv.o .libs/bcon.o .libs/bson.o .libs/bzdio.o .libs/cipher.o 
.libs/cpio.o .libs/crc.o .libs/digest.o .libs/fts.o .libs/gzdio.o 
.libs/iosm.o .libs/jsmn.o .libs/lsyck.o .libs/macro.o .libs/mire.o 
.libs/mongoc.o .libs/mount.o .libs/pcrs.o .libs/poptIO.o .libs/rpmacl.o 
.libs/rpmasn.o .libs/rpmaug.o .libs/rpmbag.o .libs/rpmbc.o .libs/rpmbf.o 
.libs/rpmcdsa.o .libs/rpmct.o .libs/rpmcudf.o .libs/rpmcvs.o 
.libs/rpmdate.o .libs/rpmdav.o .libs/rpmdir.o .libs/rpmficl.o 
.libs/rpmgc.o .libs/rpmgfs.o .libs/rpmhash.o .libs/rpmhkp.o 
.libs/rpmhook.o .libs/rpmio.o .libs/rpmiob.o .libs/rpmjni.o 
.libs/rpmku.o .libs/rpmlog.o 

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/08/2016 10:54 AM, Jeff Johnson wrote:

Jeff, can you commit the changes to cvs?



I can’t check in code blindly, nor will I check in code that cannot be tested,
nor do I have the time to generate diff’s and examine your port.

There is no way to test (even de facto, by running some other application)
rpm-python3 atm.


FWIW, rpm's python support may not be needed at all to run dnf in the 
future, if they port dnf on top of libhif as promised, and stop using a 
combination of hawkey and rpm-python. I didn't study the code in detail 
to confirm this though. It's all in flux at the moment.



Alex
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 7, 2016, at 11:28 AM, Alexander Kanavin 
>  wrote:
> 
> On 12/07/2016 06:22 PM, Alexander Kanavin wrote:
> 
>> checking whether to build with Perl ExtUtils::Embed library... no
>> ++ executing failure action
>> ++ searching location: /usr/lib64
>> configure: error: Unknown location specification /usr/lib64
>> 
>> I'm on Debian Testing, which does not have /usr/lib64 - I could dig to
>> see where that comes from, but thought you'd be interested in this/can
>> quickly suggest a fix.
> 
> Right, didn't notice this in devtool.conf. Nevermind.
> 
>--libdir=/usr/lib64 \
> 

Good.

Your next stopping point is going to be Berkeley DB as a pre-requisite.

Either install db-6.2.23 (which needs the one line patch posted last week), or
edit configure.ac to change
DBXY=62
and every occurrence of "-6,2” in macros like this
  RPM_CHECK_LIB(
[Berkeley-DB], [db],
[db-6.2], [db_create], [db.h],
[yes,external], [db3],

hth

73 de Jeff

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/07/2016 06:22 PM, Alexander Kanavin wrote:


checking whether to build with Perl ExtUtils::Embed library... no
++ executing failure action
++ searching location: /usr/lib64
configure: error: Unknown location specification /usr/lib64

I'm on Debian Testing, which does not have /usr/lib64 - I could dig to
see where that comes from, but thought you'd be interested in this/can
quickly suggest a fix.


Right, didn't notice this in devtool.conf. Nevermind.

--libdir=/usr/lib64 \

Alex

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/02/2016 07:01 PM, Jeffrey Johnson wrote:

Building RPM from CVS — once you have decided options and installed 
pre-requisites) —
is mostly this
cd rpm
./devtool checkout
./devtool somestanza

Choosing options and installing pre-requisites is far from easy however.

Can you attempt a stanza for devtool.conf? There are myriad examples.


Just tried

./devtool system

which ended in

checking whether to build with Perl ExtUtils::Embed library... no
++ executing failure action
++ searching location: /usr/lib64
configure: error: Unknown location specification /usr/lib64

I'm on Debian Testing, which does not have /usr/lib64 - I could dig to 
see where that comes from, but thought you'd be interested in this/can 
quickly suggest a fix.


Alex

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Dec 5, 2016, at 9:09 AM, Alexander Kanavin 
>  wrote:
> 
> On 12/02/2016 07:01 PM, Jeffrey Johnson wrote:
>> What I need is to be able to see what cmake options you are using in order
>> to be able to build  dnf pre-requisites without having to checkout (or 
>> install)
>> OpenEmbedded,
> 
> Sure, yes, but I guess you don't need them right now?
> 

I’m ready when you are. Jump starting the porting effort by boot strapping
from common code is rate determining. There’s just too much code, and
too many layers, to do otherwise imho.

>> OK. This is either Case #2 (I am installing into /opt/local instead of 
>> adding —enable-build-version-script
>> which is more complex) or Case #3 (building against a RPM CVS checkout) for 
>> building.
>> 
>> Which are you?
> 
> Rpm 5 is built and then installed into a openembedded-specific sysroot 
> directory (/path/to/sysroot/ which inside looks like a regular root directory 
> with /etc /usr /var etc.), and then the rpm dependencies are configured to 
> use that installation when building and running them. There is a 
> --enable-build-versionscript in use. I'm using debian as a host machine, so 
> there is no danger of clashing with host rpm.
> 

Again, packaging != development, and adds a different delay and indeterminism
while satisfying packaging policies and build systems are negotiated.

I’d like to be able to build the entire stack needed without having to debug 
packaging.

>> If you would like some help, I need to see what build options you are 
>> attempting to use with RPM5.
> 
> Thank you for the offer, I'll first try to amend existing rpm5 recipe with a 
> bit of help from Mark; if that proves too hard, I'll try with devtool.
> 

devtool, not configure, is the supported build procedure from CVS.

And truly devtool is far far more reproducible and easier to use than e-mail or 
irc
to communicate what options are chosen and needed. There are too many
Have it your own way!
disablers to test every possible combination. I work with a maximally configured
RPM to ensure functional code. But that means that there are often minor flaws
building with some options disabled. I depend on users to feed me desired
configurations to detect and repair build failures.

RPM “cd tests && make clean test” (particularly when compiled with 
-fsanitize:address)
is quite effective in finding issues in RPM.

73 de Jeff__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/02/2016 07:01 PM, Jeffrey Johnson wrote:

What I need is to be able to see what cmake options you are using in order
to be able to build  dnf pre-requisites without having to checkout (or install)
OpenEmbedded,


Sure, yes, but I guess you don't need them right now?


OK. This is either Case #2 (I am installing into /opt/local instead of adding 
—enable-build-version-script
which is more complex) or Case #3 (building against a RPM CVS checkout) for 
building.

Which are you?


Rpm 5 is built and then installed into a openembedded-specific sysroot 
directory (/path/to/sysroot/ which inside looks like a regular root 
directory with /etc /usr /var etc.), and then the rpm dependencies are 
configured to use that installation when building and running them. 
There is a --enable-build-versionscript in use. I'm using debian as a 
host machine, so there is no danger of clashing with host rpm.



If you would like some help, I need to see what build options you are 
attempting to use with RPM5.


Thank you for the offer, I'll first try to amend existing rpm5 recipe 
with a bit of help from Mark; if that proves too hard, I'll try with 
devtool.



Alex
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 12/01/2016 08:35 PM, Jeffrey Johnson wrote:


There is the additional porting issue that libhif has pre-requisites, including 
(at least)
rpm
rpm-python
libsolv
librepo
that will need to be shared for a successful porting collaboration.


FWIW, I've built and run unit tests for both libsolv and librepo 
(against rpm5) - they seem to pass fine. libsolv has explicit rpm5 
support which needs to be enabled when configuring the build. So I don't 
have anything to share for those, and at the moment libhif is where the 
known issues are.



The build procedures using CMAKE are quite complicated and in need of sharing 
as well.
I have been adding a non-brainer doit.sh that is easily modified (with paths 
and switches)
on different development platforms as needed.


I don't need the doit.sh; my builds are configured from openembedded 
recipes, and it's not at all complicated, for me at least :)


Let's decide how to merge work a bit later; at the moment I'm focused on 
getting latest cvs version of rpm5 to build, so I can continue porting 
work against that version, and easily take any necessary fixes that you 
do to rpm5.


Alex
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 30, 2016, at 7:30 AM, Alexander Kanavin 
>  wrote:
> 
> On 11/29/2016 06:02 PM, Jeffrey Johnson wrote:
> 
>> Apologies, I’m new to project/repository management thru github.
>> 
>> Should be fixed.
>> 
>> We may have to iterate thru ssh key exchange etc etc soon too, depending on
>> what is needed.
> 

After resuming work I dropped in late August, here are some further thoughts …

> Thanks, I've rearranged the branches a little bit:
> 
> - your OS X build hack is now in jeff/osxhack branch
> 

OK. My changes can be resurrected as needed from wherever.

> - I removed that commit from the master branch, so master can be used to 
> track upstream, and nothing else. That means the branch has been force 
> pushed, so delete your local copy to avoid accidentally pushing the commit 
> again.
> 

Creating a pristine rpm5/libhif:master makes merges “transparent” and 
simplifies syncing “upstream” changes, absolutely.

However, what is needed is a common point in the graph where new porting work 
can be merged and shared.

> - I'll publish rpm5 work in alex/rpm5 branch, will let you know when there's 
> something to look at.
> 

At the moment there is no (except “upstream”) common point where porting merges 
can occur.

The obvious/simple merge point (and what was already in place) was 
rpm5/libhif:master which
could merge from an “upstream” remote.

This in fact is what is suggested by github:
https://help.github.com/articles/syncing-a-fork/

Whether we follow git best practices on “feature” branches ignores the fact that
a porting collaboration needs merges, not isolation, and that there are only two
of us developing atm. If you wish to preview merges, or discuss what tasks are
needed in wiki/issues on github, we can do that. Meanwhile I would deem 
structural
changes to add testable RPM5/MACOSX booleans (i.e. basically what you have 
pushed
from rpm5/libhif:master onto a jeff/osxhack “feature” branch as rather obvious 
immediate
and necessary porting needs.

There is the additional porting issue that libhif has pre-requisites, including 
(at least)
rpm
rpm-python
libsolv
librepo
that will need to be shared for a successful porting collaboration.

The build procedures using CMAKE are quite complicated and in need of sharing 
as well.
I have been adding a non-brainer doit.sh that is easily modified (with paths 
and switches)
on different development platforms as needed.

73 de Jeff__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 30, 2016, at 7:42 AM, Alexander Kanavin 
>  wrote:
> 
>> 
>> (aside)
>> For starters, RPM5 uses keyutils(8), which has some hope of refactoring 
>> _ALL__ pubkey
>> management out of _ALL_ RPM tool chains by writing a helper that is invoked 
>> by the
>> keyutils async callback.
> 
> Maybe it's possible to provide empty keyring functions to substitute for 
> their absence in rpm5 - that way we can avoid having #ifdefs all over the 
> place. I'd like to use #ifdefs sparingly, they make the code harder to read 
> and maintain.
> 

Let try to bring ourselves into mental alignment, and try
to architect some DNF refactoring that will eliminate
some porting issues:
No developer likes #ifdef’s.

Do you know keyutils? See
man 7 keyutils

Keyutils is a perfectly sane interface for keyring management that potentially
removes the need to duplicate pubkey management throughout RPM tool chains.

The goal would be to eliminate pubkey management in RPM tool chains to
the greatest extent possible, replacing pubkey management with keyutils
system calls.

Instead of having to do dialogues like
Should I import blahblah (y/N)?
where necessary (and often inconveniently) in RPM tool chains, a request-key(8) 
invocation
would be performed by the kernel to return a cached pubkey.

(aside)
Arguably, there is as little reason to have a dialogue about pubkey importation
as there is to have a EULA dialogue while installing, but “trust” has many
definitions, and the dialogues are a useful means to notify end-users.

The existing multiple external pubkey sources like /etc/pki/rpm-gpg (rpm.org) 
and
RPMTAG_PUBKEYS in *.rpm packages (rpm5.org, used to be in rpm.org) as well
as alternatives  like SKS key-servers (rpm5.org) and other keyring sources
like ~/.gnupg can be unified through a single keyutils keyring implementation.

The existing keyring* API at rpm.org can be ported to use keyutils quite easily
for (legacy) API compatibility. RPM5 (when compiled with -lkeyutils) already
has some simple conventions for saving/retrieving pubkeys (though I haven’t
bothered yet with non-repudiable signatures/pubkeys/privkeys, not hard).

The development path would be to first enhance the rpmkey* API to handle
kernel keyrings, and then to refactor current pub key handling within both 
rpm.org
and rpm5.org to use a common approach to key management.

Note that RPM5 has internal generate/sign/verify functions, and so would also
use keyutils for private as well as public keys.

Does the above start to describe the deeper issues associated with the keyring* 
API porting?
I’m suggesting replacing the existing code in DNF with a better implementation 
throughout.

73 de Jeff
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 30, 2016, at 7:42 AM, Alexander Kanavin 
>  wrote:
> 
> On 11/29/2016 06:31 PM, Jeffrey Johnson wrote:
>> The fundamental issue is that  can only provide compile time 
>> API compatibility.
>> 
>> That’s fine for apparently simple projects that only need, say, to read a 
>> package header, and print
>> some scalar tags.
>> 
>> I use “simply” advisedly. Reading a digitally signed package header requires 
>> a pubkey from
>> somewhere, and the original RPM API from last century has no easy means to 
>> make that pub key
>> magically/transparently appear.
>> 
>> There are deeper issues with ABI compatibility and rpmdb retrieve and rpm 
>> configuration
>> that  cannot address.
>> 
>> Meanwhile many trivial incompatibility solutions can be found in 
>> .
>> 
>> But the better engineering for a port as large and complex as DNF should not 
>> (imho)
>> be attempted through the false hope of
>>  #include 
> 
> Alright. Let's drop the include and solve the issues the hard way.
> 

The API compilation issues aren’t all that hard, nor is the connection to rpm 
libraries in C pervasive
The hard issues are going to be changing dnf to handle arbitrary tags and 
mandatory signature verification
and transactions from RPM5 intelligently, and identifying equivalent 
implementations for SELinux
and rich dependencies from rpm.org.

>> For now, comment out all keyring handling and pubkey management in DNF: RPM5 
>> libraries
>> have MANDATORY signature verification (and automated pubkey retrieval) on 
>> the common
>> package header I/O path. A better longer term will have to be postponed 
>> until DNF is sufficiently
>> functional because there are different POV on what “trust” means wrto 
>> package management.
>> 
>> (aside)
>> For starters, RPM5 uses keyutils(8), which has some hope of refactoring 
>> _ALL__ pubkey
>> management out of _ALL_ RPM tool chains by writing a helper that is invoked 
>> by the
>> keyutils async callback.
> 
> Maybe it's possible to provide empty keyring functions to substitute for 
> their absence in rpm5 - that way we can avoid having #ifdefs all over the 
> place. I'd like to use #ifdefs sparingly, they make the code harder to read 
> and maintain.
> 

No need for stubs: there are rpmie/rpmkeyring.[ch] ramie/rpmtd.[ch] already in 
RPM5.

But like , none of this code is used or tested by RPM5 and cannot 
be
maintained unilaterally within RPM5 for many reasons, not the least of which is
that these are _NOT_ RPM5 interfaces, and are a huge impediment to, say, 
deploying
mandatory signature verification in RPM5.

>> These are part of the rpmrc API. The short term porting fix is just to 
>> hardwire useful strings.
>> The better RPM5 fix is to read the first line of /etc/rpm/platform and split 
>> CPU-VENDOR-OS-GNU
>> for OS and CPU.
> 
> Yeah, I need to check what rpm4 does here.
> 

Both rpm4 and rpm5 support /etc/rpm/platform containing CPU-VENDOR-OS-GNU

>> I can walk you through setting up a build from RPM CVS (and likely have to 
>> walk Mark Hatle through same ;-)
>> 
>> The Yocto rpm-5.4.16 “system” RPM can be used for now. But having a full RPM 
>> CVS build
>> will be needed to use additional functionality: RPM5 (deliberately) does not 
>> expose all include
>> files.
> 
> We actually used to have a rpm-from-cvs recipe in openembedded build 
> environment, but it got removed for some reason. Maybe I can bring it back; 
> I'd like to avoid developing on the host.
> 

Develop as you wish …

… meanwhile my main interest is RPM5 not DNF, nor a port. I can deliver
a fix through CVS much more easily than going through a patch release
through Yocto. YMMV.

73 de Jeff
> Alex
> 
> __
> RPM Package Managerhttp://rpm5.org
> Developer Communication Listrpm-devel@rpm5.org

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 11/29/2016 06:02 PM, Jeffrey Johnson wrote:


Apologies, I’m new to project/repository management thru github.

Should be fixed.

We may have to iterate thru ssh key exchange etc etc soon too, depending on
what is needed.


Thanks, I've rearranged the branches a little bit:

- your OS X build hack is now in jeff/osxhack branch

- I removed that commit from the master branch, so master can be used to 
track upstream, and nothing else. That means the branch has been force 
pushed, so delete your local copy to avoid accidentally pushing the 
commit again.


- I'll publish rpm5 work in alex/rpm5 branch, will let you know when 
there's something to look at.



Alex
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 29, 2016, at 9:49 AM, Alexander Kanavin 
>  wrote:
> 
> On 11/28/2016 06:35 PM, Jeffrey Johnson wrote:
>> 2) There are several repetitive cosmetic (but annoying) API differences.
>>  These problems could (almost) be handled with some #defines
>>  * the “match iterator” rpmmi* interface for rpmdb retrieval
>>  * the “problem set” rpmps* interface for problem sets
> 
> As far as libhif build errors related to these differences go, a 
> rpm46compat.h include seems to deal with them.
> 
> I'm not sure if this is the right thing to do, because I don't know how 
> supported that compatibility layer is, and also it produces a duplicate 
> declaration error instead:
> 
> | In file included from 
> /home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm46compat.h:10:0,
> |  from 
> /home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c:39:
> | 
> /home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm4compat.h:354:21:
>  error: static declaration of ‘fdSize’ follows non-static declaration
> |  static inline off_t fdSize(FD_t fd){
> |  ^~
> | In file included from 
> /home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm4compat.h:52:0,
> |  from 
> /home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm46compat.h:10,
> |  from 
> /home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c:39:
> | 
> /home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpmio.h:543:9:
>  note: previous declaration of ‘fdSize’ was here
> |  off_t   fdSize(FD_t fd)
> |  ^~
> 
> What would you suggest?
> 

The fundamental issue is that  can only provide compile time API 
compatibility.

That’s fine for apparently simple projects that only need, say, to read a 
package header, and print
some scalar tags.

I use “simply” advisedly. Reading a digitally signed package header requires a 
pubkey from
somewhere, and the original RPM API from last century has no easy means to make 
that pub key
magically/transparently appear.

There are deeper issues with ABI compatibility and rpmdb retrieve and rpm 
configuration
that  cannot address.

Meanwhile many trivial incompatibility solutions can be found in .

But the better engineering for a port as large and complex as DNF should not 
(imho)
be attempted through the false hope of
#include 

>> 3) Some rpm.org interfaces are intractably different and mostly useless for 
>> RPM5
>>  * the “tag data” rpmtd*
>>  * the rpmkeyring* pubkey handling
>>  While these interfaces have been merged into rpm5 libraries (rpmtd for 
>> python bindings,
>>  rpmkeyring just in case), that isn’t/wasn't the best engineering 
>> because RPM5 itself
>>  doesn’t use or need or test those interfaces. The better approach going 
>> forward (imho)
>>  will be to refactor those api’s somewhere other than in rpm5 libraries. 
>> I don’t yet
>>  know what to do with those apis, but I’d like _NOT_ to be forced to 
>> maintain
>>  multiple divergent/redundant API’s in RPM5 simply to support a dnf (or 
>> rpm-python) port.
> 
> rpmtd and rpmkeyring are used in various places around libhif - I didn't look 
> yet how exactly, but they are a serious blocker.
> 

For now, comment out all keyring handling and pubkey management in DNF: RPM5 
libraries
have MANDATORY signature verification (and automated pubkey retrieval) on the 
common
package header I/O path. A better longer term will have to be postponed until 
DNF is sufficiently
functional because there are different POV on what “trust” means wrto package 
management.

(aside)
For starters, RPM5 uses keyutils(8), which has some hope of refactoring _ALL__ 
pubkey
management out of _ALL_ RPM tool chains by writing a helper that is invoked by 
the
keyutils async callback.


The rpmtd interface is present in -lrpmio for the python bindings. The longer 
term fix will be
to move the rpmtd* API Somewhere Elese Insead, possibly by adding an #ifdef 
RPM5 compatibility
shim.

> A couple other issues I see:
> 
> 1)
> | 
> /home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c: 
> In function ‘dnf_rpmts_get_problem_str’:
> | 
> /home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c:241:10:
>  error: ‘RPMPROB_OBSOLETES’ undeclared (first use in this function)
> |  case RPMPROB_OBSOLETES:
> |   ^
> 

Use #ifdef RPM5 for now

> 2)
> | 
> /home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-context.c:
>  In function ‘dnf_context_get_base_arch’:
> | 
> /home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-context.c:297:5:
>  error: implicit declaration of function ‘rpmGetOsInfo’ 
> [-Werror=implicit-function-declaration]
> |  rpmGetOsInfo(, NULL);

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 29, 2016, at 10:10 AM, Alexander Kanavin 
>  wrote:
> 
> On 11/28/2016 06:35 PM, Jeffrey Johnson wrote:
> 
>> Make yourself happy with libhif: all of my work was largely exploratory
>> assessing what issues need to be solved. Feel free to clobber anything
>> I checked in to start getting an easily updated master branch.
>> 
>> I’ll create a “macports” branch for my MacPorts efforts going forward.
> 
> Looks like I don't have the write access yet:
> 

Apologies, I’m new to project/repository management thru github.

Should be fixed.

We may have to iterate thru ssh key exchange etc etc soon too, depending on
what is needed.

73 de Jeff

__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 11/28/2016 06:35 PM, Jeffrey Johnson wrote:


Make yourself happy with libhif: all of my work was largely exploratory
assessing what issues need to be solved. Feel free to clobber anything
I checked in to start getting an easily updated master branch.

I’ll create a “macports” branch for my MacPorts efforts going forward.


Looks like I don't have the write access yet:

ak@kanavin-desktop:~/development/libhif$ git push --set-upstream origin jeff
ERROR: Permission to rpm5/libhif.git denied to kanavin.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.


Alex


__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Alexander Kanavin]

On 11/28/2016 06:35 PM, Jeffrey Johnson wrote:

2) There are several repetitive cosmetic (but annoying) API differences.
These problems could (almost) be handled with some #defines
* the “match iterator” rpmmi* interface for rpmdb retrieval
* the “problem set” rpmps* interface for problem sets


As far as libhif build errors related to these differences go, a 
rpm46compat.h include seems to deal with them.


I'm not sure if this is the right thing to do, because I don't know how 
supported that compatibility layer is, and also it produces a duplicate 
declaration error instead:


| In file included from 
/home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm46compat.h:10:0,
|  from 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c:39:
| 
/home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm4compat.h:354:21: 
error: static declaration of ‘fdSize’ follows non-static declaration

|  static inline off_t fdSize(FD_t fd){
|  ^~
| In file included from 
/home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm4compat.h:52:0,
|  from 
/home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpm46compat.h:10,
|  from 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c:39:
| 
/home/ak/development/poky/build/tmp/sysroots/x86_64-linux/usr/include/rpm/rpmio.h:543:9: 
note: previous declaration of ‘fdSize’ was here

|  off_t   fdSize(FD_t fd)
|  ^~

What would you suggest?


3) Some rpm.org interfaces are intractably different and mostly useless for RPM5
* the “tag data” rpmtd*
* the rpmkeyring* pubkey handling
While these interfaces have been merged into rpm5 libraries (rpmtd for 
python bindings,
rpmkeyring just in case), that isn’t/wasn't the best engineering 
because RPM5 itself
doesn’t use or need or test those interfaces. The better approach going 
forward (imho)
will be to refactor those api’s somewhere other than in rpm5 libraries. 
I don’t yet
know what to do with those apis, but I’d like _NOT_ to be forced to 
maintain
multiple divergent/redundant API’s in RPM5 simply to support a dnf (or 
rpm-python) port.


rpmtd and rpmkeyring are used in various places around libhif - I didn't 
look yet how exactly, but they are a serious blocker.


A couple other issues I see:

1)
| 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c: 
In function ‘dnf_rpmts_get_problem_str’:
| 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-rpmts.c:241:10: 
error: ‘RPMPROB_OBSOLETES’ undeclared (first use in this function)

|  case RPMPROB_OBSOLETES:
|   ^

2)
| 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-context.c: 
In function ‘dnf_context_get_base_arch’:
| 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-context.c:297:5: 
error: implicit declaration of function ‘rpmGetOsInfo’ 
[-Werror=implicit-function-declaration]

|  rpmGetOsInfo(, NULL);
|  ^~~~
| 
/home/ak/development/poky/build/workspace/sources/libhif/libdnf/dnf-context.c:299:5: 
error: implicit declaration of function ‘rpmGetArchInfo’ 
[-Werror=implicit-function-declaration]

|  rpmGetArchInfo(, NULL);
|  ^~



Meanwhile:
Are you able to build the RPM5 rpm-5_4 branch from CVS?
That is likely needed pretty soon. I can help as necessary.


Openembedded build system is currently packaging rpm 5.4.16 (plus a lot 
of custom patches). I can check out the latest CVS separately and cherry 
pick needed commits from there as needed (and append them to the list of 
custom patches), if that is okay.


Otherwise, Mark Hatle needs to provide me with a working rpm recipe that 
is taking latest CVS code. I'll ask him now.



(aside)
btw, this thread likely should be moved to , I’ve added a 
CC … *shrug*


I'm CCing this message to rpm-devel too, the following messages will go 
to rpm-devel only.


Alex
__
RPM Package Managerhttp://rpm5.org
Developer Communication Listrpm-devel@rpm5.org

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 28, 2016, at 11:35 AM, Jeffrey Johnson  wrote:
> 
> 
> Yes, there were quite a few checkins since I created the forks. I updated all 
> the
> repositories that did not need merging this weekend (but that wasn’t libhif).
> 

rpm5/libhif:master is no sync’ed with “upstream”.

> Make yourself happy with libhif: all of my work was largely exploratory
> assessing what issues need to be solved. Feel free to clobber anything
> I checked in to start getting an easily updated master branch.
> 
> I’ll create a “macports” branch for my MacPorts efforts going forward.
> 
> Meanwhile its likely time to establish some structural cmake parameters for 
> porting use like
>   #ifdef  RPM5
> and
>   #ifdef  MACOSX
> on rpm5:master.
> 

Most of the reason that I attempted to avoid compilation on MACOSX is
to avoid restructuring the CMAKE build beyond setting a prefix to /opt/local.

Since you (and most devels) are likely working on some linux machine, then its 
time to describe
the 3-5 cases that need to be handled with CMAKE include/library paths for a 
dnf port.

Case #1:out-of-tree RPM5 is installed as “system” RPM
CFLAGS needs -I/usr/include/rpm (and remove “rpm/“ #include prefix 
everywhere).
LDFLAGS needs … -lrpm-5.4 -lrpmdb-5.4 -lrpmio-5.4 -lrpmmisc-5.4

Case #2: out-of-tree parallel install next to “system” RPM
RPM5 has a build option
--enable-build-versionscript
to append VERSION to directories, executables, and libraries so that 
multiple versions
of RPM can be installed in parallel. The option isn’t perfect: e.g. 
expect “make install”
to clobber your existing rpm/rpmbuild before renaming with VERSION 
appended,
most man pages and “rpm” locales will be clobbered, and modules (e.g. 
python) haven’t
been attempted (but do have different names, and so won’t clobber 
rpm.org “system” modules.

But with a little tinkeriing, one can pretty easily have multiple 
VERSIONS of RPM5 installed in parallel,
and one can adjust RPM5 configuration macros to have multiple rpmdb’s 
etc etc.

CFLAGS needs -I/usr/include/rpm-${VERSION}
LDFLAGS needs $VERSION in *.la files (for *.a static archives) and (if 
you can’t live with linking only
the last installed *.so) will need additional naming to use the 
full $VERSION in sonames which
current use a $MAJOR-$$MINOR for libraries

Case #3: in-tree RPM builds from CVS but is not installed and “system"
When RPM5 is built but not yet installed, include files and libtool 
*.la files are in these directories
build/
lib/
rpmdb/
rpmio/
misc/
with the obvious changes to CFLAGS/LDFLAGS

Case #4: in-tree RPM builds from CVS with dnf integrated within the RPM CVS tree
The RPM5 CVS tree is mostly easily extensible to build additional 
components
permitting development from a single tree. This provides a single build 
of all
components that simplifies, say, testing and continuous integration.

Case #5: configurable RPM builds to enable/disable internal/external components
RPM5 has configure options to use either external (e.g. in /usr/lib) or 
internal
(e.g. top level subdirectories) that could be extended to accommodate 
dnf components.

Typical development on platforms that are already using RPM5 are case #1.

On platforms (like Fedora/Centos) that need rpm.org, case #2 using 
—enable-build-versionscript
permits development. This is the end CMAKE problem that I need to solve on my 
Fedora/Centos boxes
(but I may just go back to MACOSX development).

There is likely some means to achieve case #3 already implemented (but its not 
exactly obvious)
in dnf, but RPM5 which uses libtool *.la files to implement embedding of 
internal components in
librpmisc.* will likely need some largish changes.

Cases #4 and #5 are included mostly for informational purposes. I will attempt 
implementations
@rpm5.org if/when the time comes.

The immediate need, however, is to start thinking/designing the changes to the 
existing dnf CMAKE build(s) that
can accommodate the above 5 usage cases with minimal configuration 
changes/efforts later.

hth

73 de Jeff

Re: Making rpm5 work with dnf

[Jeffrey Johnson]

> On Nov 28, 2016, at 9:28 AM, Alexander Kanavin 
>  wrote:
> 
> On 11/26/2016 01:07 AM, Jeffrey Johnson wrote:
>> FWIW, your invitation expired or is otherwise unusable (but at least
>> I can read your code, todo++).
> 
> Yeah, I pressed the button on github and only then figured out that it's just 
> for allowing you to write to my repository, and then I withdrew the invite. 
> We can publish and share all work via your rpm5 repos.
> 

Good: let a dnf -> rpm5 collaboration congeal! ;-)

>> You have chosen to fork libhif from
>> rpm-software-management/libhif
>> 
>> rather than a fork (of a fork) from
>> https://github.com/rpm5/libhif
>> 
>> That forces our coordination to be pulled from the only common
>> root at
>> rpm-software-management/libhif
>> 
>> which almost certainly precludes any participation from me and rpm5.org
>> 
>> for various reasons.
> 
> libhif has been in active development lately (51 commits ahead of your repo), 
> and so I wanted to use their latest code as a starting point. Also your repo 
> has a single commit that seems to be incomplete and squashing together 
> several unrelated fixes (without documenting them) [1]. Can I move that 
> commit to a 'jeff' branch, so the master branch contains only proper commits 
> and is updated from upstream regularly?
> 

Yes, there were quite a few checkins since I created the forks. I updated all 
the
repositories that did not need merging this weekend (but that wasn’t libhif).

Make yourself happy with libhif: all of my work was largely exploratory
assessing what issues need to be solved. Feel free to clobber anything
I checked in to start getting an easily updated master branch.

I’ll create a “macports” branch for my MacPorts efforts going forward.

Meanwhile its likely time to establish some structural cmake parameters for 
porting use like
#ifdef  RPM5
and
#ifdef  MACOSX
on rpm5:master.

The exploratory work identified several common porting issues:

1) #include  vs #include 
The issue is whether includes should have directory structure.
RPM5 is going to need to build both in-tree and out-of-tree, and
(imho) this is best dealt with using a -I/usr/include/rpm path in 
CFLAGS.
But we can #ifdef RPM5 around includes for now.

2) There are several repetitive cosmetic (but annoying) API differences.
These problems could (almost) be handled with some #defines
* the “match iterator” rpmmi* interface for rpmdb retrieval
* the “problem set” rpmps* interface for problem sets

3) Some rpm.org interfaces are intractably different and mostly useless for RPM5
* the “tag data” rpmtd*
* the rpmkeyring* pubkey handling
While these interfaces have been merged into rpm5 libraries (rpmtd for 
python bindings,
rpmkeyring just in case), that isn’t/wasn't the best engineering 
because RPM5 itself
doesn’t use or need or test those interfaces. The better approach going 
forward (imho)
will be to refactor those api’s somewhere other than in rpm5 libraries. 
I don’t yet
know what to do with those apis, but I’d like _NOT_ to be forced to 
maintain
multiple divergent/redundant API’s in RPM5 simply to support a dnf (or 
rpm-python) port.

Meanwhile:
Are you able to build the RPM5 rpm-5_4 branch from CVS?
That is likely needed pretty soon. I can help as necessary.

Its also likely time to get the python subdirectory out of rpm5.org CVS and 
into a github.org/rpm5
repository. rpm-python SHOULD be its own project/repository, it was added to 
rpm CVS solely because
of expediency last century.

> [1] 
> https://github.com/rpm5/libhif/commit/1515ca9e9669a4c65caccb38cbe578775c63f282
> 
>> Note that there are several other efforts attempting a dnf->…->rpm5 tool
>> chain that I
>> am aware of. Which is why I attempted RPM5 repositories to permit
>> collaboration, and
>> am perfectly willing to give write access to anyone who wishes.
>> 
>> I am also perfectly willing to let someone other than rpm5.org
>>  administrate the mess if that
>> is what is desired. I do encourage all of you to collaborate early and
>> work forward from
>> working tools. There’s a fair amount of subtle work that will be needed
>> imho.
>> What is your intent: collaboration with rpm5.org  or
>> collaboration with rpm-software-management?
> 
> My idea is that once we get something working in rpm5 repositories and have a 
> reasonable set of commits, I can try to approach rpm4 people with those 
> patches (you don't have to be involved in that). If they agree to take them, 
> great, if not, we'll figure out some way to maintain them separately.
> 

Good luck!

>> Um rpm5.org  and rpm.org  have
>> different API’s,