Re: RFC: APR 2.0 modularization

2024-04-24 Thread Joe Orton
On Tue, Apr 23, 2024 at 10:43:15AM +0200, Ivan Zhakov wrote:
> On Mon, 22 Apr 2024 at 17:01, Joe Orton  wrote:
> > Open
> > questions on where to draw the line, as to what goes in a separate
> > library, and much more I'm sure.
> >
> Well, it's a good question. I was thinking about something like that:
> "everything that can be used in libapr-*-2 should be in libapr-[core]-2".
> I mean in **general** we should avoid dependency like app -> libapr-ABC ->
> libapr-XYZ -> libapr. Another way to draw the line is to move *everything*
> that incurs external dependency to a separate library.

Avoiding multiple leves of dependencies within APR makes sense to me, 
though it will mean the "core" libapr will end up quite a bit bigger 
than libapr in 1.x. I doubt there are a more than a couple of binaries 
in our main downstream consumers which use only libapr without 
libaprutil already today, so it's not a big deal.

Regards, Joe



Re: RFC: APR 2.0 modularization

2024-04-23 Thread Ivan Zhakov
On Mon, 22 Apr 2024 at 17:01, Joe Orton  wrote:

> On Sat, Apr 20, 2024 at 12:38:20PM +0200, Ivan Zhakov wrote:
> > I would suggest separating APR in different libraries, while keeping them
> > in one project/source tree.
> >
> > Something like that:
> > - apr-2
> > - apr-dbd-2
> > - apr-dbm-2
> > - apr-xlate-2 (?)
> > - apr-crypto-2
> > - apr-xml-2
> > - apr-ldap-2
> > - apr-memcache-2
> > - apr-redis-2
> >
> > I want to clarify that I don't propose going back to apr-util time when
> APR
> > was splitted to separate projects: this proposal is only about libraries
> on
> > the disk.
>

[...]


> Open
> questions on where to draw the line, as to what goes in a separate
> library, and much more I'm sure.
>
Well, it's a good question. I was thinking about something like that:
"everything that can be used in libapr-*-2 should be in libapr-[core]-2".
I mean in **general** we should avoid dependency like app -> libapr-ABC ->
libapr-XYZ -> libapr. Another way to draw the line is to move *everything*
that incurs external dependency to a separate library.

-- 
Ivan Zhakov


Re: RFC: APR 2.0 modularization

2024-04-22 Thread Graham Leggett via dev
On 22 Apr 2024, at 14:30, Ivan Zhakov  wrote:

> Can you confirm why xlate/iconv cannot be moved to a module? (No expectation 
> for you to do such a move, just want to understand what makes it impossible).
>> 
> I think it can be done in theory, but I see potential problem: API should be 
> changed or some kind of implicit initialization should be implemented.
> 
> The same situation with xml API.

My understanding from working on this over the years is that there are layers 
of archeology. The very first external libraries as I recall were libxml2/expat 
and iconv, and these were tightly bound. Then modular DSO became a thing, and 
so DBD/DBM/etc came along with various levels of sophistication.

In theory, making libxml2/expat modular should be straightforward, it involves 
going back and making the old work like the new.

> Another benefit of modular approach is that if you link to apr-xlate-2 or to 
> apr-xml-2 you do not need to handle dynamic DSO load failures.

You would still need to handle failure in the choice of ibxml2 or expat.

>> If you've disabled modular DSO, you've disabled the split of dependent 
>> libraries into separate modules. If this is a problem, re-enable modular DSO?
> There are two things: APR modular DSO support and support for DSO itself. 
> Modular DSO can be disabled while keeping support for dynamic build. APR 
> modular support is configured at compile time and can be disabled by the 
> distribution maintainer. 

It can be disabled by a distribution maintainer, why would they do so though?

In light of what happened with xz, having the ability to harden a library at 
runtime by not-loading-unless-necessary is a very good thing, and you'd need a 
very good reason to turn that off.

Regards,
Graham
--



Re: RFC: APR 2.0 modularization

2024-04-22 Thread Joe Orton
On Sat, Apr 20, 2024 at 12:38:20PM +0200, Ivan Zhakov wrote:
> I would suggest separating APR in different libraries, while keeping them
> in one project/source tree.
> 
> Something like that:
> - apr-2
> - apr-dbd-2
> - apr-dbm-2
> - apr-xlate-2 (?)
> - apr-crypto-2
> - apr-xml-2
> - apr-ldap-2
> - apr-memcache-2
> - apr-redis-2
> 
> I want to clarify that I don't propose going back to apr-util time when APR
> was splitted to separate projects: this proposal is only about libraries on
> the disk.

Yes, exactly right. IIRC someone (Justin?) wrote a proposal on doing it 
like this which had rough consensus around the time we did the big merge 
combining APR+APR-util into a single tree.

Anything "substantial" should be in a separate library, so library 
dependency chains like [app -> libapr-xml-X.so.X -> libexpat.so] are 
maintained, but isolated so that apps don't pull in libexpat. Open 
questions on where to draw the line, as to what goes in a separate 
library, and much more I'm sure.

Regards, Joe



Re: RFC: APR 2.0 modularization

2024-04-22 Thread Ivan Zhakov
On Mon, 22 Apr 2024 at 14:41, Graham Leggett  wrote:

> On 22 Apr 2024, at 12:52, Ivan Zhakov  wrote:
>
> The question is, is it worth it for such a marginal change?
>> Have you looked in to the practicalities and verified no major stumbling
>> blocks,
>> bearing in mind that developer effort tends to be thin on the ground?
>>
>> Take the Apache Subversion for example: it uses only core features from
> APR. But linking to apr 2.0 would lead loading OpenSSL, iconv, ldap (!) to
> the process.
> Modules may help, but it's not complete solution:
> - The are unavailable in static builds
> - Some features cannot be moved to modules (see xlate/iconv and ldap for
> example)
>
>
> I'm not following - in apr-2, LDAP is purely in a module today.
>
> My bad: I misread the code. Yes, LDAP is in a module.


> This is true in apr-util as well.
>
> Can you confirm why xlate/iconv cannot be moved to a module? (No
> expectation for you to do such a move, just want to understand what makes
> it impossible).
>
> I think it can be done in theory, but I see potential problem: API should
be changed or some kind of implicit initialization should be implemented.

The same situation with xml API.

Another benefit of modular approach is that if you link to apr-xlate-2 or
to apr-xml-2 you do not need to handle dynamic DSO load failures.

- Modular DSO can be disabled in compile time for some reason
>
>
> If you've disabled modular DSO, you've disabled the split of dependent
> libraries into separate modules. If this is a problem, re-enable modular
> DSO?
>
There are two things: APR modular DSO support and support for DSO itself.
Modular DSO can be disabled while keeping support for dynamic build. APR
modular support is configured at compile time and can be disabled by the
distribution maintainer.

And modular DSO cannot be enabled in static build, because otherwise the
module will link another instance of apr-core.


>
> I see a lot of work being done on cmake, which is great. Is the isolation
> of module dependencies in modules working in cmake the same way it does in
> autotools?
>
> Thanks. Unfortunately I don't have enough knowledge about autotools, but
as far I understand CMake is similar to autotools.

Off-topic: it would be great to use CMake for all platforms to have only
one build system.


> The problems you're highlighting about static builds are the downsides of
> combining apr and apr-util.
>
>

-- 
Ivan Zhakov


Re: RFC: APR 2.0 modularization

2024-04-22 Thread Graham Leggett via dev
On 22 Apr 2024, at 12:52, Ivan Zhakov  wrote:

>> The question is, is it worth it for such a marginal change?
>> Have you looked in to the practicalities and verified no major stumbling 
>> blocks,
>> bearing in mind that developer effort tends to be thin on the ground?
>> 
> 
> Take the Apache Subversion for example: it uses only core features from APR. 
> But linking to apr 2.0 would lead loading OpenSSL, iconv, ldap (!) to the 
> process. 
> Modules may help, but it's not complete solution: 
> - The are unavailable in static builds 
> - Some features cannot be moved to modules (see xlate/iconv and ldap for 
> example)

I'm not following - in apr-2, LDAP is purely in a module today.

This is true in apr-util as well.

Can you confirm why xlate/iconv cannot be moved to a module? (No expectation 
for you to do such a move, just want to understand what makes it impossible).

> - Modular DSO can be disabled in compile time for some reason

If you've disabled modular DSO, you've disabled the split of dependent 
libraries into separate modules. If this is a problem, re-enable modular DSO?

I see a lot of work being done on cmake, which is great. Is the isolation of 
module dependencies in modules working in cmake the same way it does in 
autotools?

The problems you're highlighting about static builds are the downsides of 
combining apr and apr-util.

Regards,
Graham
--



Re: RFC: APR 2.0 modularization

2024-04-22 Thread Ivan Zhakov
On Mon, 22 Apr 2024 at 11:03, Nick Kew  wrote:

>
>
> > On 20 Apr 2024, at 11:38, Ivan Zhakov  wrote:
> >
> > Hi,
> >
> > In APR 2.0 we merged APR-Util **project** to APR. Which is IMHO a good
> thing.
>
> Um, apr-util was a separate library, but not really a separate project.
>
> By project I mean separate source tree, separate version etc.

The separation may have been a little OTT, but merging them runs into
> "ain't broke, don't fix", with likely confusion arising from the change to
> a
> familiar status-quo.  But that's just a view from about 20 years too late.
>
> > I would suggest separating APR in different libraries, while keeping
> them in one project/source tree.
> >
> > Something like that:
> > - apr-2
> > - apr-dbd-2
> > - apr-dbm-2
> > - apr-xlate-2 (?)
> > - apr-crypto-2
> > - apr-xml-2
> > - apr-ldap-2
> > - apr-memcache-2
> > - apr-redis-2
>
> Works in principle, though packagers and users should probably have an
> apr-everything
> option alongside those.

I expect that it will be one package, but with several libraries inside.


> The question is, is it worth it for such a marginal change?
> Have you looked in to the practicalities and verified no major stumbling
> blocks,
> bearing in mind that developer effort tends to be thin on the ground?
>
> Take the Apache Subversion for example: it uses only core features from
APR. But linking to apr 2.0 would lead loading OpenSSL, iconv, ldap (!) to
the process.
Modules may help, but it's not complete solution:
- The are unavailable in static builds
- Some features cannot be moved to modules (see xlate/iconv and ldap for
example)
- Modular DSO can be disabled in compile time for some reason

-- 
Ivan Zhakov


Re: RFC: APR 2.0 modularization

2024-04-22 Thread Graham Leggett via dev
On 20 Apr 2024, at 11:38, Ivan Zhakov  wrote:

> In APR 2.0 we merged APR-Util **project** to APR. Which is IMHO a good thing.

At the time, and with time and hindsight, I never understood what the point was 
at combining APR and APR-util.

The reason is because...

> But we also merged everything into one big library + modules for loadable 
> drivers, if APR_HAVE_MODULAR_DSO. 

...of this above.

We moved from a lightweight portable library, and a secondary heavier library 
that extended the lightweight library, to one big blob.

> I think it's suboptimal:
> 1. Distributions have to decide what features to compile into this big APR 
> library.
> 2. Linking to APR-2 makes applications depend on multiple libraries, even 
> this particular application is not going to use them.

This above isn't true - that's what the modules are for.

If any libraries are leaking into APR, that's a bug that needs fixing.

For example:

[minfrin@seawitch ~]# ldd /usr/lib64/libaprutil-1.so
linux-vdso.so.1 (0x7fff7159e000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x7f258b619000)
libexpat.so.1 => /lib64/libexpat.so.1 (0x7f258b5e8000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x7f258b5ae000)
libapr-1.so.0 => /lib64/libapr-1.so.0 (0x7f258b56e000)
libc.so.6 => /lib64/libc.so.6 (0x7f258b20)
/lib64/ld-linux-x86-64.so.2 (0x7f258b66b000)
libm.so.6 => /lib64/libm.so.6 (0x7f258b493000)

None of libuuid, expat or crypt should be linked above.

> I would suggest separating APR in different libraries, while keeping them in 
> one project/source tree.
> 
> Something like that:
> - apr-2
> - apr-dbd-2
> - apr-dbm-2
> - apr-xlate-2 (?)
> - apr-crypto-2
> - apr-xml-2
> - apr-ldap-2
> - apr-memcache-2
> - apr-redis-2
> 
> I want to clarify that I don't propose going back to apr-util time when APR 
> was splitted to separate projects: this proposal is only about libraries on 
> the disk.
> 
> What do you think?

I would first fix any leaking library bigs before trying to change anything.

That said, splitting the above would cause as need for eight different "stub" 
libraries, as well as the actual modules themselves, which might get out of 
hand.

Regards,
Graham
--



Re: RFC: APR 2.0 modularization

2024-04-22 Thread Nick Kew



> On 20 Apr 2024, at 11:38, Ivan Zhakov  wrote:
> 
> Hi,
> 
> In APR 2.0 we merged APR-Util **project** to APR. Which is IMHO a good thing.

Um, apr-util was a separate library, but not really a separate project.

The separation may have been a little OTT, but merging them runs into
"ain't broke, don't fix", with likely confusion arising from the change to a
familiar status-quo.  But that's just a view from about 20 years too late.

> I would suggest separating APR in different libraries, while keeping them in 
> one project/source tree.
> 
> Something like that:
> - apr-2
> - apr-dbd-2
> - apr-dbm-2
> - apr-xlate-2 (?)
> - apr-crypto-2
> - apr-xml-2
> - apr-ldap-2
> - apr-memcache-2
> - apr-redis-2

Works in principle, though packagers and users should probably have an 
apr-everything
option alongside those.  The question is, is it worth it for such a marginal 
change?
Have you looked in to the practicalities and verified no major stumbling blocks,
bearing in mind that developer effort tends to be thin on the ground?

-- 
Nick Kew