Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Tom Lane]

I wrote:
> I still think that this represents under-engineering by the ICU crew
> and not anything we're doing wrong.

After some more examination of their readme.html, I tried this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1 --enable-rpath

and that gave me sane-looking libraries:

$ otool -L /usr/local/icu-57.1/lib/libicui18n.57.1.dylib 
/usr/local/icu-57.1/lib/libicui18n.57.1.dylib:
/usr/local/icu-57.1/lib/libicui18n.57.dylib (compatibility version 
57.0.0, current version 57.1.0)
/usr/local/icu-57.1/lib/libicuuc.57.dylib (compatibility version 
57.0.0, current version 57.1.0)
/usr/local/icu-57.1/lib/libicudata.57.dylib (compatibility version 
57.0.0, current version 57.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 1238.60.2)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 
307.5.0)

--- note that now libicudata is shown as a dependency.  And then
configuring Postgres as I showed before leads to executables that
build, and initdb, and pass regression tests, and have a bunch
of ICU entries in pg_collation, without needing to fool with
DYLD_LIBRARY_PATH.

Their configure --help claims

  --enable-rpath  use rpath when linking default is only if necessary

but apparently that's for small values of "necessary" :-(

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Sandeep Thakkar]

On Thu, Jul 13, 2017 at 8:23 AM, Tom Lane  wrote:

> Sandeep Thakkar  writes:
> > On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane  wrote:
> >> and this is evidently because the libraries themselves don't know where
> >> they live:
> >> $ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
> >> /usr/local/icu-57.1/lib/libicui18n.57.dylib:
> >> libicui18n.57.dylib
>
> > Right. I got that and I fixed the names and loader_paths for ICU libs and
> > postgres and that is why initdb in my case got going and didn't complain
> > about library not found.
>
> Uh, so what did you do *exactly*?
>
> I changed the id and the loader_paths for ICU libs and also changed the
install name of ICU libs on postgres binary
$ otool -L postgres  | grep icu
@loader_path/../lib/libicuuc.57.dylib (compatibility version 57.0.0,
current version 57.1.0)
@loader_path/../lib/libicui18n.57.dylib (compatibility version 57.0.0,
current version 57.1.0)


> >> I can make it work by setting DYLD_LIBRARY_PATH:
> >> $ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
> >> ... goes through normally ...
>
> > You mean you are able to initialize cluster after this? Or you just
> > executed initdb and found that it doesn't complain about ICU lib
> location?
>
> initdb completed successfully.  I didn't try running any tests beyond
> that; I'm not sure that we have any regression tests that will exercise
> ICU locales.
>
> OK. I'll check at my end then.


> > As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the
> > loader_paths and am able to execute initdb and postgres binaries:
> > But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".
>
> Yeah, but notice that only two of the three interesting ICU libraries
> are actually linked into the postgres executable so far as otool and
> the dynamic linker are concerned.  I suspect that the other one,
> libicudata, is dynamically loaded by the ICU code --- and in your
> configuration it fails to find that library.  The error message is
> not definitive that that's what's happening, but it's suggestive.
> If that's the right interpretation, it means that setting
> DYLD_LIBRARY_PATH allows that third library to be found, but whatever
> you did doesn't.
>
> Yeah, I observed that otool -L lists only two of three ICU libs. But not
sure why it doesn't load the third one in my case. Here is the otool -D and
otool -L output on ICU libs:
$ otool -D ../lib/libicuuc.dylib
../lib/libicuuc.dylib:
libicuuc.57.dylib

$ otool -L ../lib/libicuuc.dylib
../lib/libicuuc.dylib:
libicuuc.57.dylib (compatibility version 57.0.0, current version 57.1.0)
@loader_path/../lib/libicudata.57.dylib (compatibility version 57.0.0,
current version 57.1.0)

$ otool -D ../lib/libicui18n.dylib
../lib/libicui18n.dylib:
libicui18n.57.dylib

$ otool -L ../lib/libicui18n.dylib
../lib/libicui18n.dylib:
libicui18n.57.dylib (compatibility version 57.0.0, current version 57.1.0)
@loader_path/../lib/libicuuc.57.dylib (compatibility version 57.0.0,
current version 57.1.0)
@loader_path/../lib/libicudata.57.dylib (compatibility version 57.0.0,
current version 57.1.0)

$ otool -D ../lib/libicudata.dylib
../lib/libicudata.dylib:
libicudata.57.dylib

$ otool -L ../lib/libicudata.dylib
../lib/libicudata.dylib:
libicudata.57.dylib (compatibility version 57.0.0, current version 57.1.0)


> I still think that this represents under-engineering by the ICU crew
> and not anything we're doing wrong.
>
> BTW, when I skimmed the "readme.html" docs in the ICU sources this
> morning, I noted that there were multiple ways you could configure
> it to find the ICU data.  I did not read in detail, figuring that
> their default configuration would be sane, but maybe that was an
> overly charitable assumption.
>
> Sure, thanks for checking this out. It seems the issue is with ICU and
something I did which is why the libicudata.dylib is not able to load. I'll
dig deeper. Thanks for your help.


> regards, tom lane
>



-- 
Sandeep Thakkar
EDB

Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Tom Lane]

Sandeep Thakkar  writes:
> On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane  wrote:
>> and this is evidently because the libraries themselves don't know where
>> they live:
>> $ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
>> /usr/local/icu-57.1/lib/libicui18n.57.dylib:
>> libicui18n.57.dylib

> Right. I got that and I fixed the names and loader_paths for ICU libs and
> postgres and that is why initdb in my case got going and didn't complain
> about library not found.

Uh, so what did you do *exactly*?

>> I can make it work by setting DYLD_LIBRARY_PATH:
>> $ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
>> ... goes through normally ...

> You mean you are able to initialize cluster after this? Or you just
> executed initdb and found that it doesn't complain about ICU lib location?

initdb completed successfully.  I didn't try running any tests beyond
that; I'm not sure that we have any regression tests that will exercise
ICU locales.

> As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the
> loader_paths and am able to execute initdb and postgres binaries:
> But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".

Yeah, but notice that only two of the three interesting ICU libraries
are actually linked into the postgres executable so far as otool and
the dynamic linker are concerned.  I suspect that the other one,
libicudata, is dynamically loaded by the ICU code --- and in your
configuration it fails to find that library.  The error message is
not definitive that that's what's happening, but it's suggestive.
If that's the right interpretation, it means that setting
DYLD_LIBRARY_PATH allows that third library to be found, but whatever
you did doesn't.

I still think that this represents under-engineering by the ICU crew
and not anything we're doing wrong.

BTW, when I skimmed the "readme.html" docs in the ICU sources this
morning, I noted that there were multiple ways you could configure
it to find the ICU data.  I did not read in detail, figuring that
their default configuration would be sane, but maybe that was an
overly charitable assumption.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Sandeep Thakkar]

On Thu, Jul 13, 2017 at 12:44 AM, Tom Lane  wrote:

> Sandeep Thakkar  writes:
> > On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane  wrote:
> >> Ugh.  Please provide details about the hand-built ICU --- what version,
> >> what configure options did you use for it, etc.
>
> > I tried with ICU 53.1 and 57.1 and the results is same.
> > There was no configure options other than --prefix.
>
> Well, when I try that on macOS, I get an ICU installation that's basically
> nonfunctional, apparently because they forgot to specify -install_name
> while linking the dylibs.  The symptom looks a bit different:
>
> $ initdb
> dyld: Library not loaded: libicui18n.57.dylib
>   Referenced from: /Users/tgl/testversion/bin/postgres
>   Reason: image not found
>
> but it's clear from otool that the dynamic linker doesn't know where
> to find the ICU libraries:
>
> $ otool -L ~/testversion/bin/postgres
> /Users/tgl/testversion/bin/postgres:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 1238.60.2)
> libicui18n.57.dylib (compatibility version 57.0.0, current version
> 57.1.0)
> libicuuc.57.dylib (compatibility version 57.0.0, current version
> 57.1.0)
>
> and this is evidently because the libraries themselves don't know where
> they live:
>
> $ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
> /usr/local/icu-57.1/lib/libicui18n.57.dylib:
> libicui18n.57.dylib
>
> Right. I got that and I fixed the names and loader_paths for ICU libs and
postgres and that is why initdb in my case got going and didn't complain
about library not found.


> Compare to what you get for a properly built dylib:
>
> $ otool -D /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib
> /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib:
> /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib
>
> I can make it work by setting DYLD_LIBRARY_PATH:
>
> $ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
> ... goes through normally ...
>
> You mean you are able to initialize cluster after this? Or you just
executed initdb and found that it doesn't complain about ICU lib location?
As mentioned above instead of using DYLD_LIBRARY_PATH, I fixed the
loader_paths and am able to execute initdb and postgres binaries:
$ ./initdb -V
initdb (PostgreSQL) 10beta2

$ ./postgres -V
postgres (PostgreSQL) 10beta2

But, initdb -D data fails with error code "U_FILE_ACCESS_ERROR".


> but that hardly qualifies as a production-grade solution.
>
> I've not dug into the ICU sources to see what it would take to get the
> correct link option applied.  But the short answer seems to be that ICU
> is not up to speed for installation in non-default locations on macOS.
>
> For the record's sake, I configured icu4c-57_1-src.tgz like this:
>
> $ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1
>
> and since there's no pkg-config on this machine, I did this to
> configure Postgres:
>
> $ ICU_CFLAGS=" " ICU_LIBS="-licui18n -licuuc -licudata" ./configure [usual
> options] --with-icu --with-includes=/usr/local/icu-57.1/include
> --with-libraries=/usr/local/icu-57.1/lib
>
> It's possible that if I did have pkg-config installed, it would have
> produced some different value for ICU_LIBS, but I rather doubt it.
> I'm not sure that it's possible to fix this on the consumer executable's
> link line anyway.
>
> yes, we use the same flags during configure.


> regards, tom lane
>



-- 
Sandeep Thakkar
EDB

Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Tom Lane]

Sandeep Thakkar  writes:
> On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane  wrote:
>> Ugh.  Please provide details about the hand-built ICU --- what version,
>> what configure options did you use for it, etc.

> I tried with ICU 53.1 and 57.1 and the results is same.
> There was no configure options other than --prefix.

Well, when I try that on macOS, I get an ICU installation that's basically
nonfunctional, apparently because they forgot to specify -install_name
while linking the dylibs.  The symptom looks a bit different:

$ initdb
dyld: Library not loaded: libicui18n.57.dylib
  Referenced from: /Users/tgl/testversion/bin/postgres
  Reason: image not found

but it's clear from otool that the dynamic linker doesn't know where
to find the ICU libraries:

$ otool -L ~/testversion/bin/postgres
/Users/tgl/testversion/bin/postgres:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 1238.60.2)
libicui18n.57.dylib (compatibility version 57.0.0, current version 
57.1.0)
libicuuc.57.dylib (compatibility version 57.0.0, current version 57.1.0)

and this is evidently because the libraries themselves don't know where
they live:

$ otool -D /usr/local/icu-57.1/lib/libicui18n.57.dylib
/usr/local/icu-57.1/lib/libicui18n.57.dylib:
libicui18n.57.dylib

Compare to what you get for a properly built dylib:

$ otool -D /usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib 
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib:
/usr/local/ssl-1.1.0e/lib/libssl.1.1.dylib

I can make it work by setting DYLD_LIBRARY_PATH:

$ DYLD_LIBRARY_PATH=/usr/local/icu-57.1/lib initdb
... goes through normally ...

but that hardly qualifies as a production-grade solution.

I've not dug into the ICU sources to see what it would take to get the
correct link option applied.  But the short answer seems to be that ICU
is not up to speed for installation in non-default locations on macOS.

For the record's sake, I configured icu4c-57_1-src.tgz like this:

$ ./runConfigureICU MacOSX --prefix=/usr/local/icu-57.1

and since there's no pkg-config on this machine, I did this to
configure Postgres:

$ ICU_CFLAGS=" " ICU_LIBS="-licui18n -licuuc -licudata" ./configure [usual 
options] --with-icu --with-includes=/usr/local/icu-57.1/include 
--with-libraries=/usr/local/icu-57.1/lib

It's possible that if I did have pkg-config installed, it would have
produced some different value for ICU_LIBS, but I rather doubt it.
I'm not sure that it's possible to fix this on the consumer executable's
link line anyway.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Sandeep Thakkar]

On Wed, Jul 12, 2017 at 8:13 PM, Tom Lane  wrote:

> Sandeep Thakkar  writes:
> > On MacOS, I configured PG10 beta2 sources with '--with-icu" option along
> > with ICU_LIBS and ICU_CFLAGS env variables to define the location of the
> > ICU libs that I built. Once the staging is ready, I executed initdb but
> it
> > fails with the following error:
> > [53618] FATAL:  could not open collator for locale "und":
> > U_FILE_ACCESS_ERROR*
> > *2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT
> > pg_import_system_collations('pg_catalog');*
>
> Ugh.  Please provide details about the hand-built ICU --- what version,
> what configure options did you use for it, etc.
>
> I tried with ICU 53.1 and 57.1 and the results is same.
There was no configure options other than --prefix. In environment,
CXXFLAGS was set to "-isysroot
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk
-mmacosx-version-min=10.7 -arch i386 -arch x86_64"


regards, tom lane
>



-- 
Sandeep Thakkar
EDB

Re: [HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Tom Lane]

Sandeep Thakkar  writes:
> On MacOS, I configured PG10 beta2 sources with '--with-icu" option along
> with ICU_LIBS and ICU_CFLAGS env variables to define the location of the
> ICU libs that I built. Once the staging is ready, I executed initdb but it
> fails with the following error:
> [53618] FATAL:  could not open collator for locale "und":
> U_FILE_ACCESS_ERROR*
> *2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT
> pg_import_system_collations('pg_catalog');*

Ugh.  Please provide details about the hand-built ICU --- what version,
what configure options did you use for it, etc.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] PostgreSQL10 beta2 with ICU - initdb fails on MacOS

[Sandeep Thakkar]

Hi

On MacOS, I configured PG10 beta2 sources with '--with-icu" option along
with ICU_LIBS and ICU_CFLAGS env variables to define the location of the
ICU libs that I built. Once the staging is ready, I executed initdb but it
fails with the following error:
--
$./initdb -D /tmp/data
The files belonging to this database system will be owned by user
"buildfarm".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /tmp/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
*performing post-bootstrap initialization ... 2017-07-12 11:18:21.187 BST
[53618] FATAL:  could not open collator for locale "und":
U_FILE_ACCESS_ERROR*
*2017-07-12 11:18:21.187 BST [53618] STATEMENT:  SELECT
pg_import_system_collations('pg_catalog');*
child process exited with exit code 1
initdb: removing data directory "/tmp/data"
--

Has anyone came across this? This error is not seen on Linux and Windows
though.

-- 
Sandeep Thakkar
EDB