Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-13 Thread Michael Snoyman 
On Thu, Jul 12, 2012 at 7:30 PM, Tristan Ravitch travi...@cs.wisc.eduwrote:

 On Thu, Jul 12, 2012 at 07:20:39PM +0300, Michael Snoyman wrote:
  On Jul 12, 2012 7:13 PM, Tristan Ravitch travi...@cs.wisc.edu wrote:
  
   On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote:
Are you trying this on a 32 bit system?  And when you compiled that C
program, did you try to add
   
  -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE
   
to the compile command?  When I define those the resulting object
 file
from your example correctly references stat64 instead of stat.
  
   Er sorry, saw your earlier email now.  Could this be a mismatch
   between how your sqlite.so is compiled and how the cbits in
   persistent-sqlite are compiled, particularly with largefile support?
 
  I don't think so. The test case I put together had nothing to do with
  sqlite. Also, persistent-sqlite will either use sqlite.so _or_ the
 included
  sqlite3.c file (based on a compile-time flag). The former works
 perfectly,
  only the latter causes problems.
 
  Michael

 I was looking at the symbols in libc and noticed that it doesn't
 actually export stat64/stat, so that would explain something at least.
 I think your idea about the switch to function pointers versus direct
 calls is probably right - the linker probably does some rewriting of
 calls to stat into __fxstat and company, but for some reason doesn't
 handle references to function pointers.

 I also ran across this stackoverflow post that mentions something
 similar:


 http://stackoverflow.com/questions/5478780/c-and-ld-preload-open-and-open64-calls-intercepted-but-not-stat64

 So stat64 is actually special and in this libc_nonshared.a library (it
 is on my system at least).  It would be ugly to have to link that
 manually - not sure what the right answer is, but at least this might
 explain it.


That's a great find, and it really explains a lot. It seems then that:

* GNU ld has some black magic do know about the stat/stat64 hack.
* Compiling via GHC just uses GNU ld, which is able to make the hack work.
* Interpreting with GHC doesn't get to take advantage of GNU ld's hack.

I've opened a trac ticket[1] for this issue, thank you very much for the
help!

Michael

[1] http://hackage.haskell.org/trac/ghc/ticket/7072
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Michael Snoyman 
On Wed, Jul 11, 2012 at 9:55 PM, Bardur Arantsson s...@scientician.netwrote:

 On 07/11/2012 05:12 PM, Michael Snoyman wrote:
 
  Thanks for the feedback. However, looking at sqlite3.c, I see the
  necessary #include statements:
 
  #include sys/types.h
  #include sys/stat.h
  #include unistd.h
 
  I'm confident that none of my code is making calls to stat/stat64 via
  the FFI. In case it makes a difference, this problem also disappears
  if I compile the library against the system copy of sqlite3 instead of
  using the C source.

 You may need some extra defines, see the comments in man stat64.

 Regards,


 ___
 Haskell-Cafe mailing list
 Haskell-Cafe@haskell.org
 http://www.haskell.org/mailman/listinfo/haskell-cafe


I've come up with a minimal example that demonstrates this problem. The
crux of the matter is the following C code:

#include sys/types.h
#include sys/stat.h
#include unistd.h
#include stdio.h

typedef int stat_func(const char*, struct stat*);

stat_func *foo = stat;

void stat_test(void)
{
struct stat buf;

printf(About to stat-test.c\n);
foo(stat-test.c, buf);
printf(Done\n);
}

As you can see, all of the include statements are present as necessary. The
code compiles just fine with -Wall -Werror. And when you compile the
Haskell code as well, everything works just fine. But if you follow these
steps, you can reproduce the error I saw:

* Unpack the attached tarball
* `cabal install` in that folder
* `runghc main.hs` from the `exe` folder

On my system at least, I get:

main.hs:
/home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
unknown symbol `stat'
main.hs: main.hs: unable to load package `stat-test-0.1.0.0'

One thing I noticed is that I needed to use a function pointer to trigger
the bug. When I called `stat` directly the in stat_test function, gcc
automatically inlined the call, so that the disassessmbled code just showed
a `moveq` (i.e., it's making the system call directly). But using a
function pointer, we're avoiding the inlining. I believe this is why this
issue only came up with the sqlite3 upgrade: previous versions did not use
a function pointer, but rather hard-coded in how to make a stat call.

Does this expose any other possibilities?

Michael


stat-test-0.1.0.0.tar.gz
Description: GNU Zip compressed data
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Michael Snoyman 
On Thu, Jul 12, 2012 at 6:29 PM, Michael Snoyman mich...@snoyman.comwrote:



 On Wed, Jul 11, 2012 at 9:55 PM, Bardur Arantsson s...@scientician.netwrote:

 On 07/11/2012 05:12 PM, Michael Snoyman wrote:
 
  Thanks for the feedback. However, looking at sqlite3.c, I see the
  necessary #include statements:
 
  #include sys/types.h
  #include sys/stat.h
  #include unistd.h
 
  I'm confident that none of my code is making calls to stat/stat64 via
  the FFI. In case it makes a difference, this problem also disappears
  if I compile the library against the system copy of sqlite3 instead of
  using the C source.

 You may need some extra defines, see the comments in man stat64.

 Regards,


 ___
 Haskell-Cafe mailing list
 Haskell-Cafe@haskell.org
 http://www.haskell.org/mailman/listinfo/haskell-cafe


 I've come up with a minimal example that demonstrates this problem. The
 crux of the matter is the following C code:

 #include sys/types.h
  #include sys/stat.h
 #include unistd.h
 #include stdio.h

 typedef int stat_func(const char*, struct stat*);

 stat_func *foo = stat;

 void stat_test(void)
 {
 struct stat buf;

 printf(About to stat-test.c\n);
 foo(stat-test.c, buf);
 printf(Done\n);
 }

 As you can see, all of the include statements are present as necessary.
 The code compiles just fine with -Wall -Werror. And when you compile the
 Haskell code as well, everything works just fine. But if you follow these
 steps, you can reproduce the error I saw:

 * Unpack the attached tarball
 * `cabal install` in that folder
 * `runghc main.hs` from the `exe` folder

 On my system at least, I get:

 main.hs:
 /home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
 unknown symbol `stat'
 main.hs: main.hs: unable to load package `stat-test-0.1.0.0'

 One thing I noticed is that I needed to use a function pointer to trigger
 the bug. When I called `stat` directly the in stat_test function, gcc
 automatically inlined the call, so that the disassessmbled code just showed
 a `moveq` (i.e., it's making the system call directly). But using a
 function pointer, we're avoiding the inlining. I believe this is why this
 issue only came up with the sqlite3 upgrade: previous versions did not use
 a function pointer, but rather hard-coded in how to make a stat call.

 Does this expose any other possibilities?

 Michael


Actually, I just came up with a workaround: declare some local wrappers to
the stat and fstat functions, and use those in place of stat and fstat in
the rest of the code. You can see the change here[1].

Obviously this is a hack, not a real fix. At this point it looks like a GHC
bug to me. Does anything think otherwise? If not, I'll open a ticket.

Michael

[1]
https://github.com/yesodweb/persistent/commit/d7daf0b2fa401fd97ef62e4e74228146d15d8601
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Tristan Ravitch 
On Thu, Jul 12, 2012 at 06:29:41PM +0300, Michael Snoyman wrote:
 I've come up with a minimal example that demonstrates this problem. The
 crux of the matter is the following C code:

 #include sys/types.h
 #include sys/stat.h
 #include unistd.h
 #include stdio.h

 typedef int stat_func(const char*, struct stat*);

 stat_func *foo = stat;

 void stat_test(void)
 {
 struct stat buf;

 printf(About to stat-test.c\n);
 foo(stat-test.c, buf);
 printf(Done\n);
 }

 As you can see, all of the include statements are present as necessary. The
 code compiles just fine with -Wall -Werror. And when you compile the
 Haskell code as well, everything works just fine. But if you follow these
 steps, you can reproduce the error I saw:

 * Unpack the attached tarball
 * `cabal install` in that folder
 * `runghc main.hs` from the `exe` folder

 On my system at least, I get:

 main.hs:
 /home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
 unknown symbol `stat'
 main.hs: main.hs: unable to load package `stat-test-0.1.0.0'

 One thing I noticed is that I needed to use a function pointer to trigger
 the bug. When I called `stat` directly the in stat_test function, gcc
 automatically inlined the call, so that the disassessmbled code just showed
 a `moveq` (i.e., it's making the system call directly). But using a
 function pointer, we're avoiding the inlining. I believe this is why this
 issue only came up with the sqlite3 upgrade: previous versions did not use
 a function pointer, but rather hard-coded in how to make a stat call.

 Does this expose any other possibilities?

 Michael

Are you trying this on a 32 bit system?  And when you compiled that C
program, did you try to add

  -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE

to the compile command?  When I define those the resulting object file
from your example correctly references stat64 instead of stat.


pgpTUPJgg88Ia.pgp
Description: PGP signature
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Tristan Ravitch 
On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote:
 Are you trying this on a 32 bit system?  And when you compiled that C
 program, did you try to add

   -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE

 to the compile command?  When I define those the resulting object file
 from your example correctly references stat64 instead of stat.

Er sorry, saw your earlier email now.  Could this be a mismatch
between how your sqlite.so is compiled and how the cbits in
persistent-sqlite are compiled, particularly with largefile support?


pgpFocJJrnVmJ.pgp
Description: PGP signature
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Michael Snoyman 
On Thu, Jul 12, 2012 at 7:07 PM, Tristan Ravitch travi...@cs.wisc.eduwrote:

 On Thu, Jul 12, 2012 at 06:29:41PM +0300, Michael Snoyman wrote:
  I've come up with a minimal example that demonstrates this problem. The
  crux of the matter is the following C code:
 
  #include sys/types.h
  #include sys/stat.h
  #include unistd.h
  #include stdio.h
 
  typedef int stat_func(const char*, struct stat*);
 
  stat_func *foo = stat;
 
  void stat_test(void)
  {
  struct stat buf;
 
  printf(About to stat-test.c\n);
  foo(stat-test.c, buf);
  printf(Done\n);
  }
 
  As you can see, all of the include statements are present as necessary.
 The
  code compiles just fine with -Wall -Werror. And when you compile the
  Haskell code as well, everything works just fine. But if you follow these
  steps, you can reproduce the error I saw:
 
  * Unpack the attached tarball
  * `cabal install` in that folder
  * `runghc main.hs` from the `exe` folder
 
  On my system at least, I get:
 
  main.hs:
 
 /home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
  unknown symbol `stat'
  main.hs: main.hs: unable to load package `stat-test-0.1.0.0'
 
  One thing I noticed is that I needed to use a function pointer to trigger
  the bug. When I called `stat` directly the in stat_test function, gcc
  automatically inlined the call, so that the disassessmbled code just
 showed
  a `moveq` (i.e., it's making the system call directly). But using a
  function pointer, we're avoiding the inlining. I believe this is why this
  issue only came up with the sqlite3 upgrade: previous versions did not
 use
  a function pointer, but rather hard-coded in how to make a stat call.
 
  Does this expose any other possibilities?
 
  Michael

 Are you trying this on a 32 bit system?  And when you compiled that C
 program, did you try to add

   -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE

 to the compile command?  When I define those the resulting object file
 from your example correctly references stat64 instead of stat.


I'm compiling on a 64 bit system. If I add those definitions, the program
uses stat64 instead, but the only difference is that runghc now prints:

main.hs:
/home/ubuntu/.cabal/lib/stat-test-0.1.0.0/ghc-7.4.1/HSstat-test-0.1.0.0.o:
unknown symbol `stat64'
main.hs: main.hs: unable to load package `stat-test-0.1.0.0'

In other words, it's not the symbol that the object file is referencing
(stat vs stat64) that's the problem: runghc is not able to resolve either
one.

Michael
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Michael Snoyman 
On Jul 12, 2012 7:13 PM, Tristan Ravitch travi...@cs.wisc.edu wrote:

 On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote:
  Are you trying this on a 32 bit system?  And when you compiled that C
  program, did you try to add
 
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE
 
  to the compile command?  When I define those the resulting object file
  from your example correctly references stat64 instead of stat.

 Er sorry, saw your earlier email now.  Could this be a mismatch
 between how your sqlite.so is compiled and how the cbits in
 persistent-sqlite are compiled, particularly with largefile support?

I don't think so. The test case I put together had nothing to do with
sqlite. Also, persistent-sqlite will either use sqlite.so _or_ the included
sqlite3.c file (based on a compile-time flag). The former works perfectly,
only the latter causes problems.

Michael
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-12 Thread Tristan Ravitch 
On Thu, Jul 12, 2012 at 07:20:39PM +0300, Michael Snoyman wrote:
 On Jul 12, 2012 7:13 PM, Tristan Ravitch travi...@cs.wisc.edu wrote:
 
  On Thu, Jul 12, 2012 at 11:07:05AM -0500, Tristan Ravitch wrote:
   Are you trying this on a 32 bit system?  And when you compiled that C
   program, did you try to add
  
 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE
  
   to the compile command?  When I define those the resulting object file
   from your example correctly references stat64 instead of stat.
 
  Er sorry, saw your earlier email now.  Could this be a mismatch
  between how your sqlite.so is compiled and how the cbits in
  persistent-sqlite are compiled, particularly with largefile support?

 I don't think so. The test case I put together had nothing to do with
 sqlite. Also, persistent-sqlite will either use sqlite.so _or_ the included
 sqlite3.c file (based on a compile-time flag). The former works perfectly,
 only the latter causes problems.

 Michael

I was looking at the symbols in libc and noticed that it doesn't
actually export stat64/stat, so that would explain something at least.
I think your idea about the switch to function pointers versus direct
calls is probably right - the linker probably does some rewriting of
calls to stat into __fxstat and company, but for some reason doesn't
handle references to function pointers.

I also ran across this stackoverflow post that mentions something
similar:

  
http://stackoverflow.com/questions/5478780/c-and-ld-preload-open-and-open64-calls-intercepted-but-not-stat64

So stat64 is actually special and in this libc_nonshared.a library (it
is on my system at least).  It would be ugly to have to link that
manually - not sure what the right answer is, but at least this might
explain it.


pgpFjqMeqMQZk.pgp
Description: PGP signature
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-11 Thread Michael Snoyman 
Hi all,

A quick search indicates that this problem has come up in the past,
but I haven't seen any solutions yet. I'm working on the next
Persistent release, and one of the changes is that the included
sqlite3 C library has been updated (I believe that's the trigger
here). I can compile programs against persistent-sqlite, but if
there's TH code involved, or I try to runghc the file, I get an error
message like:

test.hs: 
/home/ubuntu/.cabal/lib/persistent-sqlite-1.0.0/ghc-7.4.1/HSpersistent-sqlite-1.0.0.o:
unknown symbol `stat64'
test.hs: test.hs: unable to load package `persistent-sqlite-1.0.0'

I'm running GHC 7.4.1 on Ubuntu 12.04 64-bit. Does anyone have insight
into what might be causing this?

Thanks,
Michael

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-11 Thread Brandon Allbery 
On Wed, Jul 11, 2012 at 10:25 AM, Michael Snoyman mich...@snoyman.comwrote:

 test.hs:
 /home/ubuntu/.cabal/lib/persistent-sqlite-1.0.0/ghc-7.4.1/HSpersistent-sqlite-1.0.0.o:
 unknown symbol `stat64'
 test.hs: test.hs: unable to load package `persistent-sqlite-1.0.0'


The immediate cause is that some C source file is calling stat() or lstat()
without the right #include files; they go through several levels of
backward compatibility macros that end in different system calls.
 Alternately, something is trying to use one of those functions via the FFI
instead of System.Posix.File.

-- 
brandon s allbery  allber...@gmail.com
wandering unix systems administrator (available) (412) 475-9364 vm/sms
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-11 Thread Michael Snoyman 
On Wed, Jul 11, 2012 at 5:47 PM, Brandon Allbery allber...@gmail.com wrote:
 On Wed, Jul 11, 2012 at 10:25 AM, Michael Snoyman mich...@snoyman.com
 wrote:

 test.hs:
 /home/ubuntu/.cabal/lib/persistent-sqlite-1.0.0/ghc-7.4.1/HSpersistent-sqlite-1.0.0.o:
 unknown symbol `stat64'
 test.hs: test.hs: unable to load package `persistent-sqlite-1.0.0'


 The immediate cause is that some C source file is calling stat() or lstat()
 without the right #include files; they go through several levels of backward
 compatibility macros that end in different system calls.  Alternately,
 something is trying to use one of those functions via the FFI instead of
 System.Posix.File.

 --
 brandon s allbery  allber...@gmail.com
 wandering unix systems administrator (available) (412) 475-9364 vm/sms


Hi Brandon,

Thanks for the feedback. However, looking at sqlite3.c, I see the
necessary #include statements:

#include sys/types.h
#include sys/stat.h
#include unistd.h

I'm confident that none of my code is making calls to stat/stat64 via
the FFI. In case it makes a difference, this problem also disappears
if I compile the library against the system copy of sqlite3 instead of
using the C source.

Michael

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Re: [Haskell-cafe] ghci and TH cannot: unknown symbol `stat64`

2012-07-11 Thread Bardur Arantsson 
On 07/11/2012 05:12 PM, Michael Snoyman wrote:
 
 Thanks for the feedback. However, looking at sqlite3.c, I see the
 necessary #include statements:
 
 #include sys/types.h
 #include sys/stat.h
 #include unistd.h
 
 I'm confident that none of my code is making calls to stat/stat64 via
 the FFI. In case it makes a difference, this problem also disappears
 if I compile the library against the system copy of sqlite3 instead of
 using the C source.

You may need some extra defines, see the comments in man stat64.

Regards,


___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe