Re: [sqlite] .import error: cannot open large file

[Simon Slavin]

On 15 Nov 2013, at 9:23am, Luís Simão  wrote:

> Not sure if it helps, but you could try
> 
>sqlite3 somedb '.import /dev/stdin hugetable' < huge.file
> 
> making OS handle file reading instead of another process (cat).

If the source really is a file then the right way to do it is

sqlite3 somedb '.import huge.file hugetable'

and if that results in an error the error message should be more accurate and 
useful.  If something goes wrong with 'stdin' then it's hard for a program to 
know what happened.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[Luís Simão]

Not sure if it helps, but you could try

sqlite3 somedb '.import /dev/stdin hugetable' < huge.file

making OS handle file reading instead of another process (cat).

LS

2013/11/14 lpryszcz 

> Hi, Often I pipe tables (from .gz or multiple files). I found it also work
> for large files that otherwise fail with `Error: cannot open :huge.file"`:
>
> cat huge.file | sqlite3 somedb '.import /dev/stdin hugetable'
>
> But it could be slower that using `real` file import. Anyone have an idea?
>
> L.
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[lpryszcz]

Hi, Often I pipe tables (from .gz or multiple files). I found it also work
for large files that otherwise fail with `Error: cannot open :huge.file"`:

cat huge.file | sqlite3 somedb '.import /dev/stdin hugetable'

But it could be slower that using `real` file import. Anyone have an idea?

L. 



--
View this message in context: 
http://sqlite.1065341.n5.nabble.com/import-error-cannot-open-large-file-tp27346p72364.html
Sent from the SQLite mailing list archive at Nabble.com.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[Roger Binns]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 06/06/2011 04:47 PM, Rense Corten wrote:
> Just to report back on this issue: recompiling as per Nuno's
> instructions indeed solved the problem. Still, if anyone can explain
> to me why the original executable would work without problems on a
> different machine, I would be grateful.

You'd need to use strace to find out what system calls are behaving
differently/failing.

The lack of large file support for the shell is a known problem:

  http://www.sqlite.org/src/tktview?name=92af7da36b

Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iEYEARECAAYFAk3tocgACgkQmOOfHg372QREkgCg1M++4/zFH6EwcJQqEVakd2tt
A8AAn1G9wOEVOqIN8mC2ZTwz1cPHoQS1
=Y0ma
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[Rense Corten]

Just to report back on this issue: recompiling as per Nuno's
instructions indeed solved the problem. Still, if anyone can explain
to me why the original executable would work without problems on a
different machine, I would be grateful.

>
> On Fri, Jun 3, 2011 at 10:28 AM, Rense Corten  wrote:
>> Thanks for your answer, Nuno.
>> However, the system I am using is already 64-bit (I should have
>> mentioned that) , and the same binary can do the job on another Ubuntu
>> 64-bit  system. I'll try your suggestion nevertheless, but can there
>> be other causes?
>> Rense
>>
>>> It's what the thread says. The SQLite shell on Ubuntu (on 11.04) isn't
>>> compiled with large file support on 32-bit systems, so while the
>>> SQLite library does work with 64-bit database, the shell doesn't.
>>> The easy solution is to either use a 64-bit Ubuntu system or compile
>>> the shell yourself with large file support.
>>>
>>> To compile it, download the sqlite amalgamation files and run:
>>>
>>> gcc -o sqli -O3 -DNDEBUG=1 -D_FILE_OFFSET_BITS=64 sqlite3.c shell.c
>>> -ldl -pthread
>>>
>>> The resulting binary (sqli) will be compiled with large file support
>>> (I verified it was using strace).
>>>
>>>
>>> Regards,
>>> ~Nuno Lucas
>>>
>>> P.S.- While this could be considered an Ubuntu bug, the truth is that
>>> the linux shell binary on the sqlite site also isn't compiled with
>>> large file support, so I would consider this an SQLite bug.
>>>
>>>
>>> --
>>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[Nuno Lucas]

On Fri, Jun 3, 2011 at 18:28, Rense Corten  wrote:
> Thanks for your answer, Nuno.
> However, the system I am using is already 64-bit (I should have
> mentioned that) , and the same binary can do the job on another Ubuntu
> 64-bit  system. I'll try your suggestion nevertheless, but can there
> be other causes?
> Rense

There can always be other causes, like permission problems. But from
what you say I'm tended to believe your problem is the one I
mentioned.
Note that although your system is 64-bits, the pre-compiled sqlite
binary is 32-bits.

But you can check yourself using "strace".
Run the shell as:

$ strace -e open sqlite  ...args...

You will see all open system calls, and for you to be able to open
large files you must see something like:

open("file.csv", O_RDONLY|O_LARGEFILE) = 4

Notice the O_LARGEFILE flag. If the shell is not compiled for large
file access that flag will not be present.
The latter is what happens using the pre-compiled sqlite binary.


Regards,
~Nuno Lucas
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[Rense Corten]

Thanks for your answer, Nuno.
However, the system I am using is already 64-bit (I should have
mentioned that) , and the same binary can do the job on another Ubuntu
64-bit  system. I'll try your suggestion nevertheless, but can there
be other causes?
Rense

> It's what the thread says. The SQLite shell on Ubuntu (on 11.04) isn't
> compiled with large file support on 32-bit systems, so while the
> SQLite library does work with 64-bit database, the shell doesn't.
> The easy solution is to either use a 64-bit Ubuntu system or compile
> the shell yourself with large file support.
>
> To compile it, download the sqlite amalgamation files and run:
>
> gcc -o sqli -O3 -DNDEBUG=1 -D_FILE_OFFSET_BITS=64 sqlite3.c shell.c
> -ldl -pthread
>
> The resulting binary (sqli) will be compiled with large file support
> (I verified it was using strace).
>
>
> Regards,
> ~Nuno Lucas
>
> P.S.- While this could be considered an Ubuntu bug, the truth is that
> the linux shell binary on the sqlite site also isn't compiled with
> large file support, so I would consider this an SQLite bug.
>
>
> --
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] .import error: cannot open large file

[Nuno Lucas]

Hello,

On Fri, Jun 3, 2011 at 00:08, Rense Corten  wrote:
[...]
> So I searched the archives of this list and found two threads on this:
>
> http://www.mail-archive.com/sqlite-users@sqlite.org/msg51574.html
> http://www.mail-archive.com/sqlite-users@sqlite.org/msg48649.html
>
> The first thread got no answers, but the second suggest to either
> split the file or recompile sqlite3 with the option for large file
> support . Now I had understood that since version 3.5.9 large file
> support is switched on by default so that should not be the problem
> (http://www.sqlite.org/changes.html). Splitting the file, however,
> seems to solve the problem. I would prefer not to have to split the
> file first.
>
> Any ideas on what causes this problem?

It's what the thread says. The SQLite shell on Ubuntu (on 11.04) isn't
compiled with large file support on 32-bit systems, so while the
SQLite library does work with 64-bit database, the shell doesn't.
The easy solution is to either use a 64-bit Ubuntu system or compile
the shell yourself with large file support.

To compile it, download the sqlite amalgamation files and run:

gcc -o sqli -O3 -DNDEBUG=1 -D_FILE_OFFSET_BITS=64 sqlite3.c shell.c
-ldl -pthread

The resulting binary (sqli) will be compiled with large file support
(I verified it was using strace).


Regards,
~Nuno Lucas

P.S.- While this could be considered an Ubuntu bug, the truth is that
the linux shell binary on the sqlite site also isn't compiled with
large file support, so I would consider this an SQLite bug.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] .import error: cannot open large file

[Rense Corten]

Hi all,

I'm trying to import a large file of about 13GB using SQLite 3.7.6.3
on Ubuntu. I use the precompiled Linux binary.

The commands are:
=
.separator ";"
.import largefile.csv mytable
=

but then I get: "Error: cannot open "largefile.csv" "

I can view the file with "head" or "less", so there seems to be no
problem with readability or permissions. Moreover, I can (partially)
import the same file on a different Ubuntu system using the exact same
commands (but run out of storage space before the import completes).

So I searched the archives of this list and found two threads on this:

http://www.mail-archive.com/sqlite-users@sqlite.org/msg51574.html
http://www.mail-archive.com/sqlite-users@sqlite.org/msg48649.html

The first thread got no answers, but the second suggest to either
split the file or recompile sqlite3 with the option for large file
support . Now I had understood that since version 3.5.9 large file
support is switched on by default so that should not be the problem
(http://www.sqlite.org/changes.html). Splitting the file, however,
seems to solve the problem. I would prefer not to have to split the
file first.

Any ideas on what causes this problem?
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users