Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-11-20 Thread Sergio Fernández 
Hi Sebastian,

On Sat, Nov 19, 2016 at 2:24 PM, Sebastian Schaffert <
sebastian.schaff...@gmail.com> wrote:

> So I got that one. The dataset contains a line
>
>  <
> http://wikidata.dbpedia.org/property/birthDate> "-0106-01-03"^^<
> http://www.w3.org/2001/XMLSchema#date> .
>
> with a negative date that starts with a 0, which is invalid according to
> the XMLSchema data datatype (https://www.w3.org/TR/xmlschema11-2/#date)
> where a 0 is not allowed as first character of the year. PostgreSQL in any
> case doesn't like it.
>
> I am trying to work around the problem.
>

I saw the patch:
https://github.com/apache/marmotta/commit/08e6f79d595a3ce22d95e612825bfcbc3f9c6686

Thanks, I'll give it a try.

Honestly I didn't have much time to actually debug the issue. How do you
debug such kind of issues on wrong statements in batches produced by the
loader?

Cheers,


Sergio Fernández  schrieb am Di., 11. Okt. 2016 um
> 10:24 Uhr:
>
> > On Tue, Aug 9, 2016 at 9:55 PM, Sebastian Schaffert <
> > sebastian.schaff...@gmail.com> wrote:
> >
> > > Do you have a concrete test file for which it fails? I can try then.
> Just
> > > don't want to download all the DBPedia language versions... ;-)
> > >
> >
> > I think
> >
> > http://downloads.dbpedia.org/2016-04/core-i18n/ta/infobox_
> properties_wkd_uris_ta.ttl.bz2
> > is the one causing issues loading on KiWi... But I have to double check
> > that. It'd be great is someone else could give it a try. ;-)
> >
> >
> > --
> > Sergio Fernández
> > Partner Technology Manager
> > Redlink GmbH
> > m: +43 6602747925 <+43%20660%202747925>
> > e: sergio.fernan...@redlink.co
> > w: http://redlink.co
> >
>



-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernan...@redlink.co
w: http://redlink.co

Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-11-19 Thread Sebastian Schaffert 
Ok, I fixed this, and the dataset now imports.

Note that the problem will be worse in other databases than PostgreSQL.
MySQL doesn't support dates before Christ at all.

Sebastian

Sebastian Schaffert  schrieb am Sa., 19.
Nov. 2016 um 14:24 Uhr:

> So I got that one. The dataset contains a line
>
>  <
> http://wikidata.dbpedia.org/property/birthDate> "-0106-01-03"^^<
> http://www.w3.org/2001/XMLSchema#date> .
>
> with a negative date that starts with a 0, which is invalid according to
> the XMLSchema data datatype (https://www.w3.org/TR/xmlschema11-2/#date)
> where a 0 is not allowed as first character of the year. PostgreSQL in any
> case doesn't like it.
>
> I am trying to work around the problem.
>
>
>
>
> Sergio Fernández  schrieb am Di., 11. Okt. 2016 um
> 10:24 Uhr:
>
> On Tue, Aug 9, 2016 at 9:55 PM, Sebastian Schaffert <
> sebastian.schaff...@gmail.com> wrote:
>
> > Do you have a concrete test file for which it fails? I can try then. Just
> > don't want to download all the DBPedia language versions... ;-)
> >
>
> I think
>
> http://downloads.dbpedia.org/2016-04/core-i18n/ta/infobox_properties_wkd_uris_ta.ttl.bz2
> is the one causing issues loading on KiWi... But I have to double check
> that. It'd be great is someone else could give it a try. ;-)
>
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925 <+43%20660%202747925>
> e: sergio.fernan...@redlink.co
> w: http://redlink.co
>
>

Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-11-19 Thread Sebastian Schaffert 
So I got that one. The dataset contains a line

 <
http://wikidata.dbpedia.org/property/birthDate> "-0106-01-03"^^<
http://www.w3.org/2001/XMLSchema#date> .

with a negative date that starts with a 0, which is invalid according to
the XMLSchema data datatype (https://www.w3.org/TR/xmlschema11-2/#date)
where a 0 is not allowed as first character of the year. PostgreSQL in any
case doesn't like it.

I am trying to work around the problem.




Sergio Fernández  schrieb am Di., 11. Okt. 2016 um
10:24 Uhr:

> On Tue, Aug 9, 2016 at 9:55 PM, Sebastian Schaffert <
> sebastian.schaff...@gmail.com> wrote:
>
> > Do you have a concrete test file for which it fails? I can try then. Just
> > don't want to download all the DBPedia language versions... ;-)
> >
>
> I think
>
> http://downloads.dbpedia.org/2016-04/core-i18n/ta/infobox_properties_wkd_uris_ta.ttl.bz2
> is the one causing issues loading on KiWi... But I have to double check
> that. It'd be great is someone else could give it a try. ;-)
>
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925 <+43%20660%202747925>
> e: sergio.fernan...@redlink.co
> w: http://redlink.co
>

Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-10-11 Thread Sergio Fernández 
On Tue, Aug 9, 2016 at 9:55 PM, Sebastian Schaffert <
sebastian.schaff...@gmail.com> wrote:

> Do you have a concrete test file for which it fails? I can try then. Just
> don't want to download all the DBPedia language versions... ;-)
>

I think
http://downloads.dbpedia.org/2016-04/core-i18n/ta/infobox_properties_wkd_uris_ta.ttl.bz2
is the one causing issues loading on KiWi... But I have to double check
that. It'd be great is someone else could give it a try. ;-)


-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernan...@redlink.co
w: http://redlink.co

Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-08-09 Thread Sebastian Schaffert 
Do you have a concrete test file for which it fails? I can try then. Just
don't want to download all the DBPedia language versions... ;-)

Sergio Fernández  schrieb am Di., 9. Aug. 2016 um
17:04 Uhr:

> No idea how to debug that... but I have the same issue with DBpedia 2016-04
> Beta: http://downloads.dbpedia.org/2016-04/
>
> On Tue, Aug 2, 2016 at 11:42 AM, Sebastian Schaffert <
> sebastian.schaff...@gmail.com> wrote:
>
> > Hi Sergio,
> >
> > do you have some more debugging information? How does the SQL statement
> > look like? Is there an error?
> >
> > One option is to turn on SQL logging in PostgreSQL.
> >
> > Also, KiWiLoader is using batched CSV dumps that it then imports into
> > PostgreSQL. Perhaps there is just some CSV formatting problem? You could
> > write the CSV into a file in addition to streaming it to PostgreSQL.
> >
> > Sebastian
> >
> > Sergio Fernández  schrieb am Di., 26. Juli 2016 um
> > 10:25 Uhr:
> >
> > > Has anybody tried to load the latest release of DBpedia (2015-10) in
> > > Marmotta/PostgreSQL using our bulk-loader?
> > >
> > > Some statements produce a (SQL) SELECT query that never ends... and
> > > honestly I have no idea where to start to debug what :-/
> > >
> > > Thanks in advance.
> > >
> > > --
> > > Sergio Fernández
> > > Partner Technology Manager
> > > Redlink GmbH
> > > m: +43 6602747925
> > > e: sergio.fernan...@redlink.co
> > > w: http://redlink.co
> > >
> >
>
>
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925
> e: sergio.fernan...@redlink.co
> w: http://redlink.co
>

Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-08-09 Thread Sergio Fernández 
No idea how to debug that... but I have the same issue with DBpedia 2016-04
Beta: http://downloads.dbpedia.org/2016-04/

On Tue, Aug 2, 2016 at 11:42 AM, Sebastian Schaffert <
sebastian.schaff...@gmail.com> wrote:

> Hi Sergio,
>
> do you have some more debugging information? How does the SQL statement
> look like? Is there an error?
>
> One option is to turn on SQL logging in PostgreSQL.
>
> Also, KiWiLoader is using batched CSV dumps that it then imports into
> PostgreSQL. Perhaps there is just some CSV formatting problem? You could
> write the CSV into a file in addition to streaming it to PostgreSQL.
>
> Sebastian
>
> Sergio Fernández  schrieb am Di., 26. Juli 2016 um
> 10:25 Uhr:
>
> > Has anybody tried to load the latest release of DBpedia (2015-10) in
> > Marmotta/PostgreSQL using our bulk-loader?
> >
> > Some statements produce a (SQL) SELECT query that never ends... and
> > honestly I have no idea where to start to debug what :-/
> >
> > Thanks in advance.
> >
> > --
> > Sergio Fernández
> > Partner Technology Manager
> > Redlink GmbH
> > m: +43 6602747925
> > e: sergio.fernan...@redlink.co
> > w: http://redlink.co
> >
>



-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernan...@redlink.co
w: http://redlink.co

Re: issues loading DBpedia 2015-10 with KiWiLoader

2016-08-02 Thread Sebastian Schaffert 
Hi Sergio,

do you have some more debugging information? How does the SQL statement
look like? Is there an error?

One option is to turn on SQL logging in PostgreSQL.

Also, KiWiLoader is using batched CSV dumps that it then imports into
PostgreSQL. Perhaps there is just some CSV formatting problem? You could
write the CSV into a file in addition to streaming it to PostgreSQL.

Sebastian

Sergio Fernández  schrieb am Di., 26. Juli 2016 um
10:25 Uhr:

> Has anybody tried to load the latest release of DBpedia (2015-10) in
> Marmotta/PostgreSQL using our bulk-loader?
>
> Some statements produce a (SQL) SELECT query that never ends... and
> honestly I have no idea where to start to debug what :-/
>
> Thanks in advance.
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925
> e: sergio.fernan...@redlink.co
> w: http://redlink.co
>