[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AWesterinen]

AWesterinen added a comment.


  @AndySeaborne  Agree. I was erring on the side of explaining where the SPARQL 
endpoint came from (not Jena TDB).

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AWesterinen
Cc: AWesterinen, Osmasuominen, dcausse, Smalyshev, Aklapper, 
Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, Susannaanas, Akuckartz, 
TomT0m, Jecummings4, Krabina, So9q, Salgo60, WMDE-leszek, GreenReaper, 
Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, Lydia_Pintscher, DanBri, 
Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, Afandian, Sj, TallTed, Tpt, 
Thadguidry, danshick-wmde, Hjfocs, Mohammed_Sadat_WMDE, MarioGom, 
karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, Izno, RShigapov, Hannah_Bast, 
Kjauslin, toan, Michael, DD063520, AndreasKuczera, Versant.2612, namedgraph, 
Iamamz3, YULdigitalpreservation, BenAtOlive, nguyenm9, Fnielsen, 
accounting_data_logger, JohannesKalmbach, Dr.uesenfieber, Bovlb, AndySeaborne, 
BeautifulBold, Suran38, Invadibot, MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, 
NavinRizwi, CBogen, Isaacandy, Demian, Olson.jared.m, Nandana, Namenlos314, 
Lahi, Gq86, Bryandamon, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, 
PhotographerTom, suriyaa, Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, 
MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne added a comment.


  @AWesterinen - Fuseki is part of Jena. Most of the subsystems have informal 
names. People refer to "Jena" or "Fuseki" interchangeably and the context is 
the task they are doing. Being more specific on naming didn't catch on.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: AWesterinen, Osmasuominen, dcausse, Smalyshev, Aklapper, 
Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, Susannaanas, Akuckartz, 
TomT0m, Jecummings4, Krabina, So9q, Salgo60, WMDE-leszek, GreenReaper, 
Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, Lydia_Pintscher, DanBri, 
Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, Afandian, Sj, TallTed, Tpt, 
Thadguidry, danshick-wmde, Hjfocs, Mohammed_Sadat_WMDE, MarioGom, 
karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, Izno, RShigapov, Hannah_Bast, 
Kjauslin, toan, Michael, DD063520, AndreasKuczera, Versant.2612, namedgraph, 
Iamamz3, YULdigitalpreservation, BenAtOlive, nguyenm9, Fnielsen, 
accounting_data_logger, JohannesKalmbach, Dr.uesenfieber, Bovlb, AndySeaborne, 
BeautifulBold, Suran38, Invadibot, MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, 
NavinRizwi, CBogen, Isaacandy, Demian, Olson.jared.m, Nandana, Namenlos314, 
Lahi, Gq86, Bryandamon, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, 
PhotographerTom, suriyaa, Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, 
MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne added a comment.


  @Thadguidry -
  
  https://lists.apache.org/thread/vso02pwg4z6qcs3r1h0mcbc86ls74bhm
  
  where --parallel (the argument on sort(1) that is set by --threads)  was set 
to 16.
  
  It took 31h compared to 39h without --parallel on sort(1).

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: AWesterinen, Osmasuominen, dcausse, Smalyshev, Aklapper, 
Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, Susannaanas, Akuckartz, 
TomT0m, Jecummings4, Krabina, So9q, Salgo60, WMDE-leszek, GreenReaper, 
Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, Lydia_Pintscher, DanBri, 
Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, Afandian, Sj, TallTed, Tpt, 
Thadguidry, danshick-wmde, Hjfocs, Mohammed_Sadat_WMDE, MarioGom, 
karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, Izno, RShigapov, Hannah_Bast, 
Kjauslin, toan, Michael, DD063520, AndreasKuczera, Versant.2612, namedgraph, 
Iamamz3, YULdigitalpreservation, BenAtOlive, nguyenm9, Fnielsen, 
accounting_data_logger, JohannesKalmbach, Dr.uesenfieber, Bovlb, AndySeaborne, 
BeautifulBold, Suran38, Invadibot, MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, 
NavinRizwi, CBogen, Isaacandy, Demian, Olson.jared.m, Nandana, Namenlos314, 
Lahi, Gq86, Bryandamon, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, 
PhotographerTom, suriyaa, Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, 
MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AWesterinen]

AWesterinen added a comment.


  You add Fuseki to Jena to get a SPARQL endpoint.  Jena + Fuseki is reasonable 
to investigate as a Blazegraph Alternative.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AWesterinen
Cc: AWesterinen, Osmasuominen, dcausse, Smalyshev, Aklapper, 
Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, Susannaanas, Akuckartz, 
TomT0m, Jecummings4, Krabina, So9q, Salgo60, WMDE-leszek, GreenReaper, 
Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, Lydia_Pintscher, DanBri, 
Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, Afandian, Sj, TallTed, Tpt, 
Thadguidry, danshick-wmde, Hjfocs, Mohammed_Sadat_WMDE, MarioGom, 
karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, Izno, RShigapov, Hannah_Bast, 
Kjauslin, toan, Michael, DD063520, AndreasKuczera, Versant.2612, namedgraph, 
Iamamz3, YULdigitalpreservation, BenAtOlive, nguyenm9, Fnielsen, 
accounting_data_logger, JohannesKalmbach, Dr.uesenfieber, Bovlb, AndySeaborne, 
BeautifulBold, Suran38, Invadibot, MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, 
NavinRizwi, CBogen, Isaacandy, Demian, Olson.jared.m, Nandana, Namenlos314, 
Lahi, Gq86, Bryandamon, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, 
PhotographerTom, suriyaa, Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, 
aude, Tobias1984, Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, 
MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[Thadguidry]

Thadguidry added a comment.


  Hi @AndySeaborne What is the latest benchmarks for loading Wikidata all and 
truthy with Jena 4.4.0 release annd the new TDB2 xloader with "--threads" 
argument?  I noticed the release notes said this:
  
  > == Improved bulk loader
  >
  > This release includes the version of the TDB2 xloader for very large 
  > datasets.
  >
  > It has been used to load 16.6B triples (WikiData all) into TDB2 and 
  > loading truthy (6B) on modest hardware. Thanks to Marco, Lorenz and 
  > Øyvind for running Wikidata load trails.
  >
  > The loader now now has "--threads=" which been reported to give improved 
  > load times  (if the server has the hardware!).

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Thadguidry
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[MPhamWMF]

MPhamWMF triaged this task as "Medium" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[DD063520]

DD063520 added a comment.


  @So9q : How would you like to serve everything from one place? It is normal 
to have replica of data. One of the big bottlenecks is IO. Or do I understand 
something wrong?

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: DD063520
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[So9q]

So9q added a comment.


  I read the whole thread and just want to point out that Jena supports SPARQL 
Update also.
  
  From what I can see, it seems to be able to replace Blazegraph. But it does 
not solve the issue of having multiple parallel servers all with their own 
snapshot of the current WD triples.
  
  Maybe it is currently not possible to avoid that, but it would be nice to 
have all the triples in ONE place and serve them from multiple servers who 
handle the SPARQL-requests.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: So9q
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[dcausse]

dcausse added a comment.


  Sorry for the confusion that the rename I did of this task caused.
  Just to bring clarity on my reasoning as a maintainer of the wikidata query 
service stack as to why being specific on TDB2 might be helpful:
  
  - Some components of Jena are already being used (i.e. the sparql parser for 
query analysis)
  - Jena has been considered in 2015 but declined ref: T90112 
 (sadly no reasons were given)
  
  This task is I think about evaluating Jena and its storage component as a 
storage/query engine for Wikidata Query Service but it does not mean that all 
of what Jena offers will be discarded if this task is declined.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[Osmasuominen]

Osmasuominen added a comment.


  I have to agree with @AndySeaborne - talking about "Apache Jena with TDB2" 
makes as much sense as talking about "VW Beetle with an internal combustion 
engine". The framing makes it sound like Beetles come with all kinds of 
engines, though in reality they've all been equipped with an ICE at the factory 
so far. There are electric conversion kits for hobbyists etc. but that's a 
really marginal thing and needs its own discussion.
  
  Similarly, TDB(2) is an integral component of Apache Jena - by far the most 
common setup and the only one supported by the Apache Jena project. It would be 
possible to compare TDB1 vs. TDB2, but those are just iterations of the same 
storage technology, TDB1 is on the way out, and any new evaluations should be 
made with TDB2.
  
  Disclaimer: I'm a developer (committer & PMC member) for Apache Jena as well 
as a contributor to Wikidata (esp. mappings to controlled vocabularies such as 
YSO and GACS).

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Osmasuominen
Cc: Osmasuominen, dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, 
Andrawaag, Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, 
So9q, Salgo60, WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, 
Tagishsimon, Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, 
Justin0x2004, Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne added a comment.


  All - I'm sorry that this sub-task is being redirected to be about Virtuoso. 
This would be better moved to the Virtuoso task.
  
  Apache Jena releases a single software product. TDB is the only persistence 
layer for Apache Jena that comes from the Apache Jena project.
  
  The link @TallTed gives is to Virtuoso-specific documentation. The software 
does not come from the Apache Jena project.
  
  TDB is the _internal_ name for a component which is the B+Trees. TDB2 is the 
current generation of that component.
  
  SPARQL is important aspect for Wikidata and the first code example shows the 
bypass of Apache Jena SPARQL execution and only thin use of the Java API.
  
  ---
  
  @TallTed: The examples on the page do not describe Virtuoso used as a "low 
level storage choice" for SPARQL execution; it shows complete bypass of the 
Jena.
  
  It should be on the task for evaluating Virtuoso because it is what OpenLink 
is providing.  It is 5% Jena (API layer) and 95% Virtuoso.  All performance and 
data scale characteristics are down to Virtuoso.
  
  I do not understand why WikiData usage would want to bypass the Virtuoso 
triplestore HTTP interface but if you want that considered, it would be better 
as part of the Virtuoso evaluation. It can be compared to the same approach 
with other code APIs accessing Virtuoso.
  
  The diagram, at best, it might be said to relate to the design of the 
research prototype Jena1 (over 15 years ago) many years before Jena became 
Apache Jena. SPARQL didn't exist for that architecture which predates W3C work 
on SPARQL.
  
  - SPARQL evaluation does not go through the Model API.
  - Apache Jena does not provide storage in SQL databases anymore.
  - TDB does not store models.
  - TDB isn't even mentioned on the diagram.
  
  The page you link to talks about Jena 2.6, which is not an Apache release, 
and Jena 2.10.0 is 2013-02-24 - during the transition to Apache Jena.
  
  Virtuoso can provide fine-grained access with VirtGraph but that is not how 
TDB fits into Jena.
  Using VirtGraph might get Virtuoso users SHACL/ShEx support but that isn't 
the focus for WikiData as I understand it.
  
  If you want to discuss the general integration of Virtuoso and Jena, then 
let's take that to the Jena mailing lists.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, 
Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[TallTed]

TallTed added a comment.


  @AndySeaborne -- It has been my understanding that Apache Jena (the 
framework) performs differently (which may include different speeds of various 
actions, which may have different limitations and/or comprise a different list) 
when the active "low level storage choice" (such as TDB or TDB2) is changed, 
such as from TDB to TDB2 to Virtuoso 
,
 to any engine that offers a Data Provider for Jena (or vice versa).
  
  If my past understanding remains correct, I think the title of this task 
would be appropriately changed to //Evaluate Apache Jena with TDB2//, and that 
there ought to be some parallel tasks created with titles adjusted to include 
the "low level storage choice" made for that task, one of which should be 
//Evaluate Apache Jena with Virtuoso// in that role.
  
  There may be other variables which may make sense within a single evaluation 
task, and others which may make more sense as a distinct evaluation task.
  
  If you think it only makes sense for an omnibus task to //Evaluate Apache 
Jena//, then I submit that there should at least be multiple subtasks with the 
"low level storage choice" variability I've described above.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: TallTed
Cc: dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, 
Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne added a subscriber: dcausse.
AndySeaborne added a comment.


  Hi @dcausse - TDB2 on it's own doesn't provide SPARQL nor any of the other 
features. TDB2 is just one low level storage choice - it's not a standalone 
thing. I hope you find the description clearer now.
  
  The project has had other 3rd party organisations presenting their own naming 
and architecture descriptions of Jena so I just wanted to be clear here.
  
  If there is anything else I can help with - please do ask.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: dcausse, Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, 
Addshore, Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne renamed this task from "Evaluate Apache Jena TDB2" to "Evaluate 
Apache Jena".
AndySeaborne updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, 
Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena TDB2

[dcausse]

dcausse renamed this task from "Evaluate Apache Jena" to "Evaluate Apache Jena 
TDB2".
dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, 
Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, 
Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T299460: Evaluate Apache Jena

[AndySeaborne]

AndySeaborne created this task.
AndySeaborne added projects: Wikidata-Query-Service, Epic, Wikidata, 
MediaWiki-Stakeholders-Group.

TASK DESCRIPTION
  Apache Jena https://jena.apache.org/ provides SPARQL 1.1, SHACL (core and 
SPARQL), ShEx, and RDF-star.
  
  Jena has been reported to 
[https://lists.apache.org/thread/wqr8vg43v2kd3ofrncn1tk9lxy078p83 load Wikidata 
(20211222_latest-all.nt.gz)] (16.7 billion triples at 44.8k triples/second in 
103h 45m 15s).
  
  Jena has been reported to 
[https://lists.apache.org/thread/wvtr4ohjwm5tm9z77q702fbsz2o7gbp2 load Wikidata 
Truthy (2021-12)] (6.6 billion triples in 40 hours, at 46k
  triples/second).
  
  Jena has various extension mechanism for incorporating extensions, including
  overloading the SERVICE keyword.
  
  Jena might provide a way for users to load the data, or a focused subset of 
the
  data, for local use thereby potentially offloading the central SPARQL service.

TASK DETAIL
  https://phabricator.wikimedia.org/T299460

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: AndySeaborne
Cc: Smalyshev, Aklapper, Lucas_Werkmeister_WMDE, Gehel, Andrawaag, Addshore, 
Susannaanas, Akuckartz, TomT0m, Jecummings4, Krabina, So9q, Salgo60, 
WMDE-leszek, GreenReaper, Ostrzyciel, Samantha_Alipio_WMDE, Tagishsimon, 
Lydia_Pintscher, DanBri, Jneubert, Ivanhercaz, TheKtk, Jerven, Justin0x2004, 
Afandian, Sj, TallTed, Tpt, Thadguidry, danshick-wmde, Hjfocs, 
Mohammed_Sadat_WMDE, MarioGom, karapayneWMDE, Daniel_Mietchen, KingsleyIdehen, 
Izno, RShigapov, Hannah_Bast, Kjauslin, toan, Michael, DD063520, 
AndreasKuczera, Versant.2612, namedgraph, Iamamz3, YULdigitalpreservation, 
BenAtOlive, nguyenm9, Fnielsen, accounting_data_logger, JohannesKalmbach, 
Dr.uesenfieber, Bovlb, AndySeaborne, BeautifulBold, Suran38, Invadibot, 
MPhamWMF, Jtm-lis, maantietaja, Peteosx1x, NavinRizwi, CBogen, Isaacandy, 
Demian, Olson.jared.m, Nandana, Namenlos314, Lahi, Gq86, Bryandamon, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, Steko, Samwilson, PhotographerTom, suriyaa, 
Psychoslave, tosfos, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Darenwelsh, Dinoguy1000, Manybubbles, brion, Mbch331, MarkAHershberger
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org