Re: Jena next (AFS)

[ajs6f]

No problem! Streams and Iterators have confused me myself, but luckily Andy
and other folks helped straighten me out.

Andy, you've given a nice list of potential discussions and others have as
well. My meta-question is when do we want to switch to tickets for this
process? I don't want to smother discussion in process, but I find it very
hard to follow a multithreaded discussion over email and I much prefer
breaking things out early to more specifics.

Shall I start capturing (or trying to) ideas, or is it too soon?

ajs6f

On Tue, Nov 19, 2019, 11:49 AM Bögershausen, Merlin Michael <
merlin.boegershau...@rwth-aachen.de> wrote:

> Seems like I've got the wrong feeling from early messages, revisited Andys
> Post missed the "additional provide" part, sorry.
>
> Am 19.11.2019 16:48 schrieb "A. Soroka" :
> I think there may be some confusion here about Streams and Iterators.
> Streams are not and were never intended to be a replacement for or
> equivalent to Iterators. Iterators are a source of elements. There is no
> further complexity. Streams, on the other hand, are pipelines of
> computation which do not begin flowing until a terminal operation is called
> and then the data flows through the pipe without further action from the
> caller.
>
> We should not even consider replacing Iterators with Streams and I did not
> hear anyone suggest it. The suggestion I heard (and which I thoroughly
> support) is to offer Streams alongside Iterators for their intended
> purpose; pipelining computations.
>
> For example, consider a dataset backed by a remote SPARQL connection.
> Suppose stream() is called on a graph from this dataset, and suppose
> limit() is called on that Stream. Then in the right conditions a smart
> implementation could push that limit upstream to become a LIMIT in the
> SPARQL being sent over the wire. That's the real value of all those methods
> on Stream. They aren't merely for developer convenience. A library like our
> Iter would do for that. They are there to provide opportunities to classify
> and manage the computations being pipelined.
>
> So Stream is very useful for its purpose, but not for Iterator's purpose.
> As for crossing module boundaries, I haven't seen any problems with that
> and I'm not sure what the objection actually is. In fact, refusing to
> expose
> Streams at APIs seems to make the whole thing rather pointless.
>
> ajs6f
>
>
> On Tue, Nov 19, 2019, 10:06 AM Merlin Bögershausen <
> merlin.boegershau...@rwth-aachen.de> wrote:
>
> > Hi,
> >
> > To the Stream discussion: Streams should not be passed beyond module
> > borders. In my personal view not even beyond scopes, because the
> > exceptions thrown by an already terminated stream can be misleading.
> > Every user can feel free to use the StreamSupport class whenever the
> > application code uses streams.
> >
> > Another thing, I would like to see a graph view inside Fuseki UI, where
> > the triples are visualized like in
> > 
> > I use such visualization whenever I explain the usage of our graph data
> > model to my colleagues, whenever they have problems to develop SPARQL
> > queries. Also, I feel that it would help our support team to see why
> > the system performs unexpected steps.
> >
> > Merlin
> >
> >
>
>

Re: Jena next (AFS)

[Bögershausen , Merlin Michael]

Seems like I've got the wrong feeling from early messages, revisited Andys Post 
missed the "additional provide" part, sorry.

Am 19.11.2019 16:48 schrieb "A. Soroka" :
I think there may be some confusion here about Streams and Iterators.
Streams are not and were never intended to be a replacement for or
equivalent to Iterators. Iterators are a source of elements. There is no
further complexity. Streams, on the other hand, are pipelines of
computation which do not begin flowing until a terminal operation is called
and then the data flows through the pipe without further action from the
caller.

We should not even consider replacing Iterators with Streams and I did not
hear anyone suggest it. The suggestion I heard (and which I thoroughly
support) is to offer Streams alongside Iterators for their intended
purpose; pipelining computations.

For example, consider a dataset backed by a remote SPARQL connection.
Suppose stream() is called on a graph from this dataset, and suppose
limit() is called on that Stream. Then in the right conditions a smart
implementation could push that limit upstream to become a LIMIT in the
SPARQL being sent over the wire. That's the real value of all those methods
on Stream. They aren't merely for developer convenience. A library like our
Iter would do for that. They are there to provide opportunities to classify
and manage the computations being pipelined.

So Stream is very useful for its purpose, but not for Iterator's purpose.
As for crossing module boundaries, I haven't seen any problems with that
and I'm not sure what the objection actually is. In fact, refusing to expose
Streams at APIs seems to make the whole thing rather pointless.

ajs6f


On Tue, Nov 19, 2019, 10:06 AM Merlin Bögershausen <
merlin.boegershau...@rwth-aachen.de> wrote:

> Hi,
>
> To the Stream discussion: Streams should not be passed beyond module
> borders. In my personal view not even beyond scopes, because the
> exceptions thrown by an already terminated stream can be misleading.
> Every user can feel free to use the StreamSupport class whenever the
> application code uses streams.
>
> Another thing, I would like to see a graph view inside Fuseki UI, where
> the triples are visualized like in
> 
> I use such visualization whenever I explain the usage of our graph data
> model to my colleagues, whenever they have problems to develop SPARQL
> queries. Also, I feel that it would help our support team to see why
> the system performs unexpected steps.
>
> Merlin
>
>

Re: Jena next (AFS)

[A. Soroka]

I think there may be some confusion here about Streams and Iterators.
Streams are not and were never intended to be a replacement for or
equivalent to Iterators. Iterators are a source of elements. There is no
further complexity. Streams, on the other hand, are pipelines of
computation which do not begin flowing until a terminal operation is called
and then the data flows through the pipe without further action from the
caller.

We should not even consider replacing Iterators with Streams and I did not
hear anyone suggest it. The suggestion I heard (and which I thoroughly
support) is to offer Streams alongside Iterators for their intended
purpose; pipelining computations.

For example, consider a dataset backed by a remote SPARQL connection.
Suppose stream() is called on a graph from this dataset, and suppose
limit() is called on that Stream. Then in the right conditions a smart
implementation could push that limit upstream to become a LIMIT in the
SPARQL being sent over the wire. That's the real value of all those methods
on Stream. They aren't merely for developer convenience. A library like our
Iter would do for that. They are there to provide opportunities to classify
and manage the computations being pipelined.

So Stream is very useful for its purpose, but not for Iterator's purpose.
As for crossing module boundaries, I haven't seen any problems with that
and I'm not sure what the objection actually is. In fact, refusing to expose
Streams at APIs seems to make the whole thing rather pointless.

ajs6f


On Tue, Nov 19, 2019, 10:06 AM Merlin Bögershausen <
merlin.boegershau...@rwth-aachen.de> wrote:

> Hi,
>
> To the Stream discussion: Streams should not be passed beyond module
> borders. In my personal view not even beyond scopes, because the
> exceptions thrown by an already terminated stream can be misleading.
> Every user can feel free to use the StreamSupport class whenever the
> application code uses streams.
>
> Another thing, I would like to see a graph view inside Fuseki UI, where
> the triples are visualized like in
> 
> I use such visualization whenever I explain the usage of our graph data
> model to my colleagues, whenever they have problems to develop SPARQL
> queries. Also, I feel that it would help our support team to see why
> the system performs unexpected steps.
>
> Merlin
>
>

Re: Jena next (AFS)

[Merlin Bögershausen]

Hi,

To the Stream discussion: Streams should not be passed beyond module 
borders. In my personal view not even beyond scopes, because the 
exceptions thrown by an already terminated stream can be misleading.
Every user can feel free to use the StreamSupport class whenever the 
application code uses streams.


Another thing, I would like to see a graph view inside Fuseki UI, where 
the triples are visualized like in 

I use such visualization whenever I explain the usage of our graph data 
model to my colleagues, whenever they have problems to develop SPARQL 
queries. Also, I feel that it would help our support team to see why 
the system performs unexpected steps.


Merlin

Re: Jena next (AFS)

[Andy Seaborne]

On 17/11/2019 23:07, Claude Warren wrote:

I am a bit concerned about Streams.

I am working with some large scale streams from stored objects in another
project and keep coming up against stack overflow issues when attempting to
convert merge  them or convert from iterators.  Perhaps I have not done it
correctly but the iterator approach seems cleaner when you don't have or
can't have all the data in memory at once.


Interesting - have you got some stackOverflow links?

https://www.beyondjava.net/performance-java-8-lambdas
diucussed stream costs a bit (and interstign that lambda are not the issue)

Some of the speed reports of stream() are comparing streams with 
replacing loops - and that clearly is going to have am impact if the 
loop body is small, let along the fact the JIT probably knows how to do 
magic with plain loops.


Maybe, sometime, streams() will get JIT attention.



We might consider switching from the Jena specific iterators to
commons-collections4 (perhaps contributing some additions there).


Iter has nearly all the stream functions whereas commons-collections4 is 
a peckage of iterator classes that have to be nested.


So Iter can used like streams for nice code use (and is more complete 
than ExtendedIterator) and makes switching to streams easy, not zero but 
the syntax work needed is lessened.  In fact - that's a good goal for Iter.


(the missing bits are "collect" because Iter has Iter.toList and 
Iter.toSet - I do find the Stream.collect([Collectors.]toList) and 
absence of the direct form a bit odd - Adding Iter.collect is no bug 
deal thoiugh for completeness).


Andy



Claude

On Sun, Nov 17, 2019 at 5:34 PM Andy Seaborne  wrote:


This is a bit of a brain dump ...

== DatasetGraph

Graph Triple, Quad, DatasetGraph in a single API place.

== Graph - SPI

Graph - add a few navigation operations to make writing system directly
on Graph easier - though still not as rich as the Model API, and avoid
much of the object churn.

The operations are (not final names)

Graph.fwd(subject, predicate)
 -- return a single Node or null.
Graph.fwdList(subject, predicate)
 -- return a list of Nodes
Graph.fwdUnique(subject, predicate)
 -- return a single Node, exception if 0 or more than one.

Same for "bwk"


https://github.com/apache/jena/blob/master/jena-shacl/src/main/java/org/apache/jena/shacl/lib/G.java
is a library version of this that was helpful but adding a few
operations directly to graph

If the data is known to be good (SHACL), the application code can use
fwd()/bwk() without worrying about testing for zero or multiple predicates.

The reason for putting the basic oprations in the Graph interface and
not everything in a library is for potential efficiency. An impl may be
able to do a good job of fwd() and if that is the basis of graph
analytics efficiency matters long term, at least not to design it out.

== Assembler

The graph SPI additions is also motivated by assemblers.  Assemblers are
currently Model/Resource based but the important usage is in Fuseki - an
ideal goal is Fuseki works on Graph/Node.

Converting assemblers to Graph/Node does not look too burdensome and
with a wrapper layer we can hopefully include all the old tests to check
evolution.

== Graph - indexing

Currently, Graphs are term-indexed only or value-indexed, not both.

Graph should plain term-indexed. value-indexing, which can be calculated
on the fly, would be a separate higher-level concept.

This is motivated by scale and having the same behaviour on all graph.
At scale, canonicalizing the inputs is better than value-indexing.

"values" would only be in the Model API.

== Transactions

Unify the transaction approach (also changes Model) so complex
assemblages of graphs, and other things,  are transactional.

Remove graph transactions - replace by
org.apache.jena.sparql.core.Transactional.

Then graphs as views of datasets and also combinations of Transactionals
in single transaction (two DatasetGraph, or collection of Graphs (teh
assmebler case)) can be done.

== Events

Make events an intercepting wrapper, not built-in to Graph itself.
Add transaction lifecycle events.

== Streams - yes and no.

A Stream is several java objects so a potential cost
for a simple operations like Graph.contains() or find() or a few things
is not small.

Keep iterators, provide stream(s,p,o).

== Nodes

Lang tags - force to lower case.

Simplify - remove a layer of indirection. This relates to indexing.

Node_Literal - no LiteralLabels
Node_Blank - two longs or a string label, not using BlankNodeId

Investigate integrate nodes with ARQ's NodeValue.

== IRIs

jena-iri is general, powerful and hard to maintain.
Jena does not use all of it.
Jena needs a simpler, direct parser/checker.

https://github.com/afs/iri4ld

which is a parser in java with little copying. It parse URIs, and then
has a little on scheme specific rules for http(s), file and URN.

The various open source 

Re: Jena next (AFS)

[Claude Warren]

I am a bit concerned about Streams.

I am working with some large scale streams from stored objects in another
project and keep coming up against stack overflow issues when attempting to
convert merge  them or convert from iterators.  Perhaps I have not done it
correctly but the iterator approach seems cleaner when you don't have or
can't have all the data in memory at once.

We might consider switching from the Jena specific iterators to
commons-collections4 (perhaps contributing some additions there).

Claude

On Sun, Nov 17, 2019 at 5:34 PM Andy Seaborne  wrote:

> This is a bit of a brain dump ...
>
> == DatasetGraph
>
> Graph Triple, Quad, DatasetGraph in a single API place.
>
> == Graph - SPI
>
> Graph - add a few navigation operations to make writing system directly
> on Graph easier - though still not as rich as the Model API, and avoid
> much of the object churn.
>
> The operations are (not final names)
>
>Graph.fwd(subject, predicate)
> -- return a single Node or null.
>Graph.fwdList(subject, predicate)
> -- return a list of Nodes
>Graph.fwdUnique(subject, predicate)
> -- return a single Node, exception if 0 or more than one.
>
> Same for "bwk"
>
>
> https://github.com/apache/jena/blob/master/jena-shacl/src/main/java/org/apache/jena/shacl/lib/G.java
> is a library version of this that was helpful but adding a few
> operations directly to graph
>
> If the data is known to be good (SHACL), the application code can use
> fwd()/bwk() without worrying about testing for zero or multiple predicates.
>
> The reason for putting the basic oprations in the Graph interface and
> not everything in a library is for potential efficiency. An impl may be
> able to do a good job of fwd() and if that is the basis of graph
> analytics efficiency matters long term, at least not to design it out.
>
> == Assembler
>
> The graph SPI additions is also motivated by assemblers.  Assemblers are
> currently Model/Resource based but the important usage is in Fuseki - an
> ideal goal is Fuseki works on Graph/Node.
>
> Converting assemblers to Graph/Node does not look too burdensome and
> with a wrapper layer we can hopefully include all the old tests to check
> evolution.
>
> == Graph - indexing
>
> Currently, Graphs are term-indexed only or value-indexed, not both.
>
> Graph should plain term-indexed. value-indexing, which can be calculated
> on the fly, would be a separate higher-level concept.
>
> This is motivated by scale and having the same behaviour on all graph.
> At scale, canonicalizing the inputs is better than value-indexing.
>
> "values" would only be in the Model API.
>
> == Transactions
>
> Unify the transaction approach (also changes Model) so complex
> assemblages of graphs, and other things,  are transactional.
>
> Remove graph transactions - replace by
> org.apache.jena.sparql.core.Transactional.
>
> Then graphs as views of datasets and also combinations of Transactionals
> in single transaction (two DatasetGraph, or collection of Graphs (teh
> assmebler case)) can be done.
>
> == Events
>
> Make events an intercepting wrapper, not built-in to Graph itself.
> Add transaction lifecycle events.
>
> == Streams - yes and no.
>
> A Stream is several java objects so a potential cost
> for a simple operations like Graph.contains() or find() or a few things
> is not small.
>
> Keep iterators, provide stream(s,p,o).
>
> == Nodes
>
> Lang tags - force to lower case.
>
> Simplify - remove a layer of indirection. This relates to indexing.
>
> Node_Literal - no LiteralLabels
> Node_Blank - two longs or a string label, not using BlankNodeId
>
> Investigate integrate nodes with ARQ's NodeValue.
>
> == IRIs
>
> jena-iri is general, powerful and hard to maintain.
> Jena does not use all of it.
> Jena needs a simpler, direct parser/checker.
>
> https://github.com/afs/iri4ld
>
> which is a parser in java with little copying. It parse URIs, and then
> has a little on scheme specific rules for http(s), file and URN.
>
> The various open source libraries and JDK classes do not track the
> current standards very well (RFC 2396 vs RFC 3986). I have found that
> compliance is mixed due to legacy compatibility needs.
>


-- 
I like: Like Like - The likeliest place on the web

LinkedIn: http://www.linkedin.com/in/claudewarren

Jena next (AFS)

[Andy Seaborne]

This is a bit of a brain dump ...

== DatasetGraph

Graph Triple, Quad, DatasetGraph in a single API place.

== Graph - SPI

Graph - add a few navigation operations to make writing system directly 
on Graph easier - though still not as rich as the Model API, and avoid 
much of the object churn.


The operations are (not final names)

  Graph.fwd(subject, predicate)
   -- return a single Node or null.
  Graph.fwdList(subject, predicate)
   -- return a list of Nodes
  Graph.fwdUnique(subject, predicate)
   -- return a single Node, exception if 0 or more than one.

Same for "bwk"

https://github.com/apache/jena/blob/master/jena-shacl/src/main/java/org/apache/jena/shacl/lib/G.java 
is a library version of this that was helpful but adding a few 
operations directly to graph


If the data is known to be good (SHACL), the application code can use 
fwd()/bwk() without worrying about testing for zero or multiple predicates.


The reason for putting the basic oprations in the Graph interface and 
not everything in a library is for potential efficiency. An impl may be 
able to do a good job of fwd() and if that is the basis of graph 
analytics efficiency matters long term, at least not to design it out.


== Assembler

The graph SPI additions is also motivated by assemblers.  Assemblers are 
currently Model/Resource based but the important usage is in Fuseki - an 
ideal goal is Fuseki works on Graph/Node.


Converting assemblers to Graph/Node does not look too burdensome and 
with a wrapper layer we can hopefully include all the old tests to check 
evolution.


== Graph - indexing

Currently, Graphs are term-indexed only or value-indexed, not both.

Graph should plain term-indexed. value-indexing, which can be calculated 
on the fly, would be a separate higher-level concept.


This is motivated by scale and having the same behaviour on all graph. 
At scale, canonicalizing the inputs is better than value-indexing.


"values" would only be in the Model API.

== Transactions

Unify the transaction approach (also changes Model) so complex 
assemblages of graphs, and other things,  are transactional.


Remove graph transactions - replace by
org.apache.jena.sparql.core.Transactional.

Then graphs as views of datasets and also combinations of Transactionals 
in single transaction (two DatasetGraph, or collection of Graphs (teh 
assmebler case)) can be done.


== Events

Make events an intercepting wrapper, not built-in to Graph itself.
Add transaction lifecycle events.

== Streams - yes and no.

A Stream is several java objects so a potential cost
for a simple operations like Graph.contains() or find() or a few things 
is not small.


Keep iterators, provide stream(s,p,o).

== Nodes

Lang tags - force to lower case.

Simplify - remove a layer of indirection. This relates to indexing.

Node_Literal - no LiteralLabels
Node_Blank - two longs or a string label, not using BlankNodeId

Investigate integrate nodes with ARQ's NodeValue.

== IRIs

jena-iri is general, powerful and hard to maintain.
Jena does not use all of it.
Jena needs a simpler, direct parser/checker.

https://github.com/afs/iri4ld

which is a parser in java with little copying. It parse URIs, and then 
has a little on scheme specific rules for http(s), file and URN.


The various open source libraries and JDK classes do not track the 
current standards very well (RFC 2396 vs RFC 3986). I have found that 
compliance is mixed due to legacy compatibility needs.