Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Ken Cox]

Dan and Jason,

On Wed, Sep 21, 2016 at 9:36 AM, Jason Stephenson  wrote:
>
> On 09/21/2016 07:32 AM, Ken Cox wrote:
> > Thanks, but the GatewayRequest class I am using is from a old fork.
>
> If your Android changes are compatible with regular JDK versions, you
> could submit them as enhancements to the OpenSRF project.

The changes I made were not in pursuit of any goal higher than getting
them to compile with the Android tool chain.  I might have made some
trivial improvements along the way, like StringBuilder instead of
StringBuffer.  I also whacked every source from the tree that would
not compile for Android.  You can have a look at your leisure at the
Github repo under the opensrf/ directory.  Don't get your hopes up!

> Likewise there are Java classes for Evergreen. Are you using those? If
> so, have you needed to fork them?

Ah, yes, I use some of those, and I have had to change them. That is
where the XML parser lives, which I changed to XMLPullParser.  Removed
javax.* references.  Whacked sources that wouldn't compile. Have a
look in the repo under the core/src/org/open_ils/ directory if you
want to compare latest vs. latest.

Ken

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Dan Wells]

Hello Ken,

As one of the original devs on the Android app, it’s been on my radar (for 
forever) to get those changes merged back upstream.  Not anything I have 
foreseeable tuits for, but I definitely support the idea.  There isn’t likely 
to be a bigger Java OpenSRF client than the app anytime soon, and maybe not 
ever, so we might as well conform to existing needs.

Sincerely,
Dan


Daniel Wells
Library Programmer/Analyst
Hekman Library, Calvin College
616.526.7133

From: Open-ils-dev [mailto:open-ils-dev-boun...@list.georgialibraries.org] On 
Behalf Of Ken Cox
Sent: Wednesday, September 21, 2016 7:33 AM
To: Evergreen Development Discussion List 

Subject: Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search 
(via OSRF)?

Thanks, but the GatewayRequest class I am using is from a old fork.  The 
Android sources had to fork a while back because of limitations in the platform 
support.  For example, OSRFRegistry had to be made Serializable and the parser 
had to be replaced with XMLPullParser.

Ken

On Tue, Sep 20, 2016 at 10:08 AM, Bill Erickson 
mailto:beric...@gmail.com>> wrote:

On Mon, Sep 19, 2016 at 8:02 PM, Ken Cox 
mailto:kens...@gmail.com>> wrote:
Bill, awesome tips, thanks!  I look forward to benchmarking them.

In some places I am using the GatewayRequest class, but in many I am using a 
new GatewayJsonObjectRequest class, which extends com.android.volley.Request.  
The volley library<https://developer.android.com/training/volley/index.html> 
launches multiple parallel requests.

The GatewayRequest class supports multiple parallel requests as well.  See the 
sendAsync() example in org/opensrf/test/TestGateway.java


Is there a version of Evergreen that supports websockets?  (I don't see any 
happening on CW/MARS).  There is at least one active Android library for 
websockets, it's part of https://github.com/koush/AndroidAsync.

Any EG instance with a functioning browser client is running websockets.  If 
they are not running the browser client, they are almost certainly not running 
websockets.  It will be some time before everyone is using it.

-b




--
-Ken

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Jason Stephenson]

On 09/21/2016 07:32 AM, Ken Cox wrote:
> Thanks, but the GatewayRequest class I am using is from a old fork.  The
> Android sources had to fork a while back because of limitations in the
> platform support.  For example, OSRFRegistry had to be made Serializable
> and the parser had to be replaced with XMLPullParser.

That is likely not a problem. There hasn't been a commit to the OpenSRF
Java code since December of 2012. Pretty much none else uses it.

If your Android changes are compatible with regular JDK versions, you
could submit them as enhancements to the OpenSRF project.

Likewise there are Java classes for Evergreen. Are you using those? If
so, have you needed to fork them?


> 
> Ken
> 
> On Tue, Sep 20, 2016 at 10:08 AM, Bill Erickson  > wrote:
> 
> 
> On Mon, Sep 19, 2016 at 8:02 PM, Ken Cox  > wrote:
> 
> Bill, awesome tips, thanks!  I look forward to benchmarking them.
> 
> In some places I am using the GatewayRequest class, but in many
> I am using a new GatewayJsonObjectRequest class, which
> extends com.android.volley.Request.  The volley library
> 
> launches multiple parallel requests.  
> 
> 
> The GatewayRequest class supports multiple parallel requests as
> well.  See the sendAsync() example in org/opensrf/test/TestGateway.java
>  
> 
> 
> Is there a version of Evergreen that supports websockets?  (I
> don't see any happening on CW/MARS).  There is at least one
> active Android library for websockets, it's part
> of https://github.com/koush/AndroidAsync
> .
> 
> 
> Any EG instance with a functioning browser client is running
> websockets.  If they are not running the browser client, they are
> almost certainly not running websockets.  It will be some time
> before everyone is using it.
> 
> -b
> 
> 
> 
> 
> -- 
> -Ken

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Ken Cox]

Thanks, but the GatewayRequest class I am using is from a old fork.  The
Android sources had to fork a while back because of limitations in the
platform support.  For example, OSRFRegistry had to be made Serializable
and the parser had to be replaced with XMLPullParser.

Ken

On Tue, Sep 20, 2016 at 10:08 AM, Bill Erickson  wrote:

>
> On Mon, Sep 19, 2016 at 8:02 PM, Ken Cox  wrote:
>
>> Bill, awesome tips, thanks!  I look forward to benchmarking them.
>>
>> In some places I am using the GatewayRequest class, but in many I am
>> using a new GatewayJsonObjectRequest class, which
>> extends com.android.volley.Request.  The volley library
>>  launches
>> multiple parallel requests.
>>
>
> The GatewayRequest class supports multiple parallel requests as well.  See
> the sendAsync() example in org/opensrf/test/TestGateway.java
>
>
>>
>> Is there a version of Evergreen that supports websockets?  (I don't see
>> any happening on CW/MARS).  There is at least one active Android library
>> for websockets, it's part of https://github.com/koush/AndroidAsync.
>>
>
> Any EG instance with a functioning browser client is running websockets.
> If they are not running the browser client, they are almost certainly not
> running websockets.  It will be some time before everyone is using it.
>
> -b
>
>


-- 
-Ken

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Jason Stephenson]

On 09/20/2016 10:08 AM, Bill Erickson wrote:
> 
> Any EG instance with a functioning browser client is running
> websockets.  If they are not running the browser client, they are almost
> certainly not running websockets.  It will be some time before everyone
> is using it.

For security reasons, some sites may not make websockets available to
the entire Internet. Those ports my be behind a firewall that
selectively allows access. I don't think you should rely on websockets
being available.

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Bill Erickson]

On Mon, Sep 19, 2016 at 8:02 PM, Ken Cox  wrote:

> Bill, awesome tips, thanks!  I look forward to benchmarking them.
>
> In some places I am using the GatewayRequest class, but in many I am using
> a new GatewayJsonObjectRequest class, which extends 
> com.android.volley.Request.
> The volley library
>  launches
> multiple parallel requests.
>

The GatewayRequest class supports multiple parallel requests as well.  See
the sendAsync() example in org/opensrf/test/TestGateway.java


>
> Is there a version of Evergreen that supports websockets?  (I don't see
> any happening on CW/MARS).  There is at least one active Android library
> for websockets, it's part of https://github.com/koush/AndroidAsync.
>

Any EG instance with a functioning browser client is running websockets.
If they are not running the browser client, they are almost certainly not
running websockets.  It will be some time before everyone is using it.

-b

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Ken Cox]

Bill, awesome tips, thanks!  I look forward to benchmarking them.

In some places I am using the GatewayRequest class, but in many I am using
a new GatewayJsonObjectRequest class, which
extends com.android.volley.Request.  The volley library
 launches
multiple parallel requests.

Is there a version of Evergreen that supports websockets?  (I don't see any
happening on CW/MARS).  There is at least one active Android library for
websockets, it's part of https://github.com/koush/AndroidAsync.

Regards,
Ken

On Mon, Sep 19, 2016 at 11:04 AM, Bill Erickson  wrote:

> Hi Ken,
>
> On Sat, Sep 17, 2016 at 2:44 PM, Ken Cox  wrote:
>
>> Is there a way to load the basic metadata (title and author) about
>> multiple catalog items in one round-trip?
>>
>> Currently after a search the Android app is making the following OSRF
>> calls for each item found:
>>
>>- service=open-ils.search & method=
>>*open-ils.search.biblio.record.mods_slim.retrieve* & param=*item_id*
>>
>> There is a batch version of this API:
>
> open-ils.search.biblio.record.mods_slim.batch.retrieve[.atomic] [id1,
> id2, id3, ...]
>
> Are you using the stock OSRF Java HTTP library -- the GatewayRequest
> class?  I ask because it does not support streaming API calls.  (I'm not
> seeing any stock Java libs that support streaming, no TranslatorRequest or
> WebsocketRequest classes).  This makes the batch API's somewhat less
> appealing, since you have to wait for the entire result set to return
> before you have any results to work with.  In some cases, though, it can
> still be an improvement.
>
> In the future, we need a Translator and/or Websocket client for the Java
> libs and we need to finish https://bugs.launchpad.
> net/evergreen/+bug/1251394 (metabib display fields), which support
> returning pre-compiled metadata via (presumably) the much faster PCRUD API.
>
>
>>-
>>- service=open-ils.pcrud & method=*open-ils.pcrud.retrieve.mra*
>>& param=%22auth_token%22 & param=*item_id*
>>
>> For this one, you could probably get away with a single batch API call:
>
> open-ils.pcrud.search.mra.atomic {id : [id1, id2, id3...]}
>
> The result set for this call is relatively small and should be quick to
> build on the server, so a single call/response might be faster than a
> call/response per record.
>
> See also the "mraf" class, where you can further limit by "type", plus its
> result data is much easier to parse.
>
> -b
>
>
>


-- 
-Ken

Re: [OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Bill Erickson]

Hi Ken,

On Sat, Sep 17, 2016 at 2:44 PM, Ken Cox  wrote:

> Is there a way to load the basic metadata (title and author) about
> multiple catalog items in one round-trip?
>
> Currently after a search the Android app is making the following OSRF
> calls for each item found:
>
>- service=open-ils.search & method=
>*open-ils.search.biblio.record.mods_slim.retrieve* & param=*item_id*
>
> There is a batch version of this API:

open-ils.search.biblio.record.mods_slim.batch.retrieve[.atomic] [id1, id2,
id3, ...]

Are you using the stock OSRF Java HTTP library -- the GatewayRequest
class?  I ask because it does not support streaming API calls.  (I'm not
seeing any stock Java libs that support streaming, no TranslatorRequest or
WebsocketRequest classes).  This makes the batch API's somewhat less
appealing, since you have to wait for the entire result set to return
before you have any results to work with.  In some cases, though, it can
still be an improvement.

In the future, we need a Translator and/or Websocket client for the Java
libs and we need to finish https://bugs.launchpad.net/evergreen/+bug/1251394
(metabib display fields), which support returning pre-compiled metadata via
(presumably) the much faster PCRUD API.


>-
>- service=open-ils.pcrud & method=*open-ils.pcrud.retrieve.mra*
>& param=%22auth_token%22 & param=*item_id*
>
> For this one, you could probably get away with a single batch API call:

open-ils.pcrud.search.mra.atomic {id : [id1, id2, id3...]}

The result set for this call is relatively small and should be quick to
build on the server, so a single call/response might be faster than a
call/response per record.

See also the "mraf" class, where you can further limit by "type", plus its
result data is much easier to parse.

-b

[OPEN-ILS-DEV] Fastest way to load titles after catalog search (via OSRF)?

[Ken Cox]

Is there a way to load the basic metadata (title and author) about multiple
catalog items in one round-trip?

Currently after a search the Android app is making the following OSRF calls
for each item found:

   - service=open-ils.search & method=
   *open-ils.search.biblio.record.mods_slim.retrieve* & param=*item_id*
   - service=open-ils.pcrud & method=*open-ils.pcrud.retrieve.mra* &
param=%22auth_token%22
   & param=*item_id*

Is there a method for bulk loading data for multiple items?

*Background and more details*

I am trying to improve the search experience in the Android app.
Originally, the app loaded the metadata for all items synchronously before
allowing the user to interact with the search results.  This was lousy,
because it takes on the order of 15 seconds to search CW/MARS for "harry
potter" and then make 20 calls to load the title and search format (e.g.
"ebook") for each of the first 10 results.

So I changed the behavior so that all metadata is loaded asynchronously.
Now the results list appears as soon as the search is complete, on the
order of 5 seconds.  However, the titles and search then start appearing as
the web requests finish.  This can be slightly odd on a slow network.

The reason for the *open-ils.pcrud.retrieve.mra* call is so that I can
determine whether to enable the "Online Access" action, if the item is an
online resource, or the "Place Hold" action, if the item is a physical
resource.  This is not as easy as looking at the online_loc field in the
mra response, because some items have that field set even for physical
resources (e.g. https://bark.cwmars.org/eg/opac/record/3306629).

Thanks,
Ken
--
get the app at
https://play.google.com/store/apps/details?id=net.kenstir.apps.hemlock