Re: apps vs publishers vs handlers vs importers confusion

[Erik Paulson]

Both the new sourceRoot option works for me to be able to override
gallery.html at runtime, and files are served with MIME types set so
hitting a jpg gets displayed right in the browser - thanks!

-Erik


On Wed, Aug 31, 2016 at 7:10 PM, Mathieu Lonjaret <
mathieu.lonja...@gmail.com> wrote:

> I've also started something which I think might help you with your end
> goal:
>
> https://camlistore-review.googlesource.com/8188
>
>
> On 21 August 2016 at 05:17, Erik Paulson  wrote:
> > I think I've got a pretty good handle on how data is modeled in
> camlistore,
> > with blobs, metadata blobs, permanodes, and claims building a graph
> > stitching it all together, ala git.
> >
> > I'm still a little bit confused by data is expected to flow in and out of
> > camlistore, between apps, handlers, importers, and the UI. (I do
> understand
> > that this part of camlistore is still a work in progress)
> >
> > The use case I was trying to do today was basically just webserving. I
> > wanted to take a directory, load it into camlistore, and then serve those
> > files via HTTP. I tried:
> >
> > mkdir junk
> > echo "first" >stuff/first.txt
> > echo "second" >stuff/second.txt
> >
> > camput file --permanode stuff/
> >
> > Then, in the UI, I added an claim to the stuff/ permanode to set
> camliRoot
> > to 'junkRoot'
> >
> > Then, in my server-config.json, I added:
> >
> > publish: {
> >   "/stuff/": {
> > "camliRoot": "stuffRoot",
> > "goTemplate": "gallery.html"
> >   }
> > }
> >
> > and restarted camlistored. I tried to not specify a goTemplate, but it
> > insisted on having that key present. (I tried setting "goTemplate": ""
> but
> > no dice there, either)
> >
> > I was hoping that I'd be able to get localhost:3179/stuff/first.txt with
> > this setup, but no go.
> >
> > First, in order to get anything to show up under localhost:3179/stuff/, I
> > couldn't just have the file schema blobs as camliMembers of my 'stuff'
> > permanode the way camput created it- I had to create permanodes for
> > first.txt and second.txt and add those refs as camliMembers to the stuff
> > permanode.
> >
> > Second, it renders everything some entries in the gallery template. I
> guess
> > that's not surprising since I told it to use the gallery. Camlistore
> doesn't
> > have a way of knowing that I didn't actually want to set that config
> > entry...
> >
> > Finally, it does eventually give me a link to my file, but it's buried
> in a
> > URL like:
> > http://localhost:3179/stuff//-/hd27fc6a636/hb174f6a39c/=f/first.txt
> >
> > (I saw Mathieu's warning about the publisher being in progress in the
> latest
> > code, so this is all against master checked out here:
> >
> >> commit ce8b329d1d62a4ff6ab87812f657914ae7625f6d
> >> Author: mpl 
> >> Date:   Thu Jun 23 16:07:57 2016 +0200
> >> website: update CLA instruction
> >>
> >> Change-Id: I399ad44a0ebdeff05efd1dabb3137299b01f112b
> >
> >
> > Is there a way to make the 'publisher' app just serve up files directly
> with
> > a more regular httpd dirlisting sort of view? Is there a legacy handler
> that
> > I could use? (I'm not super-keen on mounting this in FUSE and serving
> that
> > way, but if that's the answer, so be it)
> >
> > I understand that many of the handlers and importers will be moving out
> of
> > bin/camlistored and into separate app processes, and camlistored will
> become
> > more like systemd/init/inetd in that it watches child processes and
> directs
> > traffic to and from those children. Do you have targets for other apps
> that
> > might be written?
> >
> > Is it expected that users will use 3rd party apps or write their own
> apps as
> > regular operation, or is it more that anyone who writes an app will do so
> > with the eventual intention of getting it included in the regular
> camlistore
> > repo and releases? Is there a reason to prefer running an app under
> > camlistored vs spawning my own process next to camlistored?
> >
> > My eventual use case is that I want to use camlistore as a host for a
> blog
> > generated by a static site generator like pelican or jeykll - besides
> > storing and serving the static HTML created by pelican, I want to write a
> > plugin to pelican to insert the permanode id for the post into the HTML
> > metadata about the page, so even if URL changes some day, the permanode
> > gives a really great rel=canonical opportunity in a decentralized world.
> For
> > this, I was hoping to use no code outside of camlistore for the actual
> > serving - camlistore would take care of the naming, and if a big file is
> > chunked by the rolling checksum into multiple blobs, the
> > publisher/handler/whatever takes care of reassembling the parts.
> >
> > Are "publishers" a first-class concept, or is it just that the only real
> app
> > going right now is the publisher, and having multiple "publishers" is
> really
> > just running the same app multiple times, pointed at different camliRoots
> 

Re: apps vs publishers vs handlers vs importers confusion

[Mathieu Lonjaret]

I've also started something which I think might help you with your end goal:

https://camlistore-review.googlesource.com/8188


On 21 August 2016 at 05:17, Erik Paulson  wrote:
> I think I've got a pretty good handle on how data is modeled in camlistore,
> with blobs, metadata blobs, permanodes, and claims building a graph
> stitching it all together, ala git.
>
> I'm still a little bit confused by data is expected to flow in and out of
> camlistore, between apps, handlers, importers, and the UI. (I do understand
> that this part of camlistore is still a work in progress)
>
> The use case I was trying to do today was basically just webserving. I
> wanted to take a directory, load it into camlistore, and then serve those
> files via HTTP. I tried:
>
> mkdir junk
> echo "first" >stuff/first.txt
> echo "second" >stuff/second.txt
>
> camput file --permanode stuff/
>
> Then, in the UI, I added an claim to the stuff/ permanode to set camliRoot
> to 'junkRoot'
>
> Then, in my server-config.json, I added:
>
> publish: {
>   "/stuff/": {
> "camliRoot": "stuffRoot",
> "goTemplate": "gallery.html"
>   }
> }
>
> and restarted camlistored. I tried to not specify a goTemplate, but it
> insisted on having that key present. (I tried setting "goTemplate": "" but
> no dice there, either)
>
> I was hoping that I'd be able to get localhost:3179/stuff/first.txt with
> this setup, but no go.
>
> First, in order to get anything to show up under localhost:3179/stuff/, I
> couldn't just have the file schema blobs as camliMembers of my 'stuff'
> permanode the way camput created it- I had to create permanodes for
> first.txt and second.txt and add those refs as camliMembers to the stuff
> permanode.
>
> Second, it renders everything some entries in the gallery template. I guess
> that's not surprising since I told it to use the gallery. Camlistore doesn't
> have a way of knowing that I didn't actually want to set that config
> entry...
>
> Finally, it does eventually give me a link to my file, but it's buried in a
> URL like:
> http://localhost:3179/stuff//-/hd27fc6a636/hb174f6a39c/=f/first.txt
>
> (I saw Mathieu's warning about the publisher being in progress in the latest
> code, so this is all against master checked out here:
>
>> commit ce8b329d1d62a4ff6ab87812f657914ae7625f6d
>> Author: mpl 
>> Date:   Thu Jun 23 16:07:57 2016 +0200
>> website: update CLA instruction
>>
>> Change-Id: I399ad44a0ebdeff05efd1dabb3137299b01f112b
>
>
> Is there a way to make the 'publisher' app just serve up files directly with
> a more regular httpd dirlisting sort of view? Is there a legacy handler that
> I could use? (I'm not super-keen on mounting this in FUSE and serving that
> way, but if that's the answer, so be it)
>
> I understand that many of the handlers and importers will be moving out of
> bin/camlistored and into separate app processes, and camlistored will become
> more like systemd/init/inetd in that it watches child processes and directs
> traffic to and from those children. Do you have targets for other apps that
> might be written?
>
> Is it expected that users will use 3rd party apps or write their own apps as
> regular operation, or is it more that anyone who writes an app will do so
> with the eventual intention of getting it included in the regular camlistore
> repo and releases? Is there a reason to prefer running an app under
> camlistored vs spawning my own process next to camlistored?
>
> My eventual use case is that I want to use camlistore as a host for a blog
> generated by a static site generator like pelican or jeykll - besides
> storing and serving the static HTML created by pelican, I want to write a
> plugin to pelican to insert the permanode id for the post into the HTML
> metadata about the page, so even if URL changes some day, the permanode
> gives a really great rel=canonical opportunity in a decentralized world. For
> this, I was hoping to use no code outside of camlistore for the actual
> serving - camlistore would take care of the naming, and if a big file is
> chunked by the rolling checksum into multiple blobs, the
> publisher/handler/whatever takes care of reassembling the parts.
>
> Are "publishers" a first-class concept, or is it just that the only real app
> going right now is the publisher, and having multiple "publishers" is really
> just running the same app multiple times, pointed at different camliRoots
> and using different HTML templates? If my only option is to write my own
> code to serve dirlisting files, is it more correct for me to say I'm writing
> my own "app" or I am writing my own "publisher"?
>
> I apologize if I'm not entirely clear in my email, and for asking so many
> questions.
>
> Thanks,
>
> -Erik
>
> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to 

Re: apps vs publishers vs handlers vs importers confusion

[Mathieu Lonjaret]

On 22 August 2016 at 18:36, Mathieu Lonjaret  wrote:
> On 21 August 2016 at 18:32, Erik Paulson  wrote:
>> On Sun, Aug 21, 2016 at 9:21 AM, Mathieu Lonjaret
>>  wrote:
>>>
>>> 2) the template configuration (gallery.html) is exactly what should
>>> allow you to achieve what you want. The default (and only existing
>>> one) is meant for images, because that was our first need. The point
>>> of making it user-configurable was indeed to encourage contributions
>>> for other layouts, such as a blog. But since nobody's taken any
>>> interest in it, I haven't been extra careful in keeping it functional
>>> while improving of other aspects of the app handler/publisher. So I'm
>>> not even sure it's still usable as a "plug-in" solution.
>>
>>
>> Is there a way to add or modify templates at runtime without needing to
>> recompile? gallery.html is only in src, and it's literally embedded in the
>> publisher binary:
>>
>> epaulson:~/development/camlistore/bin $ strings publisher  |grep
>> camliViewIsOwner
>>
>> var camliViewIsOwner = {{$header.ViewerIsOwner}};
>
> yeah, everything is embedded in the binary by default.
> According to the publisher code, I see that one was supposed to have
> the option to specify the SourceRoot parameter in the publisher
> high-level config. Then whatever goTemplate you specify should be used
> at runtime if found at $SourceRoot/app/publisher/$goTemplate.
> Unfortunately, there's a bug in the low level config generator; it
> ignores this sourceRoot parameter (but this is an easy fix, I could it
> soon).

Done. I only gave it a quick test, but that change should do it:
https://camlistore-review.googlesource.com/8046
Please feel free to test it and +1 if that works for you.

> So for now, if you really want to do that, you'd have to use a
> low-level config file for camlistored instead, and specify this
> SourceRoot for the publisher section.
>
>> in the server-config.json, for the "goTemplate" key under publisher, what
>> directory path is that relative to? I tried dropping a modified gallery.html
>> in the same bin directory as camlistored and publisher, but I didn't see any
>> of my changes to the HTML reflected.
>
> If the above SourceRoot is not specified, all *.html found in the
> source directory of the publisher
> ($GOPATH/src/camlistore.org/app/publisher/) is embedded in the binary.
>
>> Thanks!
>>
>> -Erik
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Camlistore" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to camlistore+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to camlistore+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: apps vs publishers vs handlers vs importers confusion

[Mathieu Lonjaret]

On 21 August 2016 at 18:32, Erik Paulson  wrote:
> On Sun, Aug 21, 2016 at 9:21 AM, Mathieu Lonjaret
>  wrote:
>>
>> 2) the template configuration (gallery.html) is exactly what should
>> allow you to achieve what you want. The default (and only existing
>> one) is meant for images, because that was our first need. The point
>> of making it user-configurable was indeed to encourage contributions
>> for other layouts, such as a blog. But since nobody's taken any
>> interest in it, I haven't been extra careful in keeping it functional
>> while improving of other aspects of the app handler/publisher. So I'm
>> not even sure it's still usable as a "plug-in" solution.
>
>
> Is there a way to add or modify templates at runtime without needing to
> recompile? gallery.html is only in src, and it's literally embedded in the
> publisher binary:
>
> epaulson:~/development/camlistore/bin $ strings publisher  |grep
> camliViewIsOwner
>
> var camliViewIsOwner = {{$header.ViewerIsOwner}};

yeah, everything is embedded in the binary by default.
According to the publisher code, I see that one was supposed to have
the option to specify the SourceRoot parameter in the publisher
high-level config. Then whatever goTemplate you specify should be used
at runtime if found at $SourceRoot/app/publisher/$goTemplate.
Unfortunately, there's a bug in the low level config generator; it
ignores this sourceRoot parameter (but this is an easy fix, I could it
soon).
So for now, if you really want to do that, you'd have to use a
low-level config file for camlistored instead, and specify this
SourceRoot for the publisher section.

> in the server-config.json, for the "goTemplate" key under publisher, what
> directory path is that relative to? I tried dropping a modified gallery.html
> in the same bin directory as camlistored and publisher, but I didn't see any
> of my changes to the HTML reflected.

If the above SourceRoot is not specified, all *.html found in the
source directory of the publisher
($GOPATH/src/camlistore.org/app/publisher/) is embedded in the binary.

> Thanks!
>
> -Erik
>
> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camlistore+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to camlistore+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: apps vs publishers vs handlers vs importers confusion

[Erik Paulson]

On Sun, Aug 21, 2016 at 9:21 AM, Mathieu Lonjaret <
mathieu.lonja...@gmail.com> wrote:

> 2) the template configuration (gallery.html) is exactly what should
> allow you to achieve what you want. The default (and only existing
> one) is meant for images, because that was our first need. The point
> of making it user-configurable was indeed to encourage contributions
> for other layouts, such as a blog. But since nobody's taken any
> interest in it, I haven't been extra careful in keeping it functional
> while improving of other aspects of the app handler/publisher. So I'm
> not even sure it's still usable as a "plug-in" solution.
>

Is there a way to add or modify templates at runtime without needing to
recompile? gallery.html is only in src, and it's literally embedded in the
publisher binary:

epaulson:~/development/camlistore/bin $ strings publisher  |grep
camliViewIsOwner

var camliViewIsOwner = {{$header.ViewerIsOwner}};


in the server-config.json, for the "goTemplate" key under publisher, what
directory path is that relative to? I tried dropping a modified
gallery.html in the same bin directory as camlistored and publisher, but I
didn't see any of my changes to the HTML reflected.

Thanks!

-Erik

-- 
You received this message because you are subscribed to the Google Groups 
"Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to camlistore+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: apps vs publishers vs handlers vs importers confusion

[Mathieu Lonjaret]

On 21 August 2016 at 05:17, Erik Paulson  wrote:
> I think I've got a pretty good handle on how data is modeled in camlistore,
> with blobs, metadata blobs, permanodes, and claims building a graph
> stitching it all together, ala git.
>
> I'm still a little bit confused by data is expected to flow in and out of
> camlistore, between apps, handlers, importers, and the UI. (I do understand
> that this part of camlistore is still a work in progress)
>
> The use case I was trying to do today was basically just webserving. I
> wanted to take a directory, load it into camlistore, and then serve those
> files via HTTP. I tried:
>
> mkdir junk
> echo "first" >stuff/first.txt
> echo "second" >stuff/second.txt
>
> camput file --permanode stuff/
>
> Then, in the UI, I added an claim to the stuff/ permanode to set camliRoot
> to 'junkRoot'
>
> Then, in my server-config.json, I added:
>
> publish: {
>   "/stuff/": {
> "camliRoot": "stuffRoot",
> "goTemplate": "gallery.html"
>   }
> }
>
> and restarted camlistored. I tried to not specify a goTemplate, but it
> insisted on having that key present. (I tried setting "goTemplate": "" but
> no dice there, either)
>
> I was hoping that I'd be able to get localhost:3179/stuff/first.txt with
> this setup, but no go.
>
> First, in order to get anything to show up under localhost:3179/stuff/, I
> couldn't just have the file schema blobs as camliMembers of my 'stuff'
> permanode the way camput created it- I had to create permanodes for
> first.txt and second.txt and add those refs as camliMembers to the stuff
> permanode.
>
> Second, it renders everything some entries in the gallery template. I guess
> that's not surprising since I told it to use the gallery. Camlistore doesn't
> have a way of knowing that I didn't actually want to set that config
> entry...
>
> Finally, it does eventually give me a link to my file, but it's buried in a
> URL like:
> http://localhost:3179/stuff//-/hd27fc6a636/hb174f6a39c/=f/first.txt
>
> (I saw Mathieu's warning about the publisher being in progress in the latest
> code, so this is all against master checked out here:
>
>> commit ce8b329d1d62a4ff6ab87812f657914ae7625f6d
>> Author: mpl 
>> Date:   Thu Jun 23 16:07:57 2016 +0200
>> website: update CLA instruction
>>
>> Change-Id: I399ad44a0ebdeff05efd1dabb3137299b01f112b
>
>
> Is there a way to make the 'publisher' app just serve up files directly with
> a more regular httpd dirlisting sort of view? Is there a legacy handler that
> I could use? (I'm not super-keen on mounting this in FUSE and serving that
> way, but if that's the answer, so be it)

Out of the box, I don't think there's exactly that. That said,

1) If you set the permanode for first.txt directly as a camliPath of
your camliRoot, there'll be one less "hop" in the URL you mention
above, if the length of it is what bothers you.

2) the template configuration (gallery.html) is exactly what should
allow you to achieve what you want. The default (and only existing
one) is meant for images, because that was our first need. The point
of making it user-configurable was indeed to encourage contributions
for other layouts, such as a blog. But since nobody's taken any
interest in it, I haven't been extra careful in keeping it functional
while improving of other aspects of the app handler/publisher. So I'm
not even sure it's still usable as a "plug-in" solution.

> I understand that many of the handlers and importers will be moving out of
> bin/camlistored and into separate app processes, and camlistored will become
> more like systemd/init/inetd in that it watches child processes and directs
> traffic to and from those children. Do you have targets for other apps that
> might be written?

Brad's scanning cabinet port has been mostly done for a while now.
It's just a matter of getting the CL merged someday.

> Is it expected that users will use 3rd party apps or write their own apps as
> regular operation, or is it more that anyone who writes an app will do so
> with the eventual intention of getting it included in the regular camlistore
> repo and releases?

The former. Even the web UI handler should eventually be decoupled
from camlistored.

> Is there a reason to prefer running an app under
> camlistored vs spawning my own process next to camlistored?

The usual advantages of modularity.

> My eventual use case is that I want to use camlistore as a host for a blog
> generated by a static site generator like pelican or jeykll - besides
> storing and serving the static HTML created by pelican, I want to write a
> plugin to pelican to insert the permanode id for the post into the HTML
> metadata about the page, so even if URL changes some day, the permanode
> gives a really great rel=canonical opportunity in a decentralized world. For
> this, I was hoping to use no code outside of camlistore for the actual
> serving - camlistore would take care of the naming, and if a big file is