I think at the end it all boils down to project goals. If I understand it correctly, Perkeep does not have any reason to design toward index and frontend scalability, which of course makes sense.
For example there is no reason for Perkeep to support scalable UI service (multiple ui servers running in parallel and serving requests and keeping the index (es?) in sync in near real time. The blobservers of course doesn't seem to have any problem with scalability. But index seems to be the bottleneck as it currently stands (as I understand it). Looks to me, moving the index to a distributed key-value-store like dynamodb, seems to solve (some of?) the problems, but it might be a bit in conflict with the main project goals as a distributed storage for the index maybe too limiting for the primary use case of Perkeep (dealing with latency and possibility of network/service failures and limitations of querying the datastore as well as potential for conflicts). Am I making sense? Do you agree with my interpretation? On Monday, May 28, 2018 at 6:20:28 AM UTC-4, alimoeeny wrote: > > Thanks Tamás and Mathieu, I appreciate it. > Tamás, that is a neat example, thanks for sharing. > Mathieu, I need to think about my overall architecture a bit more then. > Like to run a scalable service, if I want to make sure I am not creating > bottle necks, I need to be able to just spin up instances to serve http > requests, if then they need to wait a long time fetching blobs and creating > indexes ... > If I am not mistaken there is support for dynamodb as the storage for the > index > I need to read more, > Thanks a lot, > Ali > > > > > > Ali > > > On Sat, May 26, 2018 at 2:45 PM Mathieu Lonjaret wrote: > >> On 26 May 2018 at 14:15, alimoeeny wrote: >> > Hey Perkeep people, >> > >> > I want to use Perkeep's, blob storage + indexing + search inside a >> larger >> > service. >> > I want to have my own auth and access control and CDN in front. >> > >> > I of course want to keep up to date with all the progress you are and >> will >> > be making, >> > >> > Does it make more sense to use perkeep as libraries and build my >> services on >> > top, or would it make more sense to run per keep as is behind my >> service and >> > have my service call perkeep api and act a some sort of proxy ? >> > >> > My question is more about your longer term plans, do you expect people >> to >> > use Perkeep as library(ies) and will keep your "internal" api rather >> stable? >> > or you'd rather be free to make changes to your internal api frequently >> and >> > break things but keep your "external" api more stable? >> >> Both approaches are viable depending on what parts of Perkeep you >> need. If you want the whole of it (as seems to be the case), then >> you'd better off using it as a service (second approach). But there's >> nothing preventing you from building something on top of just e.g. the >> blob server, in which case it's totally fine to just import >> pkg/blobserver/ in your app and rely on it. We may break things there, >> but the interface in general should not change too much I think >> >> The long term plan is for more components (which are now integrated) >> to become third-party clients of whatever perkeepd trims down to. The >> web UI is an example of such a component. As it is, and as Tamás >> mentioned, we already have such third-party applications (the >> publisher and the scanning-cabinet), so maybe you want to have a look >> at them, to see how they interact with Perkeep at the moment. >> >> Does that answer your question? >> >> > Am I making sense? >> > >> > Ali >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Perkeep" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to [email protected]. >> > For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Perkeep" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
