Do we really need indexing? My main problem with the current index is that is it is in-memory, instead of using the provided DBMS/KV-store.
alimoeeny <[email protected]> ezt írta (időpont: 2018. máj. 28., H, 16:36): > I think at the end it all boils down to project goals. > If I understand it correctly, Perkeep does not have any reason to design > toward index and frontend scalability, which of course makes sense. > > For example there is no reason for Perkeep to support scalable UI service > (multiple ui servers running in parallel and serving requests and keeping > the index (es?) in sync in near real time. > The blobservers of course doesn't seem to have any problem with > scalability. > But index seems to be the bottleneck as it currently stands (as I > understand it). > > Looks to me, moving the index to a distributed key-value-store like > dynamodb, seems to solve (some of?) the problems, but it might be a bit in > conflict with the main project goals as a distributed storage for the index > maybe too limiting for the primary use case of Perkeep (dealing with > latency and possibility of network/service failures and limitations of > querying the datastore as well as potential for conflicts). > > Am I making sense? Do you agree with my interpretation? > > > > > On Monday, May 28, 2018 at 6:20:28 AM UTC-4, alimoeeny wrote: >> >> Thanks Tamás and Mathieu, I appreciate it. >> Tamás, that is a neat example, thanks for sharing. >> Mathieu, I need to think about my overall architecture a bit more then. >> Like to run a scalable service, if I want to make sure I am not creating >> bottle necks, I need to be able to just spin up instances to serve http >> requests, if then they need to wait a long time fetching blobs and creating >> indexes ... >> If I am not mistaken there is support for dynamodb as the storage for the >> index >> I need to read more, >> Thanks a lot, >> Ali >> >> >> >> >> >> Ali >> >> >> On Sat, May 26, 2018 at 2:45 PM Mathieu Lonjaret wrote: >> >>> On 26 May 2018 at 14:15, alimoeeny wrote: >>> > Hey Perkeep people, >>> > >>> > I want to use Perkeep's, blob storage + indexing + search inside a >>> larger >>> > service. >>> > I want to have my own auth and access control and CDN in front. >>> > >>> > I of course want to keep up to date with all the progress you are and >>> will >>> > be making, >>> > >>> > Does it make more sense to use perkeep as libraries and build my >>> services on >>> > top, or would it make more sense to run per keep as is behind my >>> service and >>> > have my service call perkeep api and act a some sort of proxy ? >>> > >>> > My question is more about your longer term plans, do you expect people >>> to >>> > use Perkeep as library(ies) and will keep your "internal" api rather >>> stable? >>> > or you'd rather be free to make changes to your internal api >>> frequently and >>> > break things but keep your "external" api more stable? >>> >>> Both approaches are viable depending on what parts of Perkeep you >>> need. If you want the whole of it (as seems to be the case), then >>> you'd better off using it as a service (second approach). But there's >>> nothing preventing you from building something on top of just e.g. the >>> blob server, in which case it's totally fine to just import >>> pkg/blobserver/ in your app and rely on it. We may break things there, >>> but the interface in general should not change too much I think >>> >>> The long term plan is for more components (which are now integrated) >>> to become third-party clients of whatever perkeepd trims down to. The >>> web UI is an example of such a component. As it is, and as Tamás >>> mentioned, we already have such third-party applications (the >>> publisher and the scanning-cabinet), so maybe you want to have a look >>> at them, to see how they interact with Perkeep at the moment. >>> >>> Does that answer your question? >>> >>> > Am I making sense? >>> > >>> > Ali >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> > "Perkeep" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an >>> > email to [email protected]. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Perkeep" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Perkeep" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Perkeep" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
