[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247325#comment-15247325 ] ASF subversion and git services commented on SOLR-8349: --- Commit

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247323#comment-15247323 ] ASF subversion and git services commented on SOLR-8349: --- Commit

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245450#comment-15245450 ] ASF subversion and git services commented on SOLR-8349: --- Commit

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245447#comment-15245447 ] ASF subversion and git services commented on SOLR-8349: --- Commit

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245428#comment-15245428 ] ASF subversion and git services commented on SOLR-8349: --- Commit

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244256#comment-15244256 ] Gus Heck commented on SOLR-8349: also deleted the misnamed patch since it seems to sort to the top. >

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242847#comment-15242847 ] Noble Paul commented on SOLR-8349: -- The patch is not up to date with the master. Can you please update it

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242841#comment-15242841 ] Gus Heck commented on SOLR-8349: Actually looking closer it seems to be happening when I try to add the

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242833#comment-15242833 ] Gus Heck commented on SOLR-8349: re-applied patch to trunk (no conflicts) and ran tests, and hit this... I

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242718#comment-15242718 ] Noble Paul commented on SOLR-8349: -- If you thing it is good enough, I can review and go ahead > Allow

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242710#comment-15242710 ] Gus Heck commented on SOLR-8349: Would be good to sort out what needs to be done for SOLR-8772 in time for

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175661#comment-15175661 ] Gus Heck commented on SOLR-8349: Right I got around that by creating .system first and uploading before I

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174902#comment-15174902 ] Noble Paul commented on SOLR-8349: -- Thanks. the tests are fine. But for one thing, The blob store is not

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172624#comment-15172624 ] Gus Heck commented on SOLR-8349: Ah it seems a brand new class for cloud unit testing has magically

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172386#comment-15172386 ] Gus Heck commented on SOLR-8349: Hmm, Ive been wondering how I might write a good test case for it. Are

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172260#comment-15172260 ] Noble Paul commented on SOLR-8349: -- The patch looks fine. If we have a testcase for this w could commit

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171924#comment-15171924 ] Gus Heck commented on SOLR-8349: Oooops! sorry, it seems I uploaded the wrong file (and missed your

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15171699#comment-15171699 ] Noble Paul commented on SOLR-8349: -- [~gus_heck] can u post the updated patch? > Allow sharing of large in

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169453#comment-15169453 ] Noble Paul commented on SOLR-8349: -- It's fine . But the patch doesn't have this extra method for decoder

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169187#comment-15169187 ] Gus Heck commented on SOLR-8349: Perhaps you were looking for a second hash map? We don't need one if we

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169057#comment-15169057 ] Noble Paul commented on SOLR-8349: -- really? I still think it doesn't take care of multiple decoders. >

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166522#comment-15166522 ] Noble Paul commented on SOLR-8349: -- One problem I see with the patch is, with decoding the object in two

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163464#comment-15163464 ] Gus Heck commented on SOLR-8349: I had updated on the 22nd. Should have pulled before I created the patch,

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163001#comment-15163001 ] Noble Paul commented on SOLR-8349: -- Are you working on the latest trunk? I have already committed my other

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159601#comment-15159601 ] Gus Heck commented on SOLR-8349: Sure I could, but other folks who try to use this after us will likely

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158135#comment-15158135 ] Noble Paul commented on SOLR-8349: -- bq.but I think there might be some holes WRT decoders decoding the

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158113#comment-15158113 ] Gus Heck commented on SOLR-8349: Looked at the patch some today, Mostly it looks easy enough to adapt to my

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156521#comment-15156521 ] David Smiley commented on SOLR-8349: This is a nice suggestion [~noble.paul]; it looks remarkably

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156323#comment-15156323 ] Noble Paul commented on SOLR-8349: -- bq.So are you proposing changing JarRepository, or adding a similar

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156319#comment-15156319 ] Gus Heck commented on SOLR-8349: hmm looks like I didn't realize my ide had navigated me to another class

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156315#comment-15156315 ] Gus Heck commented on SOLR-8349: I do see how what you suggest could be made to work if a decoder was

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156295#comment-15156295 ] Noble Paul commented on SOLR-8349: -- bq.That may come in handy if/when a refresh mechanism is desired. The

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156293#comment-15156293 ] Noble Paul commented on SOLR-8349: -- OK , I got the point. The framework can be easily extended to making

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156290#comment-15156290 ] Gus Heck commented on SOLR-8349: That sounds like a very cool way to store the source data, but this issue

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156236#comment-15156236 ] Noble Paul commented on SOLR-8349: -- [~gus_heck] There is another easy existing solution for this problem

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156203#comment-15156203 ] Gus Heck commented on SOLR-8349: *WRT #3/derministic behavior*: Here's the use case: # server is started,

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150714#comment-15150714 ] Gus Heck commented on SOLR-8349: Actually in my particular case since all cores need it anyway I'm just

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149913#comment-15149913 ] David Smiley commented on SOLR-8349: bq. One of the reasons I had the goal of not blocking is the case

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149420#comment-15149420 ] David Smiley commented on SOLR-8349: I see your point on CacheLoader needing to define its loading

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149286#comment-15149286 ] Gus Heck commented on SOLR-8349: I did see .build(CacheLoader). :) My writeup was an explanation of why

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148911#comment-15148911 ] David Smiley commented on SOLR-8349: bq. Change our goals, and use guava (allow cores loading the same

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148797#comment-15148797 ] Gus Heck commented on SOLR-8349: Executive summary of the last overly long comment: # Guava's cache is

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147835#comment-15147835 ] Gus Heck commented on SOLR-8349: Guava sounded like a great idea. I love guava caches and have used them

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142878#comment-15142878 ] David Smiley commented on SOLR-8349: First; I propose the container level cache be committed by itself

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142799#comment-15142799 ] Gus Heck commented on SOLR-8349: I am not sure I understand what you mean by load the analysis component

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143872#comment-15143872 ] Gus Heck commented on SOLR-8349: Certainly we can punt enabling this for lucene level things to start with,

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15144017#comment-15144017 ] David Smiley commented on SOLR-8349: 1. the classLoader separation is simply because there are libs and

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140881#comment-15140881 ] Gus Heck commented on SOLR-8349: Can this make it into 6.0? I'm slightly surprised that a feature that can

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140931#comment-15140931 ] David Smiley commented on SOLR-8349: I take back my claim that I'll commit it if only it used Guava. I

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140924#comment-15140924 ] David Smiley commented on SOLR-8349: Instead of a hand-rolled cache with separate per-key locks I

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141171#comment-15141171 ] Gus Heck commented on SOLR-8349: Thanks for the feedback Dave, I generally like guava caches, so this

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142190#comment-15142190 ] David Smiley commented on SOLR-8349: I put a little more thought into this issue now. The cache at the

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15071258#comment-15071258 ] Gus Heck commented on SOLR-8349: Has anyone had time to look at this? > Allow sharing of large in memory

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045423#comment-15045423 ] Gus Heck commented on SOLR-8349: I'm sure folks have been busy getting stuff into 5.4, but with the RC

[jira] [Commented] (SOLR-8349) Allow sharing of large in memory data structures across cores

[ https://issues.apache.org/jira/browse/SOLR-8349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032007#comment-15032007 ] Gus Heck commented on SOLR-8349: Further background for this feature... a precursor version of this patch