Thanks Pat for the follow-up. Yes, was happening on each and every request. As I stated, it only happened through Passenger also, not through console.
I think I'll switch to real-time indices, if you think of anything else, let me know. It's got me stumped. On Sunday, March 11, 2018 at 12:12:56 AM UTC-5, Pat Allan wrote: > > I know it’s hard to test now that you’ve disabled deltas, but I wondering… > did it happen on every single request? > > It’s just that, previously - as already noted - failed indexing didn’t > raise exceptions. If two requests came at the same time that both made > alterations to the same model, one of the delta processing attempts would > work, the other wouldn’t - but the end result is still both alterations > would be in Sphinx (the successful processing captured both changes). > > Now, in the same scenario, one would work, and the other would fail and > raise an exception. The data would still be persisted in Sphinx (and given > deltas are invoked in after_commit, in the database as well), but the 500 > error resulting from the failure is a terrible user experience. > > If this is the situation, then the best solution is to either shift delta > processing into a background worker (via ts-sidekiq-delta, > ts-delayed-delta, or ts-resque-delta), or to use real-time indices. > > Of course, if this isn’t the case - if a single request raises the > exception, and there’s no other concurrent requests altering the same > model, then the cause will be something else (and I’m really not sure what > it could be!). Perhaps real-time indices are worth investigating either > way? They’ve been around in TS for a good while now and I use them for any > of my new projects that need Sphinx, as do significantly sized Flying > Sphinx customers. > > On 11 Mar 2018, at 2:39 pm, mrmanishs <[email protected] <javascript:>> > wrote: > > Thanks for getting back Pat. Yes deltas were working great on the old > server. > > The permission is 644 and owner is root. > > On Saturday, March 10, 2018 at 7:45:48 PM UTC-5, Pat Allan wrote: >> >> So, one thing that’s probably changed in that upgrade is that the newer >> versions of Riddle are now a little fussier about running indexing >> commands, and raise exceptions depending on the status code of the >> `indexer` command. So, it could be possible this issue was always happening >> but the errors weren’t apparent. I feel that’s unlikely though - you’d >> probably have noticed if delta results weren’t appearing? >> >> Also worth considering is the underlying error: >> >> FATAL: failed to lock /sites/cap/ >> classifieds.acceleweb.com/db/sphinx/production/address_delta.spl: Resource >> temporarily unavailable, will not index >> >> >> I presume the path there is correct? What are the permissions on that >> file? Is the owner root? >> >> — >> Pat >> >> On 10 Mar 2018, at 1:12 am, mrmanishs <[email protected]> wrote: >> >> Thanks. I actually migrated to a new server and installed the latest >> version of Sphinx/TS. >> >> New server: Sphinx 3.0.2, TS 3.4.2 on Rails 5.1.5 >> Old server: Sphinx 2.2.6, TS 3.2.0 on Rails 4.1.6 >> >> Could I have missed something in the upgrade that these symptoms may show? >> >> >> I have disabled deltas for now to get it going, but it's not ideal. >> >> >> On Friday, March 9, 2018 at 8:58:02 AM UTC-5, Pat Allan wrote: >>> >>> Hmm. Have you tried running a rebuild? And has this problem started only >>> recently? >>> >>> As a stop-gap solution, you could disable deltas and run the `ts:index` >>> task more regularly, but granted, that’s not ideal. >>> >>> On 9 Mar 2018, at 2:36 pm, mrmanishs <[email protected]> wrote: >>> >>> >>> I have nginx in front of passenger, so all the passenger log entries go >>> into the nginx one. The only item in nginx is the url calls come back with >>> a 500. >>> >>> Thanks for the note. >>> >>> Any other ideas? Also, up for doing some real-time debugging if it would >>> help. I am not really sure what's going on and it's impacting functionality >>> on site :( >>> >>> >>> >>> On Thursday, March 8, 2018 at 5:14:47 PM UTC-5, Pat Allan wrote: >>>> >>>> Hmm. Is there anything related to this issue in your passenger logs? >>>> And/or if you’re using nginx/apache/etc in front of that, their logs? >>>> >>>> Unrelated, but: indexed_models is no longer needed/used by TS v3+. >>>> >>>> — >>>> Pat >>>> >>>> On 9 Mar 2018, at 1:04 am, mrmanishs <[email protected]> wrote: >>>> >>>> Thanks Pat. >>>> >>>> - searchd and nginx are both being run as root. Everything's root >>>> actually... >>>> - I have searchd in /usr/local/bin and confirmed the production >>>> bin_path is /usr/local/bin . I have this in my thinking_sphinx.yml file >>>> for >>>> prod: >>>> >>>> production: >>>> mysql41: 9401 >>>> configuration_file: "<path_to_config>" >>>> log: "<path_to_log>" >>>> query_log: "<path_to_query_log>" >>>> pid_file: "<path_to_pid_file>" >>>> min_infix_len: 3 >>>> bin_path: '/usr/local/bin' >>>> indexed_models: >>>> - Address >>>> quiet_deltas: true >>>> >>>> Any other thoughts? >>>> >>>> On Wednesday, March 7, 2018 at 11:45:43 PM UTC-5, Pat Allan wrote: >>>>> >>>>> Hi Manish, >>>>> >>>>> I’m at a conference today, so please excuse the brevity, but it sounds >>>>> like this might be a permissions issue. See this note in the >>>>> documentation: >>>>> https://freelancing-gods.com/thinking-sphinx/common_issues.html#deltas >>>>> (Both the deltas section and possibly the following section for >>>>> passenger) >>>>> >>>>> Certainly, real-time indices shouldn’t hit these issues, but you >>>>> should certainly be able to get your current setup working. >>>>> >>>>> Hope this helps! >>>>> >>>>> — >>>>> Pat >>>>> >>>>> On 8 Mar 2018, at 2:58 pm, mrmanishs <[email protected]> wrote: >>>>> >>>>> After further investigation, it looks like an issue with delta >>>>> indexing. Is there a fix? Should real-time indices be used instead, is >>>>> that >>>>> a better approach? >>>>> >>>>> On Wednesday, March 7, 2018 at 4:39:16 PM UTC-5, mrmanishs wrote: >>>>>> >>>>>> Whenever I try to do anything in my app through passenger, I get an >>>>>> error similar to this: >>>>>> >>>>>> F, [2018-03-07T21:35:13.949318 #30762] FATAL -- : >>>>>> [d6484d23-620d-4f33-8f17-13c12b7dab53] Riddle::CommandFailedError >>>>>> (Sphinx >>>>>> command failed to execute): >>>>>> F, [2018-03-07T21:35:13.949426 #30762] FATAL -- : >>>>>> [d6484d23-620d-4f33-8f17-13c12b7dab53] >>>>>> F, [2018-03-07T21:35:13.949536 #30762] FATAL -- : >>>>>> [d6484d23-620d-4f33-8f17-13c12b7dab53] >>>>>> app/controllers/listing_controller.rb: >>>>>> >>>>>> >>>>>> That's all I get... no other information. >>>>>> >>>>>> The same instructions work in console and there are no issues. >>>>>> Everything else on the rails app works fine. >>>>>> >>>>>> Any idea? What else can I do to get more information? >>>>>> >>>>>> Manish >>>>>> >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Thinking Sphinx" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to thinking-sphi...@googlegroups. <http://googlegroups.com/> >>>>> com <http://googlegroups.com/>. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/thinking-sphinx. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Thinking Sphinx" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/thinking-sphinx. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Thinking Sphinx" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/thinking-sphinx. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Thinking Sphinx" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/thinking-sphinx. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Thinking Sphinx" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at https://groups.google.com/group/thinking-sphinx. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/thinking-sphinx. For more options, visit https://groups.google.com/d/optout.
