In examining the difference in the generated sphinx.conf file on the affected systems between TS2 and TS3, we found that TS3 is not creating a "sql_query_pre = SET NAMES utf8" entry in the source blocks. If we manually add that line back on the generated sphinx.conf file, indexing works properly again.
On Thursday, February 28, 2013 12:45:01 PM UTC-5, Lonnie Warpup wrote: > > Further investigation leads me to think that the Chinese strings are not > getting indexed for some reason. I did the following test on my machine: > > indextool --config config/development.sphinx.conf --dumphitlist > translated_type_dd_core "交流" > Sphinx 2.0.6-release (r3473) > Copyright (c) 2001-2012, Andrew Aksyonoff > Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com) > > using config file 'config/development.sphinx.conf'... > dumping hitlist for index 'translated_type_dd_core' keyword '交流'... > keyword=交流, tok=交, wordid=2654699693 > doc=1526580, hit=0x01000001 > doc=1529924, hit=0x01000002 > doc=1546815, hit=0x01000001 > doc=1550159, hit=0x01000002 > doc=1570394, hit=0x01000002 > > > And then on the other servers that are having issues...: > > indextool --config config/staging.sphinx.conf --dumphitlist > translated_type_dd_core "交流" > Sphinx 2.0.6-release (r3473) > Copyright (c) 2001-2012, Andrew Aksyonoff > Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com) > > using config file 'config/staging.sphinx.conf'... > dumping hitlist for index 'translated_type_dd_core' keyword '交流'... > keyword=交流, tok=交, wordid=2654699693 > FATAL: failed to setup keyword > > > I also noted that the several of the > db/{environment}/translated_type_dd_core files were considerably smaller on > the other machines. > > I have quite a large charset_table defined, as well as an ngram_chars > definition. WHY it works on MY machine and not the others, I don't know. > > defaults: &defaults > max_matches: 5000 > charset_table: "0..9, a..z, _, -, U+0026, A..Z->a..z, U+00C0->a, > U+00C1->a, > U+00C2->a, U+00C3->a, U+00C4->a, U+00C5->a, U+00C7->c, U+00C8->e, > U+00C9->e, U+00CA->e, U+00CB->e, U+00CC->i, U+00CD->i, U+00CE->i, > U+00CF->i, U+00D1->n, U+00D2->o, U+00D3->o, U+00D4->o, U+00D5->o, > U+00D6->o, U+00D8->o, U+00D9->u, U+00DA->u, U+00DB->u, U+00DC->u, > U+00DD->y, U+00E0->a, U+00E1->a, U+00E2->a, U+00E3->a, U+00E4->a, > U+00E5->a, U+00E7->c, U+00E8->e, U+00E9->e, U+00EA->e, U+00EB->e, > U+00EC->i, U+00ED->i, U+00EE->i, U+00EF->i, U+00F1->n, U+00F2->o, > U+00F3->o, U+00F4->o, U+00F5->o, U+00F6->o, U+00F8->o, U+00F9->u, > U+00FA->u, U+00FB->u, U+00FC->u, U+00FD->y, U+00FF->y, U+0100->a, > U+0101->a, U+0102->a, U+0103->a, U+0104->a, U+0105->a, U+0106->c, > U+0107->c, U+0108->c, U+0109->c, U+010A->c, U+010B->c, U+010C->c, > U+010D->c, U+010E->d, U+010F->d, U+0112->e, U+0113->e, U+0114->e, > U+0115->e, U+0116->e, U+0117->e, U+0118->e, U+0119->e, U+011A->e, > U+011B->e, U+011C->g, U+011D->g, U+011E->g, U+011F->g, U+0120->g, > U+0121->g, U+0122->g, U+0123->g, U+0124->h, U+0125->h, U+0128->i, > U+0129->i, U+012A->i, U+012B->i, U+012C->i, U+012D->i, U+012E->i, > U+012F->i, U+0130->i, U+0134->j, U+0135->j, U+0136->k, U+0137->k, > U+0139->l, U+013A->l, U+013B->l, U+013C->l, U+013D->l, U+013E->l, > U+0142->l, U+0143->n, U+0144->n, U+0145->n, U+0146->n, U+0147->n, > U+0148->n, U+014C->o, U+014D->o, U+014E->o, U+014F->o, U+0150->o, > U+0151->o, U+0154->r, U+0155->r, U+0156->r, U+0157->r, U+0158->r, > U+0159->r, U+015A->s, U+015B->s, U+015C->s, U+015D->s, U+015E->s, > U+015F->s, U+0160->s, U+0161->s, U+0162->t, U+0163->t, U+0164->t, > U+0165->t, U+0168->u, U+0169->u, U+016A->u, U+016B->u, U+016C->u, > U+016D->u, U+016E->u, U+016F->u, U+0170->u, U+0171->u, U+0172->u, > U+0173->u, U+0174->w, U+0175->w, U+0176->y, U+0177->y, U+0178->y, > U+0179->z, U+017A->z, U+017B->z, U+017C->z, U+017D->z, U+017E->z, > U+01A0->o, U+01A1->o, U+01AF->u, U+01B0->u, U+01CD->a, U+01CE->a, > U+01CF->i, U+01D0->i, U+01D1->o, U+01D2->o, U+01D3->u, U+01D4->u, > U+01D5->u, U+01D6->u, U+01D7->u, U+01D8->u, U+01D9->u, U+01DA->u, > U+01DB->u, U+01DC->u, U+01DE->a, U+01DF->a, U+01E0->a, U+01E1->a, > U+01E6->g, U+01E7->g, U+01E8->k, U+01E9->k, U+01EA->o, U+01EB->o, > U+01EC->o, U+01ED->o, U+01F0->j, U+01F4->g, U+01F5->g, U+01F8->n, > U+01F9->n, U+01FA->a, U+01FB->a, U+0200->a, U+0201->a, U+0202->a, > U+0203->a, U+0204->e, U+0205->e, U+0206->e, U+0207->e, U+0208->i, > U+0209->i, U+020A->i, U+020B->i, U+020C->o, U+020D->o, U+020E->o, > U+020F->o, U+0210->r, U+0211->r, U+0212->r, U+0213->r, U+0214->u, > U+0215->u, U+0216->u, U+0217->u, U+0218->s, U+0219->s, U+021A->t, > U+021B->t, U+021E->h, U+021F->h, U+0226->a, U+0227->a, U+0228->e, > U+0229->e, U+022A->o, U+022B->o, U+022C->o, U+022D->o, U+022E->o, > U+022F->o, U+0230->o, U+0231->o, U+0232->y, U+0233->y, U+1E00->a, > U+1E01->a, U+1E02->b, U+1E03->b, U+1E04->b, U+1E05->b, U+1E06->b, > U+1E07->b, U+1E08->c, U+1E09->c, U+1E0A->d, U+1E0B->d, U+1E0C->d, > U+1E0D->d, U+1E0E->d, U+1E0F->d, U+1E10->d, U+1E11->d, U+1E12->d, > U+1E13->d, U+1E14->e, U+1E15->e, U+1E16->e, U+1E17->e, U+1E18->e, > U+1E19->e, U+1E1A->e, U+1E1B->e, U+1E1C->e, U+1E1D->e, U+1E1E->f, > U+1E1F->f, U+1E20->g, U+1E21->g, U+1E22->h, U+1E23->h, U+1E24->h, > U+1E25->h, U+1E26->h, U+1E27->h, U+1E28->h, U+1E29->h, U+1E2A->h, > U+1E2B->h, U+1E2C->i, U+1E2D->i, U+1E2E->i, U+1E2F->i, U+1E30->k, > U+1E31->k, U+1E32->k, U+1E33->k, U+1E34->k, U+1E35->k, U+1E36->l, > U+1E37->l, U+1E38->l, U+1E39->l, U+1E3A->l, U+1E3B->l, U+1E3C->l, > U+1E3D->l, U+1E3E->m, U+1E3F->m, U+1E40->m, U+1E41->m, U+1E42->m, > U+1E43->m, U+1E44->n, U+1E45->n, U+1E46->n, U+1E47->n, U+1E48->n, > U+1E49->n, U+1E4A->n, U+1E4B->n, U+1E4C->o, U+1E4D->o, U+1E4E->o, > U+1E4F->o, U+1E50->o, U+1E51->o, U+1E52->o, U+1E53->o, U+1E54->p, > U+1E55->p, U+1E56->p, U+1E57->p, U+1E58->r, U+1E59->r, U+1E5A->r, > U+1E5B->r, U+1E5C->r, U+1E5D->r, U+1E5E->r, U+1E5F->r, U+1E60->s, > U+1E61->s, U+1E62->s, U+1E63->s, U+1E64->s, U+1E65->s, U+1E66->s, > U+1E67->s, U+1E68->s, U+1E69->s, U+1E6A->t, U+1E6B->t, U+1E6C->t, > U+1E6D->t, U+1E6E->t, U+1E6F->t, U+1E70->t, U+1E71->t, U+1E72->u, > U+1E73->u, U+1E74->u, U+1E75->u, U+1E76->u, U+1E77->u, U+1E78->u, > U+1E79->u, U+1E7A->u, U+1E7B->u, U+1E7C->v, U+1E7D->v, U+1E7E->v, > U+1E7F->v, U+1E80->w, U+1E81->w, U+1E82->w, U+1E83->w, U+1E84->w, > U+1E85->w, U+1E86->w, U+1E87->w, U+1E88->w, U+1E89->w, U+1E8A->x, > U+1E8B->x, U+1E8C->x, U+1E8D->x, U+1E8E->y, U+1E8F->y, U+1E96->h, > U+1E97->t, U+1E98->w, U+1E99->y, U+1EA0->a, U+1EA1->a, U+1EA2->a, > U+1EA3->a, U+1EA4->a, U+1EA5->a, U+1EA6->a, U+1EA7->a, U+1EA8->a, > U+1EA9->a, U+1EAA->a, U+1EAB->a, U+1EAC->a, U+1EAD->a, U+1EAE->a, > U+1EAF->a, U+1EB0->a, U+1EB1->a, U+1EB2->a, U+1EB3->a, U+1EB4->a, > U+1EB5->a, U+1EB6->a, U+1EB7->a, U+1EB8->e, U+1EB9->e, U+1EBA->e, > U+1EBB->e, U+1EBC->e, U+1EBD->e, U+1EBE->e, U+1EBF->e, U+1EC0->e, > U+1EC1->e, U+1EC2->e, U+1EC3->e, U+1EC4->e, U+1EC5->e, U+1EC6->e, > U+1EC7->e, U+1EC8->i, U+1EC9->i, U+1ECA->i, U+1ECB->i, U+1ECC->o, > U+1ECD->o, U+1ECE->o, U+1ECF->o, U+1ED0->o, U+1ED1->o, U+1ED2->o, > U+1ED3->o, U+1ED4->o, U+1ED5->o, U+1ED6->o, U+1ED7->o, U+1ED8->o, > U+1ED9->o, U+1EDA->o, U+1EDB->o, U+1EDC->o, U+1EDD->o, U+1EDE->o, > U+1EDF->o, U+1EE0->o, U+1EE1->o, U+1EE2->o, U+1EE3->o, U+1EE4->u, > U+1EE5->u, U+1EE6->u, U+1EE7->u, U+1EE8->u, U+1EE9->u, U+1EEA->u, > U+1EEB->u, U+1EEC->u, U+1EED->u, U+1EEE->u, U+1EEF->u, U+1EF0->u, > U+1EF1->u, U+1EF2->y, U+1EF3->y, U+1EF4->y, U+1EF5->y, U+1EF6->y, > U+1EF7->y, U+1EF8->y, U+1EF9->y, > U+0622->U+0627, U+0623->U+0627, U+0624->U+0648, U+0625->U+0627, > U+0626->U+064A, U+06C0->U+06D5, U+06C2->U+06C1, U+06D3->U+06D2, > U+FB50->U+0671, U+FB51->U+0671, U+FB52->U+067B, U+FB53->U+067B, > U+FB54->U+067B, U+FB56->U+067E, U+FB57->U+067E, U+FB58->U+067E, > U+FB5A->U+0680, U+FB5B->U+0680, U+FB5C->U+0680, U+FB5E->U+067A, > U+FB5F->U+067A, U+FB60->U+067A, U+FB62->U+067F, U+FB63->U+067F, > U+FB64->U+067F, U+FB66->U+0679, U+FB67->U+0679, U+FB68->U+0679, > U+FB6A->U+06A4, U+FB6B->U+06A4, U+FB6C->U+06A4, U+FB6E->U+06A6, > U+FB6F->U+06A6, U+FB70->U+06A6, U+FB72->U+0684, U+FB73->U+0684, > U+FB74->U+0684, U+FB76->U+0683, U+FB77->U+0683, U+FB78->U+0683, > U+FB7A->U+0686, U+FB7B->U+0686, U+FB7C->U+0686, U+FB7E->U+0687, > U+FB7F->U+0687, U+FB80->U+0687, U+FB82->U+068D, U+FB83->U+068D, > U+FB84->U+068C, U+FB85->U+068C, U+FB86->U+068E, U+FB87->U+068E, > U+FB88->U+0688, U+FB89->U+0688, U+FB8A->U+0698, U+FB8B->U+0698, > U+FB8C->U+0691, U+FB8D->U+0691, U+FB8E->U+06A9, U+FB8F->U+06A9, > U+FB90->U+06A9, U+FB92->U+06AF, U+FB93->U+06AF, U+FB94->U+06AF, > U+FB96->U+06B3, U+FB97->U+06B3, U+FB98->U+06B3, U+FB9A->U+06B1, > U+FB9B->U+06B1, U+FB9C->U+06B1, U+FB9E->U+06BA, U+FB9F->U+06BA, > U+FBA0->U+06BB, U+FBA1->U+06BB, U+FBA2->U+06BB, U+FBA4->U+06C0, > U+FBA5->U+06C0, U+FBA6->U+06C1, U+FBA7->U+06C1, U+FBA8->U+06C1, > U+FBAA->U+06BE, U+FBAB->U+06BE, U+FBAC->U+06BE, U+FBAE->U+06D2, > U+FBAF->U+06D2, U+FBB0->U+06D3, U+FBB1->U+06D3, U+FBD3->U+06AD, > U+FBD4->U+06AD, U+FBD5->U+06AD, U+FBD7->U+06C7, U+FBD8->U+06C7, > U+FBD9->U+06C6, U+FBDA->U+06C6, U+FBDB->U+06C8, U+FBDC->U+06C8, > U+FBDD->U+0677, U+FBDE->U+06CB, U+FBDF->U+06CB, U+FBE0->U+06C5, > U+FBE1->U+06C5, U+FBE2->U+06C9, U+FBE3->U+06C9, U+FBE4->U+06D0, > U+FBE5->U+06D0, U+FBE6->U+06D0, U+FBE8->U+0649, U+FBFC->U+06CC, > U+FBFD->U+06CC, U+FBFE->U+06CC, U+0621, U+0627..U+063A, > U+0641..U+064A, > U+0660..U+0669, U+066E, U+066F, U+0671..U+06BF, U+06C1, > U+06C3..U+06D2, > U+06D5, U+06EE..U+06FC, U+06FF, U+0750..U+076D, U+FB55, U+FB59, > U+FB5D, > U+FB61, U+FB65, U+FB69, U+FB6D, U+FB71, U+FB75, U+FB79, U+FB7D, > U+FB81, > U+FB91, U+FB95, U+FB99, U+FB9D, U+FBA3, U+FBA9, U+FBAD, U+FBD6, > U+FBE7, > U+FBE9, U+FBFF" > ngram_len: 1 > ngram_chars: "U+4E00..U+9FBB, U+3400..U+4DB5, U+20000..U+2A6D6, U+FA0E, > U+FA0F, > U+FA11, U+FA13, U+FA14, U+FA1F, U+FA21, U+FA23, U+FA24, U+FA27, > U+FA28, > U+FA29, U+3105..U+312C, U+31A0..U+31B7, U+3041, U+3043, U+3045, > U+3047, > U+3049, U+304B, U+304D, U+304F, U+3051, U+3053, U+3055, U+3057, > U+3059, > U+305B, U+305D, U+305F, U+3061, U+3063, U+3066, U+3068, > U+306A..U+306F, > U+3072, U+3075, U+3078, U+307B, U+307E..U+3083, U+3085, U+3087, > U+3089..U+308E, > U+3090..U+3093, U+30A1, U+30A3, U+30A5, U+30A7, U+30A9, U+30AD, > U+30AF, > U+30B3, U+30B5, U+30BB, U+30BD, U+30BF, U+30C1, U+30C3, U+30C4, > U+30C6, > U+30CA, U+30CB, U+30CD, U+30CE, U+30DE, U+30DF, U+30E1, U+30E2, > U+30E3, > U+30E5, U+30E7, U+30EE, U+30F0..U+30F3, U+30F5, U+30F6, U+31F0, > U+31F1, > U+31F2, U+31F3, U+31F4, U+31F5, U+31F6, U+31F7, U+31F8, U+31F9, > U+31FA, > U+31FB, U+31FC, U+31FD, U+31FE, U+31FF, U+AC00..U+D7A3, > U+1100..U+1159, > U+1161..U+11A2, U+11A8..U+11F9, U+A000..U+A48C, U+A492..U+A4C6" > > > > On Thursday, February 28, 2013 12:01:01 PM UTC-5, Lonnie Warpup wrote: >> >> I cannot figure out the differences and this is driving me crazy. >> >> Do you have any suggestions on how I can interact with the sphinx engine >> directly for a test, and not go through TS? I want to see if it is TS, or >> sphinx itself. These queries worked prior to my upgrade to TS3. I tried >> using the search command line tool, but that appears to be broken and not >> supported anymore. >> >> On Wednesday, February 27, 2013 10:19:18 PM UTC-5, Pat Allan wrote: >>> >>> brew adds a few flags in automatically - what flags are you compiling >>> with on the Ubuntu/Debian boxes? >>> https://github.com/mxcl/homebrew/blob/master/Library/Formula/sphinx.rb >>> >>> If you're using stemming, this could be related… although unlikely. >>> >>> -- >>> Pat >>> >>> On 27/02/2013, at 1:15 AM, Lonnie Warpup wrote: >>> >>> > The OS X installation was done with brew, and the others were done >>> from source (http://sphinxsearch.com/docs/2.0.6/installing.html). Does >>> this help any? >>> > >>> > On Monday, February 25, 2013 8:53:06 PM UTC-5, Pat Allan wrote: >>> > This is a little surprising $)A !- >>> > What versions of Ruby are you using across all machines? And how was >>> Sphinx installed on each? Compiled by hand? Apt packages? Homebrew? >>> > >>> > -- >>> > Pat >>> > >>> > On 26/02/2013, at 6:04 AM, Lonnie Warpup wrote: >>> > >>> > > Also of note, my development machine is an OS X machine, the staging >>> server is Debian Squeeze, and another developer's machine (also not finding >>> any records) is ubuntu 12.04. >>> > > >>> > > On Monday, February 25, 2013 12:46:24 PM UTC-5, Lonnie Warpup wrote: >>> > > Pat, >>> > > >>> > > I know that Thinking Sphinx is an API to Sphinx itself, but perhaps >>> you know this. I have a local development machine and a staging server, >>> both with the same version of thinking_sphinx (213447d230) and the same >>> version of sphinx (2.0.6 r3473). The sphinx indexes have been built the >>> same way on both machines, however if I attempt to search for a non-English >>> character set on the staging server I am coming up with zero results, yet >>> it works on my local machine. Below are the console histories and the >>> query logs to see that they are going through identically, but with >>> different results. Does Sphinx use its own database or whatever? I'm >>> trying to figure out what the differences are... >>> > > >>> > > Local Development Machine: >>> > > >>> > > 1.9.3p194 :001 > tt = >>> Type.find('motors-ac').translations.where(:locale => 'zh-CN').first >>> > > Type Load (0.5ms) SELECT `types`.* FROM `types` WHERE >>> `types`.`cached_slug` = 'motors-ac' AND (`types`.`name` != 'znull') LIMIT 1 >>> > > TranslatedType Load (0.3ms) SELECT `translated_types`.* FROM >>> `translated_types` WHERE `translated_types`.`type_id` = 1220 AND >>> `translated_types`.`locale` = 'zh-CN' LIMIT 1 >>> > > => #<TranslatedType id: 80346, type_id: 1220, type_class: >>> "ToolingType", locale: "zh-CN", name: " $)A =;Aw5g;z ", parent_name: " 5g;z >>> ", created_at: "2013-02-14 16:58:12", updated_at: "2013-02-14 16:58:12"> >>> > > 1.9.3p194 :002 > TranslatedType.search("#{tt.name[0,2]}*") >>> > > TranslatedType Load (0.4ms) SELECT `translated_types`.* FROM >>> `translated_types` WHERE `translated_types`.`id` IN (80346, 81411) >>> > > => [#<TranslatedType id: 80346, type_id: 1220, type_class: >>> "ToolingType", locale: "zh-CN", name: " $)A =;Aw5g;z ", parent_name: " 5g;z >>> ", created_at: "2013-02-14 16:58:12", updated_at: "2013-02-14 16:58:12">, >>> #<TranslatedType id: 81411, type_id: 1220, type_class: "ToolingType", >>> locale: "zh-TW", name: " =;Aw $)GgYq" ", parent_name: " gYq" ", created_at: >>> "2013-02-14 16:58:17", updated_at: "2013-02-14 16:58:17">] >>> > > >>> > > development.searchd.query.log >>> > > >>> > > [Mon Feb 25 12:22:29.866 2013] 0.001 sec [ext2/1/ext 2 (0,20)] >>> [translated_type_dd_core`, `translated_type_dds_core] $)A =;Aw * >>> @sphinx_internal_class_name (TranslatedType) >>> > > >>> > > >>> > > Staging Server: >>> > > >>> > > irb(main):001:0> tt = >>> Type.find('motors-ac').translations.where(:locale => 'zh-CN').first >>> > > Type Load (0.2ms) SELECT `types`.* FROM `types` WHERE >>> `types`.`cached_slug` = 'motors-ac' AND (`types`.`name` != 'znull') LIMIT 1 >>> > > TranslatedType Load (0.1ms) SELECT `translated_types`.* FROM >>> `translated_types` WHERE `translated_types`.`type_id` = 1220 AND >>> `translated_types`.`locale` = 'zh-CN' LIMIT 1 >>> > > => #<TranslatedType id: 22439, type_id: 1220, type_class: >>> "ToolingType", locale: "zh-CN", name: " $)A =;Aw5g;z ", parent_name: " 5g;z >>> ", created_at: "2013-02-25 16:30:07", updated_at: "2013-02-25 16:30:07"> >>> > > irb(main):002:0> TranslatedType.search("#{tt.name[0,2]}*") >>> > > => [] >>> > > >>> > > staging.searchd.query.log >>> > > >>> > > [Mon Feb 25 11:21:20.121 2013] 0.000 sec [ext2/1/ext 0 (0,20)] >>> [translated_type_dd_core`, `translated_type_dds_core] $)A =;Aw * >>> @sphinx_internal_class_name (TranslatedType) >>> > > >>> > > Thanks. >>> > > >>> > > -Lonnie >>> > > >>> > > -- >>> > > You received this message because you are subscribed to the Google >>> Groups "Thinking Sphinx" group. >>> > > To unsubscribe from this group and stop receiving emails from it, >>> send an email to [email protected]. >>> > > To post to this group, send email to [email protected]. >>> > > Visit this group at >>> http://groups.google.com/group/thinking-sphinx?hl=en. >>> > > For more options, visit https://groups.google.com/groups/opt_out. >>> > > >>> > > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Thinking Sphinx" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To post to this group, send email to [email protected]. >>> > Visit this group at >>> http://groups.google.com/group/thinking-sphinx?hl=en. >>> > For more options, visit https://groups.google.com/groups/opt_out. >>> > >>> > >>> >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Thinking Sphinx" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/thinking-sphinx?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
