[sup-devel] Cannot query Japanese characters

2011-04-24 Thread Horacio Sanson
27.0.0.1 - - [25/Apr/2011 10:17:28] "GET /favicon.ico HTTP/1.1" 404 441 0.0008 localhost.localdomain - - [25/Apr/2011:10:17:28 JST] "GET /favicon.ico HTTP/1.1" 404 441 I am running the latest heliotrope from git with ruby 1.9.2 from the default Kubuntu 10.10 distribution. -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

[sup-devel] turnsole cannot handle Japanese labels.

2011-04-24 Thread Horacio Sanson
3:in `step' from bin/turnsole:134:in `' -- regards, Horacio Sanson __

Re: [sup-devel] Cannot query Japanese characters

2011-05-01 Thread Horacio Sanson
Installed whistelpig 0.6 but now I get a different error that looks similar to the turnsole problem. Below the backtrace: http://localhost:8042/search?q=primo -> /search?q=%7Einbox&start=0&num=20 127.0.0.1 - - [02/May/2011 00:31:58] "GET /favicon.ico HTTP/1.1" 404 447 0.0008 localhost - - [02/May/

Re: [sup-devel] Cannot query Japanese characters

2011-05-01 Thread Horacio Sanson
start. regards, Horacio On Mon, May 2, 2011 at 12:35 AM, Horacio Sanson wrote: > Installed whistelpig 0.6 but now I get a different error that looks > similar to the turnsole problem. Below the backtrace: > > http://localhost:8042/search?q=primo -> /search?q=%7Einbox&start=0&nu

Re: [sup-devel] Cannot query Japanese characters

2011-05-03 Thread Horacio Sanson
", "研究" => body:"研究" > results1 = index.search q1 => [] I will now dig in Whistelpig source code to see if I can fix this but any pointer/directions or tips were to start looking would be greatly appreciated. On Mon, May 2, 2011 at 12:46 AM, Horacio Sanson wrote: &

Re: [sup-devel] Cannot query Japanese characters

2011-05-03 Thread Horacio Sanson
Forgot to mention you need the mecab ruby gem. In Ubuntu 10.04 this gem is part of the distribution and can be installed with the command: sudo apt-get install libmecab-ruby1.8 libmecab-ruby1.9.1 mecab-ipadic-utf8 regards Horacio On Wed, May 4, 2011 at 10:42 AM, Horacio Sanson wrote: > Cha

Re: [sup-devel] Cannot query Japanese characters

2011-05-03 Thread Horacio Sanson
atch [:punct:]. This > is all pretty ad-hoc and undocumented. Providing simpler whitespace-only > tokenizer as an alternative is in the works. > -- > William > ___ > Sup-devel mailing list > Sup-devel@rubyforge.org > http://ru

Re: [sup-devel] Cannot query Japanese characters

2011-05-05 Thread Horacio Sanson
Great, let me know when you have the modifications so I can stress test them. regards, Horacio On Thu, May 5, 2011 at 1:56 AM, William Morgan wrote: > Hi Horacio, > > Thanks for all your help so far. > > Reformatted excerpts from Horacio Sanson's message of 2011-05-04: >> After some hacking I go

[sup-devel] Query for largest msg_id?

2011-05-10 Thread Horacio Sanson
Is there a way to query Heliotrope what is the largest msg_id currently in the index? I am trying to improve the imap-dumper.rb so it does not download all my emails every time but only the new ones. >From rfc4549.txt (see section 4.3.1) I learned that the way to do this is to FETCH all the message

Re: [sup-devel] Query for largest msg_id?

2011-05-16 Thread Horacio Sanson
On Mon, May 16, 2011 at 12:01 AM, William Morgan wrote: > Reformatted excerpts from Horacio Sanson's message of 2011-05-10: >> Is there a way to query Heliotrope what is the largest msg_id >> currently in the index? > > Sort of---it's a hack, but if you search for e.g. "a OR -a" you'll get > every

Re: [sup-devel] Query for largest msg_id?

2011-05-17 Thread Horacio Sanson
Sorry the patch I sent has a small problem. Use this patch instead. On Tue, May 17, 2011 at 11:59 PM, Horacio Sanson wrote: > I implemented a new version of the GMail -> Heliotrope sync script and > attach it here in hopes > someone will test it and provide some feedback/comments. T

Re: [sup-devel] Query for largest msg_id?

2011-05-17 Thread Horacio Sanson
tached patch). regards, Horacio On Tue, May 17, 2011 at 12:02 AM, Horacio Sanson wrote: > On Mon, May 16, 2011 at 12:01 AM, William Morgan > wrote: >> Reformatted excerpts from Horacio Sanson's message of 2011-05-10: >>> Is there a way to query Heliotrope what is the lar

Re: [sup-devel] Query for largest msg_id?

2011-05-20 Thread Horacio Sanson
On Wed, May 18, 2011 at 2:05 PM, William Morgan wrote: > Reformatted excerpts from Horacio Sanson's message of 2011-05-17: >> I implemented a new version of the GMail -> Heliotrope sync script and >> attach it here in hopes someone will test it and provide some >> feedback/comments. > > Great! Thi

[sup-devel] Heliotrope limitations for backward synchronization, , ,

2011-05-23 Thread Horacio Sanson
Delete next pending action on mailbox POST ///message # Add new message to an account/mailbox GET///message # Get all messages from account/mailbox This of course is how I think IMAP offline synchronization could be done. If there are other ways to do this I am

Re: [sup-devel] Query for largest msg_id?

2011-05-23 Thread Horacio Sanson
On Tue, May 24, 2011 at 6:22 AM, Hamish wrote: > Excerpts from Horacio Sanson's message of Fri May 20 16:38:30 +0100 2011: >> On Wed, May 18, 2011 at 2:05 PM, William Morgan wrote: >> > Reformatted excerpts from Horacio Sanson's message of 2011-05-17: >> >> I implemented a new version of the GMail

Re: [sup-devel] Cannot query Japanese characters

2011-06-09 Thread Horacio Sanson
, Jun 9, 2011 at 10:48 PM, Horacio Sanson wrote: > On Wed, Jun 8, 2011 at 2:21 PM, William Morgan > wrote: >> Reformatted excerpts from Horacio Sanson's message of 2011-05-06: >>> Great, let me know when you have the modifications so I can stress >>> test them. &

Re: [sup-devel] Cannot query Japanese characters

2011-06-09 Thread Horacio Sanson
tails. regards, Horacio > -- > William > ___ > Sup-devel mailing list > Sup-devel@rubyforge.org > http://rubyforge.org/mailman/listinfo/sup-devel > gmail From 4bf24f16612c954bbbdcdb9b48a70571c3bb1a4d Mon Sep 17 00:00:00 2001 F

[sup-devel] Invalid meta data error in OklahomaMixer

2011-06-11 Thread Horacio Sanson
After finishing sync of my gmail account to heliotrope I tried to start the server but was received with this error message: ruby1.9.1 -Ilib bin/heliotrope-server -d ~/.heliotrope /var/lib/gems/1.9.1/gems/oklahoma_mixer-0.4.0/lib/oklahoma_mixer/hash_database.rb:337:in `try': invalid meta data (er

Re: [sup-devel] Invalid meta data error in OklahomaMixer

2011-06-13 Thread Horacio Sanson
Not even close... my store.tch is 76MB only. This metadata corruption in Tokyo Cabinet seems to be a common occurrence and what is scary about this is that there seems to be no way to recover from it (as far as google can tell). The only option I see is delete my store/index and restart sync again.

Re: [sup-devel] Invalid meta data error in OklahomaMixer

2011-06-13 Thread Horacio Sanson
I managed to get the store.tch working again by installing tokyocabinet-bin and running this command on the file: tchmgr optimize -nl store.tch # without -nl nothing works... at the end of this command I got a write error but the file got back to a usable state. I am afraid this command simpl

[sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-14 Thread Horacio Sanson
Well I will retract what I said earlier... maybe is a good idea to replace TokyoCabinet with something else. Now the store.tch got corrupted again for some reason and the only error I get is "invalid record header". Some googling does not help and my previous fix (tchmgr optimiza -nl) does not fix

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-14 Thread Horacio Sanson
Interesting yesterday I started again to sync my gmail account but this time using ruby1.8.7 instead of ruby1.9.1 and without any hooks configured and it finished without problems. I think the problem is the hooks so I will rerun again with ruby 1.9.1 but without the hooks. On Wed, Jun 15, 20

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-19 Thread Horacio Sanson
ails to find it. -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-19 Thread Horacio Sanson
version 0.4. I believe it should work for oyu. -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-19 Thread Horacio Sanson
http://rubygems.org/gems/leveldb-ruby -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-20 Thread Horacio Sanson
with standard Ubuntu 11.04 distribution. -- regards, Horacio Sanson __

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-22 Thread Horacio Sanson
e leveldb branch. > Thanks! Thanks to you -- regards, Horacio Sanson ___

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-22 Thread Horacio Sanson
gem > > And tell me if that works? > > If so, I will release that as 0.5. > > Thanks! -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

Re: [sup-devel] Tokyo Cabinet fails with invalid record header

2011-06-23 Thread Horacio Sanson
no messages even though it reached 130378 messages indexed before crashing with a nil error. -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

[sup-devel] Heliotrope improving but still found some issues

2011-07-04 Thread Horacio Sanson
ly stop reading emails for some time when we get the first nil response. Overall heliotrope is now usable for Japanese language users (at least for me ). Now I will start playing with turnsole to see if it can handle japanese. -- regards,

Re: [sup-devel] Heliotrope improving but still found some issues

2011-07-05 Thread Horacio Sanson
require a separate hook or something like that. regards, Horacio On Tue, Jul 5, 2011 at 10:52 AM, Horacio Sanson wrote: > > So I tried the latest heliotrope with the leveldb-ruby 0.6 gem, whistlepig 0.7 > and MeCab hooks for Japanese text support and it works better than befor

[sup-devel] How are the queries supposed to work?

2011-07-06 Thread Horacio Sanson
quot;, "[Rails]")) index.set_query(Query.new("subject", "Test fixtures")) index.set_query(Query.new("subject", "test fixtures")) On all tests I made sure to run index.reset_query! before setting the new query with index.set_query. Is this the correct way???

Re: [sup-devel] How are the queries supposed to work?

2011-07-07 Thread Horacio Sanson
clty. Maybe I was already tired of so much testing and forgot to actually run the query after setting it. > > On all tests I made sure to run index.reset_query! before setting the > > new query with index.set_query. Is this the correct way??? > > The reset_query! is unnecessary. > > Thanks for all your testing. Much of this is undocumented, so I ask you > to bear with me. Once the UTF-7 encoding issue with the labels get's fixed I will test querying with Japanese labels. -- regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

Re: [sup-devel] Heliotrope improving but still found some issues

2011-07-09 Thread Horacio Sanson
k 0.00s and was NOT continued" # 0 # # # end If I put a force_encoding at the end of the hook I get: start # # # # "Search took 0.00s and was NOT continued" # 20 # # # end I need to re-index my emails with the new UTF-8 hooks and test the search again. -- regards,

Re: [sup-devel] sup 0.13

2013-04-30 Thread Horacio Sanson
Great to see Sup getting back on track again.. I submitted some patches for the Gmail dumper of Heliotrope some time ago but the lack of non alphabet languages (Japanese, Chinese) made it impossible for me to keep using heliotrope/turnesole. The main issue to support Japanese/Chinese with heliotr

Re: [sup-devel] sup 0.13

2013-05-08 Thread Horacio Sanson
stem language can be configured but there are no CJK stemmers for Xapian that I can find. On Thu, May 2, 2013 at 5:17 PM, Gaute Hope wrote: > > > On 30. april 2013 11:44, Horacio Sanson wrote: > > Great to see Sup getting back on track again.. > > > > I submitted some p

[sup-devel] Experimental Gmail Source

2013-05-09 Thread Horacio Sanson
the maildir source does it would be appreciated. regards Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

[sup-devel] How to get labels from sup for a specific message id?

2013-05-10 Thread Horacio Sanson
For my gmail source [1] I need to find out the labels for each message in sup. This is to update the remote Gmail with the labels the user set using sup. I am not very familiar with sup source code and my attempts to extract the message labels based on the message id have failed. My last try was:

Re: [sup-devel] Experimental Gmail Source

2013-05-11 Thread Horacio Sanson
On Sat, May 11, 2013 at 7:15 PM, Gaute Hope wrote: > > > On 09. mai 2013 11:28, Horacio Sanson wrote: > > I am trying to implement a new source for Gmail accounts. This is > > copied from my efforts to do the same in Heliotrope. > > > > Here is an experimental

Re: [sup-devel] Experimental Gmail Source

2013-05-14 Thread Horacio Sanson
ource provided id? - Is a good idea to add a term to the index that keeps the source id along with the message id? regards, Horacio On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson wrote: > > > > On Sat, May 11, 2013 at 7:15 PM, Gaute Hope wrote: > >> >> >> On 09

Re: [sup-devel] Experimental Gmail Source

2013-06-01 Thread Horacio Sanson
Thanks for checking the source and sorry for the late response... I can only look into this on rare free weekends. On Wed, May 22, 2013 at 6:47 AM, Matthieu Rakotojaona < matthieu.rakotoja...@gmail.com> wrote: > Hey Horacio, > > I took a stab at your gmail_source branch, and made a few > fixes/i

[sup-devel] Gmail source sync back.

2013-07-03 Thread Horacio Sanson
nd may not finish before the next polling starts. How to do this efficiently? regards, Horacio Sanson ___ Sup-devel mailing list Sup-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/sup-devel

[sup-devel] How do I force flush the log file in sup?

2013-08-02 Thread Horacio Sanson
Is there a way to force sup to flush the log file after each write? It is difficult to debug my gmail source as I cannot see real time what is going on. Maybe set sup to always flush when SUP_LOG_LEVEL is set to debug? regards, Horacio ___ Sup-devel mail

[sup-devel] Xapian 1.2.15 dependency causes issues on Ubuntu 13.04

2013-08-02 Thread Horacio Sanson
Ubuntu 13.04 installs the distribution xapian dev and lib packages that are version 1.2.12. This causes sup to complain that the xapian 1.2.15 or later is required and bails out. I understand xapian-ruby installs its own xapian lib but sup still tries to load the system one. I changed the index.r