Re: [Gossip] Partial match search
>What type of partial match search is supported on mail-archive? Here's the short and long explanation of search syntax. https://www.mail-archive.com/faq.html#search https://www.mail-archive.com/searching.html ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] The Mail Archive transitioning away from a small business
Dear Friends, The Mail Archive has been running for 20 years. It started as hobby and grew into a small business 14 years ago. We have now come full circle. The business will end on December 31, 2018 and the service will revert to a hobby. What happened? Well, traffic has steadily declined for many years and with that so does revenue. It's finally reached the point where it no longer makes financial sense to pay the overhead of running a small business, even a super lean one like The Mail Archive. So what does this mean? For the rest of 2018, nothing at all. In 2019 we switch to my personal hobby and see how that goes. The financial costs of running the service are not that big of a concern. However, I am a little worried about being back on the front line of customer service. That stuff is no fun and is probably the biggest risk to The Mail Archive's ultimate lifespan. I would like to deeply thank my business partners Jeff & Tom who have done great things over the years. And also to the many other helpers, you know who you are. It's been a wild ride and I'm proud to have helped some people along the way. Cheers, Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] uptime
I know nobody cares, but I noticed that The Mail Archive achieved over 99.99% uptime for a year. Kind of neat. We'll see if this message jinxes it. For comparison, if you count the recent total eclipse in the United States as a failure, the sun had about 99.9995% uptime. https://www.mail-archive.com/faq.html#monitoring ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] Some mails missing from us...@open-mpi.org archive
We use MHonArc to render emails for the web, which is open source. So any impatient programmers who are really hungry for this feature may want to dust off their Perl programming skills. Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] The Mail Archive, 2016
A lot of crazy stuff happened in 2016. Looking back, what parts of that touched The Mail Archive? Let's start with mundane computer stuff. Uptime was great, the service was online for the entire year except for 9 hours, 12 minutes. Some additional hard drives were converted to SSD, and some RAM was added just to make things a little faster. Our SSL certificate vendor surprised us when they were bought by another company. We plan to change vendors in 2017, and will be dropping the extended validation certificate as part of this. Software was very stable. The most visible improvement was making The Mail Archive's homepage mobile friendly, but there were a few behind the scenes tweaks especially to the search engine. Version control says only about 600 lines code changed in total over the year. In July, Lars Magne Ingebrigtsen decided to stop running Gmane, which is another long running email archiving service. We offered him support (mostly emotional!) and so far it looks like that service has transferred to new hands. Hopefully that will be successful. It's another reminder just how much people behind infrastructure matter. In November, we changed policy to make message deletion a little easier than before. So far there has not been a significant difference in the number of requested deletions. From a traffic perspective, overall visitors continues to decline. We are keeping an eye on this and will make adjustments as appropriate. This year we donated to The Marine Mammal Center, Doctors Without Borders, and Planned Parenthood. As always, thank you for using the service and I hope we can contribute positively to 2017. Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] policy change for message deletion
We’ve decided to make a policy change for The Mail Archive regarding message removal. For the last eighteen years (time flies!) list administrators have been responsible for decisions on removing content from publicly archived lists. We have had this policy in place in order to preserve the integrity of the public lists and to avoid The Mail Archive having to make decisions on removing archived public list content. Thank you to all the admins who have made those decisions over the years. The new policy is a strict expansion. List administrators may still remove any message from their list. In addition, any poster may request removal of messages that that they themselves have written without having to get administrator approval. Deletion requests for messages written by another user (including messages where an individual’s public post may have been quoted by someone else) will still require list administrator approval. We feel this policy strikes a balance between respect for individual privacy and respect for the concept of public mailing lists and open public discussion. As always, removal requests should be sent to our support address ( themailarch...@gmail.com). Thank you again for using The Mail Archive. Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] Replacing StartCom certificate
Thanks for the heads up. Highly appreciated. I'm impressed that you know the certificate vendor for The Mail Archive. I was not aware of the drama going on with StartCom. Is it correct that the removal only applies to new certificates, and therefore the deadline for action is May 3, 2017 when the current certificate expires? Or is it more urgent than that? Also, does the trust store removal include extended validation certificates? This quite a bummer, as it took a whole lot of paperwork to get that EV certificate which presumably will have to be redone with a new vendor. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] Gmane
Today I want to express my support and appreciation for Lars Magne Ingebrigtsen. For those not familiar, Lars created and has been running the utterly terrific email archiving service Gmane for 14 years. I've long admired both Gmane's engineering excellence and integrity. Unfortunately, one downside of wide impact is stress. Angry people. Vandalism, in the form of denial of service computer attacks. I don't know everything Lars has been through, but it's substantial and he's frustrated. It's time for change. https://lars.ingebrigtsen.no/2016/07/28/the-end-of-gmane/ I don't know exactly what the change will be, but The Mail Archive will do anything we can to help. I've contacted Lars and made that clear. I suspect he's getting an outpouring of support, and hopefully will be able to choose amongst good options for the future. Thank you Lars, for everything. Especially the fun. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] certificate chain is incomplete
Thanks for the detailed report. I made some changes and now get a 'A' rating on the online test. Does this fix the Android problems? ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] 2015 end of year
Once upon a time, an ancient fish crawled out of the water and into the mud. That was an important moment in the grand journey of life. Its children learned to live on the land and eventually became us. On December 21, I watched the a young company called SpaceX successfully land a rocket booster stage, opening the door to affordable space flight. Perhaps some of our descendents will remember this moment similarly, wherever and however they might be living. When I was a little boy, there was a book with a timeline in it with predictions for all sorts of space related accomplishments. There was a projected date for the Hubble Telescope and for a permanent space station, and for people visiting Mars and the Jovian moons, and well beyond. I'd sit there calculating how long I might live, and therefore what wonders I would see. It doesn't work that way. Things happen because people do them. Because we do them. And often we don't. We put people on the moon in 1969 with half our current world population. The atom was split in earnest in 1945. But these and so many other fields including my personal field of study, optical holography, have stagnated and declined for decades. By the time I had the opportunity to listen to Neal Armstrong and Buzz Aldrin speak at MIT in 1995, I asked Mr. Armstrong if he thought people would walk on the moon during my lifetime. And then, a group of very hard working people basically said, "Screw that, we're going to do something great." And landed that rocket. Which brings us now, finally, to The Mail Archive. When I started it 18 years ago, it was not an ambition in any way; it was a set of personal email filtering rules that were fun to play with. It was safe and easy to grow into a low key small business, and we did some good stuff at the usual slow but steady pace. In 2015 we donated some of our earnings to aid for Syrian refugees, a search and rescue team in the Sierra mountains, and to an animal shelter. We tweaked the search interface to allow full thread reading and introduced easier hotkeys (try clicking the subject line of any email then hitting 'e' for expand.) We switched to 100% encryption and upgraded to a fancier digital certificate. Backups are even more serious now and we even keep on set on SSD. Message-ids are at the bottom of every single message page for all three users who like that. Plus an obscure bugfix here and there usually related to search. I'm proud of that. I'm proud that we're still alive. I'm proud that we've held up as a small business, through thick and thin. 2015 was relatively thin due to less visitors than years past. And I'm proud that in some very small way we've helped some people do their own things. https://www.mail-archive.com/discuss-gnuradio%40gnu.org/msg52584.html As you think about the year ahead, please enjoy The Mail Archive, put it to good use, and don't be afraid to reach for the stars. The Mail Archive hasn't accomplished anything bold and fundamental. But maybe you can. And should. Happy 2016. Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] archive not updated for board-disc...@documentfoundation.org mailinglist
The spam filtering service we use (SpamHero) quarantined that message along with some others. I've released the messages from quarantine and also adjusted the whitelist to hopefully reduce or prevent this from happening in the future. I'm sorry about this and we would definitely consider switching to a better spam filtering service if we can find one. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] Porting digested new list archives to mail-archive
Yes, you can safely leave out To, Message-id, and Received. Consequences are what you'd expect, like the inability to do a message-id search and find that particular message. You are correct. Posting address is manually assigned during the bulk import process, and automatically determined from headers for regular inbound mail. Think of it as if Mail Archive was trying to put every message in a folder, where the folder name is the posting address. The folder name is indexed, so is available for search. You are exactly correct about the 'l' parameter. This is probably going to work fine, and let's go ahead and give it a try. If it doesn't work, we'll discuss, figure it out, and try again. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] Porting digested new list archives to mail-archive
The only things indexed for search are: message-id, subject, date (usually extracted from the Recieved: header), sender name (extracted from From: header), posting address (for example, gossip@mail-archive.com), archival message number, and message body. Every message is sorted and organized according to posting address. The To: header is examined when sorting regular inbound mail and is a factor when deciding where it belongs. But the To: header is never indexed for search, never used during import, and there is no benefit for you to adjust it. A merged archive will have the same posting address for every message, with no memory about what life was like before the merge. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] Porting digested new list archives to mail-archive
Statute of limitations is typically 3 kilomessages on a normal non-import list, but should (I think) be unlimited on bulk import. Conversion to unix newlines is required and is manual; doesn't matter who does it. Still prefer to do whole import at once especially if tricky; less labor, also less likely to break URLs if it takes multiple attempts to get it right. But we can accommodate two stages. Imports are done on weekends only. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com https://www.mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] experimental search interface, feedback requested
We're experimenting with a new user interface for search. It works a little differently, what do people think? To try this on your own list, just replace search in the URL with searchdev. Cheers, Jeff === OLD http://www.mail-archive.com/search?q=squirrell=cayugabirds-l%40cornell.edu NEW http://www.mail-archive.com/searchdev?q=squirrell=cayugabirds-l%40cornell.edu ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] 2014 spring update
Happy Spring everyone. Here are some updates for The Mail Archive. Search is about 10 times faster than before. This is due to a complete rewrite that shaves off a ton of initialization time. I'm really happy about this. As an experiment, we're changing the way we serve ads. Previously, direct visitors saw no ads, while visitors from global internet search engines received two ad blocks per page. As of today everybody gets the same thing, a single vertical ad to to the right of the message page. If this is a problem, please speak up here or contact our support team. As computers have gotten faster over the years, we can get away with using fewer of them. We took advantage of this and moved to a smaller location in the same datacenter. Some downtime was involved, but it should be worth it in the long run. Finally, in personal news I had some fun watching our dog play in the deep snow this spring. I laughed when he sank up to his belly and started floundering around. About ten seconds later I took a step and sank up to my armpits. Good times. Cheers, Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/options/gossip
Re: [Gossip] Temporary archive dysfunction
After investigation, this turned out to be an issue with an X-No-Archive: Yes header on the list itself. Cheers, Jeff ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] test message #1
___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] archiving suspension over
You may have noticed that archiving was suspended at The Mail Archive recently. Things are fine now, read on if you want gory details. We are hosted at a professional datacenter, complete with building wide uninterruptible power supply (UPS) and backup generators. About 5 years ago, the datacenter botched some maintenance work and accidentally cut power. In response, we deployed an individual UPS on our primary server to make things even safer. That turned out to be a mistake. On Thursday that UPS failed in the worst possible way, abruptly dropping power while a lot of data was in flight. This is normally just an annoyance, but in this case it caused enough damage to the filesystem that we had suspend archiving and switch to read-only mode. To fix things right, we express ordered and installed some additional storage. Because everything is now on large solid state drives (SSD) we can afford switching to a filesystem that is tuned more towards robustness, at the cost of some performance. For the filesystem junkies that means migrating from the fairly exotic XFS setup below to a fairly stock EXT4 setup. The new filesystem is about 20% less space efficient, but that's okay. We now have enough room for years to come. It took almost a day to fully diagnose, an overnight parts delivery, a few hours to get everything set up correctly, then 10 more hours to move all the data. We did not have to resort to restoring from backups, but they are certainly there if we need it. I'd like to emphasize that the data is safe. We were able to reconstruct everything that was in flight at the time of the outage. And while we had archiving paused, inbound mail was queuing up patiently. The system is crazy fast and we burned off the archiving backlog in just a few hours. Thanks for your patience and I hope you enjoyed this peek into what goes on behind the scenes. I think the biggest benefit of using a service like The Mail Archive is we get the fun of dealing with problems like this so you don't have to. Cheers, Jeff === mkfs.xfs -n size=16k -i attr=2 -l lazy-count=1,version=2,size=32m \ -b size=512 noatime,logbufs=8,logbsize=256k mount -onoatime,logbufs=8,logbsize=256k ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/options/gossip
[Gossip] change of mailing list name
The gossip mailing list has a new name: gossip@mail-archive.com It's been over fifteen years, but I've finally got around to moving this list to where it should have been the whole time. All subscribers have been migrated. If there are any issues, please let me know. http://www.mail-archive.com/faq.html#support Jeff PS. The motivation for finally making the change was due to the failure of a very old computer. I pulled the machine from the datacenter. First, I thought I'd extract the subscriber data directly from the disk drive, but it had the old IDE interface. Then I looked around home and realized that I also had no USB keyboard. And also no computer monitor! Amazing how technology has changed over the years. Fortunately I was able to get data out just fine from the weekly backup taken over the network. ___ Gossip mailing list https://www.mail-archive.com/gossip@mail-archive.com Unsubscribe: %(user_optionsurl)s
[Gossip] test #7
___ Gossip mailing list Gossip@mail-archive.com http://mail-archive.com/cgi-bin/mailman/listinfo/gossip
Re: Archived-At links - another non-working example
I've now had some time to spruce up the search feature. Amazing what progress Lucene has made in the last few years. Search is slightly faster now, there is less code on our end, and I found and fixed a couple of rare bugs involving HTML escaping. What I can't do is reproduce your problem; it works fine for me. We seem to be calculating the link differently. Is something unclear in the documentation? http://www.mail-archive.com/faq.html#listserver -Jeff import hashlib import base64 message_id = 51f69250.5090...@libreoffice.org list_post = disc...@de.libreoffice.org sha = hashlib.sha1(message_id) sha.update(list_post) hash = base64.urlsafe_b64encode(sha.digest()) url = http://go.mail-archive.com/%s; % hash print url http://go.mail-archive.com/GNXPfUlFDrZhnUZ4SH7cDfXZSKQ= On Tue, Jul 30, 2013 at 3:47 AM, Christian Lohmaier lohma...@googlemail.com wrote: Hi Jeff, *, we stumbled across another non-working archive-link - the targeted message is in the archive, and newer and older messages are accessible using the hash-URLs. The message in question is: http://www.mail-archive.com/discuss@de.libreoffice.org/msg15830.html and the corresponding, non-working archived-at link is http://go.mail-archive.com/c-LzIp2g0mD_PUzGjm4_-yBorXU= Contrary to the problems with the links in the past, searching by the message-ID alone returns the result 51f69250.5090...@libreoffice.org Hope you can dig up what is wrong. ciao Christian -- To unsubscribe, send mail to gossip-unsubscr...@jab.org. -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: Search returning 404
Individual list search is very important. Thank you for reporting the problem. This turned out to be a configuration mistake on the webserver, involving the MultiViews configuration directive. Search should be working now. Let us know if you see any anomalies, and in the meantime we're looking into how the mistake was introduced in order to prevent recurrence. -Jeff
Happy 2013
Happy New Year. As The Mail Archive enters its 15th year of operation, let's take a quick look back. This year we had a record uptime percentage of 99.69%. That number jumps to 99.96% if you forgive the day we were deliberately dark in protest of the proposed SOPA law in the United States. There are two reasons for the improvement. First, switching all messages over to solid state storage mid-year really helped; the disk subsystem used to lockup every few months, and that has gone away entirely. Second, internally we paid a lot more attention to this topic. Which you can see from the heckling I took during an operating system upgrade last week. (Warning: strong language.) http://www.mail-archive.com/heckle.html The message page redesign was also a lot of fun. We used the services of a professional designer, paying particular attention to recent web standards. The process even included a usability study under controlled conditions. Thanks to everyone who has provided feedback so far. Finally, as you may know, I grew up in Vermont, and was inspired by the business example set by Ben Jerry's. While we didn't give away any free ice cream, we continued our charter of donating a portion of revenue to good causes. Aside from computer-ish stuff, this year included welfare of animals, a forestry organization, and a sustainable energy initiative. Less well known is our goal of having fun. 100% of The Mail Archive staff is now trained in TIG Welding (just in case). We also got to experience flying in a zeppelin over the San Francisco Bay, which was pretty amazing. That's possibly a little more fun than our financials would really encourage, but fortunately things continue to be sustainable and healthy. I wish a happy and healthy new year to everyone, and to every byte of data. Cheers, Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: New look to Mail Archive message pages
Okay, so here's a quick status update. There are have been suggestions about fonts. Font weight. Line spacing. Typeface. Line spacing is interesting, a big design goal was to make more information available with less scrolling. However, we've found several references that suggest 1.4 is good in terms of helping readability. There is probably some personal preference; for example I personally like Courier better than Monospace. We're going to let the current design ride for a few months then revisit. The left vs. center justification of the entire page seems similarly subjective. One thing that would make a difference - more quickly - is an extremely authoritative source. For example a paper by Donald Knuth explaining how and why some other line height is better. There have been users who like to read in date order who are disappointed that we dropped some of the navigation buttons. We continue thinking, but are not sure how to make them happy without increasing confusion for everyone else. The visited link color has been changed. That's all I have for feedback so far. We continue to find and work on small bugs, like certain pages not validating, or funny looking navigation buttons in several non-English languages. If you find more of these, that is also helpful. I don't have strong data for how happy people are overall, but guessing from tone of comments, I think it is overall positive so far. -Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: New look to Mail Archive message pages
Thank you for the feedback, keep it coming. Interesting screenshots. By the way, my everyday platform is also Ubuntu (10.04 and 12.04). So far I can't reproduce font problems with courier. As for the ordering of font-family, that's a good question. Let me check with graphic designer. -Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: New look to Mail Archive message pages
What I dislike is that visited links are indistinguishable from non visited ones. The difference in color just is way too little. I didn't notice this until you mentioned it. Now it is driving me crazy. Thank you for the feedback. -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
site design
Hello all, The Mail Archive is now 14 years old (that's a long time in dog years) and we've been thinking about some design updates. The mockup below is intended for visitors from global search engines. Direct visitors will continue to have no advertisements. I hope that the proposed design is an improvement. We have already done some formal user testing, and now want to get thoughts from expert users, including folks who currently entrust data to the service. Any opinions or thoughts are appreciated, especially specific suggestions. Also don't get too excited; even if everything goes smoothly it will probably be several months from deployment. Thank you for your time. Cheers, Jeff http://www.maxdesign.com.au/jobs/318-mail-archive/38.htm -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: Archived-At links for new mails not working anymore (since June-10)
Okay, found the bug. On June 10th I made a change to deal with archives containing more than one million messages. A piece of code had decided that one million was a really big number and was starting to write scientific notation to some internal log files. Unfortunately my change was flawed and I've disrupted both the Archived-At database and sitemap entries for the last 20 days or so. I have hopefully corrected the code, so there should be good data going forward. Can you remind me which lists are actively using the Archived-At feature? -Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: arch...@mail-archive.com Slow to List New Posts?
Fair enough, I found how to have http://cr.yp.to/ezmlm.html subscribe something other than the envelope or header from address. If you think this would be useful for others, please consider sharing either here, or we can put in the http://mail-archive.com/faq.html if appropriate. I was a bit surprised to see some of the confirms go directly to the new list's page though, e.g. http://www.mail-archive.com/spiped@tarsnap.com/, rather than http://www.mail-archive.com/archive@mail-archive.com/ where some of the others appeared. The automatic sorting algorithm noticed the List-Post header pointing to spi...@tarsnap.com and decided that the message was list traffic. Sorry for the confusion. -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: Archived-At links for new mails not working anymore (since June-10)
Thanks for the note, we'll take a look. By the way, there's a discussion about using a shorter URL. Would that be useful to you? http://www.mail-archive.com/mailman-developers@python.org/msg12770.html
Re: Dark for 24 hours, starting now
The only change is all web pages are serving HTTP 503 (Service Temporarily Unavailable) for the next 24 hours. There is no disruption to email archival. We noticed at least one other email archival service (marc.info) is also participating. -Jeff
Re: Happy 2012
Finally, one of the charter goals of Mail Archive, Inc. is to have fun Mission accomplished. http://www.airshipventures.com/sightings/1290/2012/01/08 -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
100 million messages
I'm pleased to announce The Mail Archive has passed the 100 million message mark this past week. It took 13 years and eight generations of hardware, but we made it. -Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
June weirdness
June has been a weird month. Some of the automatic software programs stopped running, resulting in many search indexes falling behind. Plus we had a crash this morning causing about 90 minutes downtime. Crazy! I think I've finally traced the problem to a May 30 operating system update. It disrupted several periodic processes, including the daily operating system updater. The vendor seems to have fixed the problem, and after a manual nudge I think we're back on track. But give us some time to burn off both our archiving backlog (from today) and the search indexes (from all of June). That is a lot of data to work through, plus we're running some self checks on the storage system which makes things a little slower. Thank you for your patience and understanding. -Jeff === https://launchpad.net/ubuntu/+source/pam/0.99.7.1-5ubuntu6.4
Re: search failure
On Sun, Jun 12, 2011 at 3:11 PM, e-letter inp...@gmail.com wrote: Unable to use the search feature, e.g. search for text or a returns 0 results, clearly incorrect. Works for me, from both the home page and in an individual archive. Can you please supply the exact search URL that is giving trouble? http://www.mail-archive.com/search?q=textl=gossip%40jab.org http://www.mail-archive.com/find.php?cx=partner-pub-7266757337600734%3A2228981072cof=FORID%3A9ie=UTF-8q=textsa=Search -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: Re: [libreoffice-website] Archived-At: links not working since May 31th?
I assume the indexing is not running at their end for some reason. That is exactly correct. If I run by hand it works fine, and that go link will resolve now. Still looking into why the cron job that kicks off indexing had trouble. Thanks for problem report. -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: What happened to search? (only google custom search with crippled possibilities?)
Perhaps it's time for us to revisit [Lucene for site-wide search] A few weeks ago I tested Lucene's ability to search across multiple indexes (MultiSearcher) and it is hopelessly slow; queries take 5 seconds across just a few hundred indexes. Right now I'm trying index merging (addIndexesNoOptimize). With an HDD destination there is a load spike (around 20 to 30) and merge at a rate of about 2GB per minute. That's approximately 3 hours. With an SSD destination the load spike is gone and merge rate jumps 3GB minute. It remains to be seen what the search performance will be once indexing is complete. There is also chatter that Lucene 3 has a slightly different API for index merging, but if I'm reading it right there is no performance difference. -Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: What happened to search? (only google custom search with crippled possibilities?)
I was doing some experiments today, and managed to briefly knock over a server in the process. I looked at searching multiple indexes (Lucene's MultiSearcher) and merging (IndexWriter.addIndexesNoOptimize). The former is unusably slow. The latter seems to be on track for about 6 hours if the destination is HDD. About 4 if destination is SDD. When it finishes, we'll see what query speed looks like. -Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: What happened to search? (only google custom search with crippled possibilities?)
Oops, sorry for the more-or-less duplicate message. The extra factor of 2 in time is because the temporary files are turning out twice as big as I was expecting. Earl, good suggestion, and no we haven't explored it (yet). -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: List not appearing (from a google apps 'group')
Progress report: We are receiving your messages but our system is failing to recognize them as belonging to a new list. It took me a while to find them since gg1 is not logged the same way as regular inbound messages. I will take the sorting engine out back into the parking lot and try to talk some sense into it. Then we will reprocess a bunch of data. Stay tuned, but my prediction is results won't be visible until next weekend. Again, thanks for the problem report. -Jeff Note to self -- example message is numbered 267742. -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: List not appearing (from a google apps 'group')
Fixed faster than expected. Only a few hundred messages affected across the entire corpus as far as I can tell, and they are all being dealt with. One question: the gg1 address is really designed for Google Groups, not for other stuff. Did you try the regular archival address first and get some sort of error? http://www.mail-archive.com/techadvisorypanel@swordapp.org/ -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
season's greetings
Season's greetings. Thank you all for sticking with The Mail Archive as we close out the decade. Here's a quick rundown of the trials, tribulations, and triumphs over the past year. First, let's talk about infrastructure. Our uptime was 99.57% which is similar to previous years. This year's main problem turned out to be the 10K searches per day; sometimes these would clump together and overload computers. Took a while to figure out, and was addressed in November with significant algorithm changes plus moving all search data to solid state storage. In 2010 we continued the longstanding trend of no data loss. Pages continue to serve from rotating rust, specifically traditional disk drives with 8X data redundancy to help minimize latency. I don't know how much to trust third party statistics, but Alexa claims we are faster than 95% of the world's web sites. The corpus grew by 25% this year; not bad for a data set started last millennium. And somebody finally used that nifty embed an archival link in the message feature, working out a lot of kinks in the process. For a system designed to run entirely on autopilot, there's a lot of day to day upkeep. Our indefatigable support team responded to almost 600 inquiries during 2010. Wait, who am I kidding? Some of these were exhausting. There have been interoperability challenges with a couple of list service providers. YahooGroups has been particularly evil, they managed to break interoperability with The Mail Archive and we're still not sure if it was deliberate or incompetence. If your list is affected, consider changing service providers or contact our support team for the current workaround. What else? If you enjoyed the multiweek advertisement holiday in November, sorry that's history. We provided a form based interface for advanced search (but I'm sure you all use the advanced command syntax) including sort by date. Total donation dollars remained on par with previous years, but concentrated on better support for fewer entities. None of them computer related for a change. And in personal news, I've found that wearing an extra ring on my hand doesn't slow down typing speed at all. But it very happily means a little more time away from the keyboard. http://traffic.alexa.com/graph?w=400h=220r=3mu=mail-archive.comu=mit.eduu=slashdot.org
Re: mail-archive search function
On Wed, Dec 22, 2010 at 4:46 AM, e-letter inp...@gmail.com wrote: Please offer the ability to search using international date format (-mm-dd) as a search criterion Done. (Actually, this has always worked).
Re: Urls containing @ screwed up in archived Mails
The interesting question: is it possible? Are the originating mails stored so that the visible html can be repaired? A mathematician, a physicist, and an engineer walk into a bar. The mathematician says The raw mail exists even the old stuff is in offline cold storage. It can be matched by message-id against the HTML files and therefore it is possible. Q.E.D. Next problem, please? . The physicist says, I bet some of the decade old raw mail will be hard to find. And regeneration with URL preservation will take some serious hacking. Never been done before, tricky, and will require experimentation. But that's what we live for. Give me a small team of bright people and we can solve this problem in six months. The engineer says Fredrich, do you have a specific URL in mind that you really want fixed? If so, I'll go spend 30 seconds and fix it by hand. The bartender looks at the three sorry customers and says, Guys. Guys. Your mailto: links are still broken.
Re: Urls containing @ screwed up in archived Mails
The Mail Archive does have to be very aggressive to obfuscate email
addresses, otherwise a lot of people go bonkers. But yes, it is dumb to
break a hyperlink, especially a hyperlink to The Mail Archive. Your feature
request is valid and if you are feeling eager, feel free to send in a patch.
Otherwise we'll get to it when inspiration strikes (and I have to warn you,
inspiration can be rather slow to strike sometimes...).
http://www.mhonarc.org/MHonArc/doc/resources/addressmodifycode.html
AddressModifyCode
$orig_address = $_;
$address = lc($orig_address);
if ($ENV{'MAILLIST'} eq $address) {
# If it's the list address, leave it alone.
$final_address = $orig_address;
} else {
# Otherwise, conceal the address. (Choose ONE option below.)
$address =~ s/(.).{0,3}(@.*)/$1\.\.\.$2/; # usern...@domain.com
$final_address = $address;
}
$_ = $final_address;
/AddressModifyCode
Re: Problem with search by message-id / Archived-At hashes
You found a bug. In The Mail Archive's hash calculation, there is an incorrect urlib.unquote run on the message id. This is escaping the minus sign, and therefore calculating based on an incorrect message id. We're going to have to regenerate the entire message-id index after the bug is fixed. That will take a while, but we will prioritize recent data. aanlkti=uvu1a-ddef0x07i0s4fmd+c5=sndgk2uns...@mail.gmail.com AANLkTi=uVU1A#45;ddef0x07i0s4fmd+c5=sndgk2uns...@mail.gmail.com This doesn't affect you, but there is also a secondary bug where messages archived in the last few minutes of the day (GMT -0800) are not having their message id indexed. It's a very small fraction of the messages, and I am just mentioning it here so I don't forget to fix it at the same time. Stay tuned for the re-index (probably this coming weekend) and thank you for the excellent bug report. It goes without saying, but I'll say it anyway, do not write code to work around this flaw. The fix must be on our end. Jeff
Re: Problem with search by message-id / Archived-At hashes
The gossip malling list uses a somewhat obscure and very limited list server called Enemies of Carlotta. Another fun fact is it runs on a 5 watt NSLU2, which has a grand total of 32 megabytes of memory. That's less memory than the very first hardware iteration of The Mail Archive, which was a 90 megahertz Pentium pumped up with 80 megabytes of SDRAM back in 1998. But I digress. We'll switch to gossip over to Mailman 3 once it is ready. As for the Archived-At: header, this is great news! You may be the first entity actually using the feature for real, so I'm delighted and thanks for the bug report. We'll take a look and see what is wrong as soon as practical, which is probably this coming weekend. Jeff
degraded search on home page
If you visit the home page on The Mail Archive, you may notice search is broken. You can not currently search the entire corpus. We're working on it but it will take some time. The other search features still work, e.g. search works fine for an individual archive, and you can still search list names from the homepage. Jeff
Special test message - please ignore
Testing obfuscation, data below: j...@jab.org http://j...@jab.org mailto:j...@jab.org http://www.mail-archive.com/gossip@jab.org/msg01358.html http://www.mail-archive.com/gossip@jab.org/ http://www.mail-archive.com/gossip@jab.org http://mail-archive.com/gossip@jab.org/msg01358.html http://mail-archive.com/gossip@jab.org/ http://mail-archive.com/gossip@jab.org http://www.mail-archive.com/talk-it@openstreetmap.org/msg19258.html http://www.mail-archive.com/talk-it@openstreetmap.org/ http://www.mail-archive.com/talk-it@openstreetmap.org http://mail-archive.com/talk-it@openstreetmap.org/msg19258.html http://mail-archive.com/talk-it@openstreetmap.org/ http://mail-archive.com/talk-it@openstreetmap.org http://mail-archive.com/talk-it%40openstreetmap.org -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: Advanced search...
I've finally completed localization of advanced search. If you speak a language other than English, now is a great time to click around the user interface and see if there are any silly language mistakes. (To get to advanced search, do a regular search first, then you'll see a link) Cheers, Jeff http://www.mail-archive.com/faq.html#i18n
Advanced search...
This is general interest, so we are responding publicly. We've discovered that many of the advanced query results have been leaking out through a fractured fiber optic line in the Gulf of Mexico. It is hard to get a precise measurement, but we believe 13 to 20 thousand bits of information per day are entering the water. We are attempting to siphon some of the data into entropy tanks, but may have to make a second network connection to re-route the data entirely. Okay, I just made that up. But without naming any names, it is true that one of us actually worked in the petroleum industry a long long time ago. Thanks for pointing out the flaw. It's embarrassing, we missed it, and to my surprise you were the first to report. Only 16 advanced queries actually tried looking past the first page of results yesterday, that might explain part of it. As for the feature requests, priority is on localization; we've only made a small dent on French, Indonesian, and Portuguese; that means there are over 20 languages left to go. Goal is to complete localization by end of quarter. The other requests sound reasonable to me, but will probably take a back seat for the time being. It will take a little work on the user interface side to let folks start with advanced search; if we just stuck in the link without some massaging users would see a confusing 0 results page before doing their initial query. As for sorting by name or other fields, my only complaint is the extra work for localization. But it isn't that crazy, and we'll put it on the drawing board. If we do implement it, the other Jeff would probably be the one to do it. Be forewarned, he's very tied up for at least a month, but we'll definitely discuss. Thanks again for the feedback. -Jeff -- Forwarded message -- From: M. G. Devour mdev...@eskimo.com Date: Sat, Jun 5, 2010 at 3:04 PM Subject: Advanced search... To: Mail Archive Support themailarch...@gmail.com Hi Jeff and Jeff, The new advanced search form is an interesting development. It makes creating author, subject, and time-based searches much easier. Two things: If the advanced search yields multiple pages of hits, anything past the first page returns 0 results. A member was making happy noises about the great new search facility, until he noticed this problem! grin Second, would it be too burdensome to just put a link somewhere near the search box for 'advanced search' instead of burying it two pages deep? If I know the terms of a complex search I want to make, I'm forced to do what amounts to a throw away search just to get to a page with a link to the advanced search form. Lastly, on a slightly different tack, how about making it possible to sort search results by various values, e.g., date, author, subject, etc.? It sure would make it easier to locate just the right post among a lot of returns. Otherwise, the archives have been reliable and are getting good reviews from those who use them. Thank you for your continued efforts! Be well, Mike D. owner, silver-l...@eskimo.com [Mike Devour, Citizen, Patriot, Libertarian] [mdev...@eskimo.com] [Speaking only for myself... ] -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: [Gossip] search experiment, feedback requested
Ok, it only took ... 4 years ... but we now have sort-by-date available in the advanced search interface. Enjoy. -Jeff I wasn't clear. Is there some way to organize the search results? When I used the trial search engine on the sundial list and typed in oglesby the 550 results were all accessible (a VERY GOOD thing), but were listed in seemingly random order. Is there some way to order the results by date? And/or in other ways? -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
Re: Improvements
Hi Andrius, I wasn't aware that there were adult content ads being served at all. Can you please supply a URL so I can take a look? If you don't want to supply to the entire group, send to M-A staff at themailarch...@gmail.com. I'm not keen on adjusting aspects of individual messages because that is labor intensive, but we will look at problem cases and possibly make global adjustments when appropriate. Especially if any of this is spam-related. Jeff On Fri, Feb 19, 2010 at 5:04 AM, Andrius Kurtinaitis andrius.kurtinai...@mif.vu.lt wrote: Hello, it would be nice if I could turn off ads on certain messages. Some of them attract sex-related google ads. They make the authors of these messages feel very uncomfortable seeing their content used for such advertising. Kind regards Andrius Kurtinaitis
Re: Search syntax...
An advanced search form would basically have a bunch of fields like subject and date and from then string them together into a query syntax described below. Then redirect that query to the standard search URL. Implementation would most likely be in PHP. Hard part isn't the programming, it is designing an elegant suer friendly form that matches the query syntax. Actually that isn't so hard either. Hard part is making time to get it done. http://www.mail-archive.com/faq.html#search .
Re: Improvements
The list name links to the info page; optionally not displayed if made
redundant by the nature of the logo.
This one requires too much per-list thinking; we'll only consider changes
that are fully automatic. The other parts sound reasonable to me, but it
would be nice to know if other people have the same preference. Logos are
done with a tiny bit of CSS., just in case some impatient web programmer
wants to help out.
a#mail-archive-logo { float: left; top: 2px; left: 2px;
margin: 0px; padding: 0px;
background-image: url(/silver-l...@eskimo.com/logo.png); background-repeat:
no-repeat;
width: 273px; height: 50px; }
#listlink { float: right; margin: 0 75px 0 0; }
Re: Header for list web site?
I have a web site for the list rules and, very soon, a link to the archives. Is there a header I can add to my messages that will let the archiver pick up and display my site URL on its own line? We'll honor RFC2369 headers. A quick glance suggests List-Help is most appropriate. Also, is themailarchive @ gmail.com the right address to send my request to turn off the Reply button and ask about or send my mbox archives to? Yes, customer service type stuff not of general interest should go there.
Re: A few pre-purchase questions...
Bring it on. The Mail Archive gets tens of thousands of inbound messages daily, and currently serves on the order of queries per second. So I don't think there is any concern about swamping the service given the numbers mentioned. Doesn't matter to use how many mbox files are involved for imports. You are pretty much right about control; we ask list admins to be the primary contact with M-A staff and they may request deletion of a message or their entire archive at any time. As far as the logos go, uploads are reviewed by M-A staff before going live to prevent logo spam. So far that has been a sufficient level of review and I'm not aware of any logo problems - at least nobody has complained so far. Language localization is handled the same way, and that's pretty much the extent of archive customization. Finally, no problem if you want to create your own index. Thanks for asking and we hope to, um, win your business. Cheers, Jeff
Re: Improvements
Hi Randy, Bad news first.It is not such a great idea for mail-archive.com to re-send mail. That's asking for trouble with respect to abuse and spammers. On the good side, we can get the same effect if the mail server is in cahoots with the archiving service. A specification (RFC5604) is in place and implementation is done on our side. The hard and slow part is cajoling list server software to participate. What list server software do you use? http://www.mail-archive.com/faq.html#listserver Jeff
Re: Welcome to gossip@jab.org
Ok, I checked; the message was dropped because it was bigger than our size limit. We limit inbound message size for a variety of reasons; one is historically attachment heavy archives consume a lot of resources and - statistically speaking - are much more likely to be spammy. (There were some horrific abuse cases involving YahooGroups around the turn of the century). Unfortunately your message was collateral damage, which I regret and apologize for. In addition the FAQ entry is poorly worded and misleading, so I will update it. http://www.mail-archive.com/faq.html#attachments Bottom line - mystery solved but problem still exists. I am not inclined to open up floodgates on attachments or large mails for the archiving service at this point, but am willing to listen to feedback.
Re: Access forbidden?
Thanks for the problem report, Joseph. Looks like a problem with the RAID filesystem after a power outage at the datacenter. I'm running some integrity checks, and this is going to take a while. In the meantime, I've mounted backup disk from two days ago. This is going to be a little stale (the data is two days old) and because it is just one disk, we aren't going to be able to handle the enormous amount of traffic M-A receives. So there will be quite a few web requests timing out. In summary, the website is going to limp along for a little bit until we get the high throughput storage system back online. Let me know if you have questions. Cheers, Jeff -- To unsubscribe, send mail to gossip-unsubscr...@jab.org.
full site search
Ok, that one got away from me. Let's try this again. I've enabled an experimental full site search. It's definitely not ready for prime time; too slow and we aren't going to update the index regularly. And it might go away at any time. But if you want to play around, have fun. We don't really have a good use case for full site search and it takes some work to do it well. So speak up if you think this is worth pursuing. http://www.mail-archive.com/search?l=all -- To unsubscribe, send mail to [EMAIL PROTECTED]
Re: Slight change in address for my mailing list
Hi Terrence, The domain for the mailing list is lists.metaperl.com, not metaperl.com. Sure, I think we can help you out. Let's move this to the customer service address instead of using gossip. By the way, you've got an X-No-Archive: yes header in your new setup, which is preventing archiving. Please take care of that first. Cheers, Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
new feature: hints
Remember the info pages? http://www.mail-archive.com/petbunny%40lsv.uky.edu/info.html If you squint carefully, you can see a new field called hints. So what's a hint? Hints are gentle way to tell the world what a list is about. Let's say you have a mailing list about pet bunnies. Then you add a hint called pet bunnies. This will make it a little easier for pet bunny lovers to find the archive. It will also help target any advertisements for things like carrots instead of linux kernel hacking jobs. And maybe other good uses in the future. Hints are optional, but please try it out if you get a chance. A good hint is one that you could type into a global search engine, and would be happy to see the list archive as a result. So pet bunnies is a good hint, Mr. Snuggles is having a pwarty! is not so good. By the way, you now know my favorite list archive of all time. I currently have the hints set to be world editable (except for a few I've personally set by hand) and we'll see how that works out. If there is a spam problem we'll do something different. Have fun, Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
change of pace
As you may know, The Mail Archive has been an advertising supported service for several years now. We have recently decided to shift things around a bit. We are completely removing advertising for regulars. This includes list admins, list members and lurkers - anyone that uses The Mail Archive directly. We'll keep and experiment with advertising only on pages viewed by fly by visitors that show up from off-site searches. We think this will improve the experience for all of you, and for all of your mailing list's readers. Note that this means you should be seeing The Mail Archive ad-free automatically now. The idea is to enhance usability for people more deeply involved with lists, with the costs covered by those remaining folks who likely don't care as much. Are people generally happy with the idea? Any comments or questions? Cheers, Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
Re: request for help: indonesian
On 4/10/07, Ronny Haryanto [EMAIL PROTECTED] wrote: [...] Everything else looks OK to me (I speak Indonesian natively) except [...] Thank you, your feedback is integrated and will be active for all new messages. I don't know if we can easily localize the dates (the M-A search engine currently parses them) but thank for mentioning it. We'll think about it. According to Alexa (which might have bogus numbers) the service is really popular in Indonesia, so you just had a big impact. If you notice anything else in the future, please let me know. http://www.alexa.com/site/ds/top_sites?cc=IDts_mode=country PS. While Indonesian has my attention at the moment, wording suggestions are highly appreciated for any of the supported languages. I wouldn't be surprised if many of the word choices are a little funny. -- To unsubscribe, send mail to [EMAIL PROTECTED]
request for help: indonesian
The Mail Archive is getting a lot of visitors from Indonesia. Can someone who speaks Indonesian please take a look at the localization and tell me if looks ok? Should any wording be improved or replaced? Example archive: http://www.mail-archive.com/origami_indonesia%40yahoogroups.com/ http://www.mail-archive.com/origami_indonesia%40yahoogroups.com/msg00275.html Thanks! Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
ebb and flow
Jeff Marshall recently pointed out that archiving latencies had been rising again - to at least several hours and occasionally the good part of a day. So I pulled out the programming hammer and stated whacking off milliseconds here and there - eventually it starts to add up. I'd guess I probably convinced the system to process about 30% faster overall. It's hard to measure because processing gracefully switches towards a more efficient batch mode when dealing with large backlogs. Today I saw the backlog almost clear - not quite but close. It was witin 40 messages at one point. That would have been fun as the backlog hasn't hit zero for about 6 million messages. Maybe we'll get there next week, or maybe traffic will grow again and take back these gains. In any case, expect lower (but not zero) archiving latencies at least for a while.And I'm happy to not fret about replacing hardware for a change. Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
uptime report
The Mail Archive uses a third party to monitor availability. so if a computer goes offline someone usually hears about it pretty quickly. As a side effect, we also get weekly, monthly, quarterly and yearly uptime reports. The 2006 numbers just came in, and I'm pleased to report M-A significantly outperformed ivory soap. Ivory soap advertises being 99 44/100 % pure and M-A's uptime percentage handily beat that number. Hurray! Maybe for 2007 we'll try outperforming the purity of fine silver. Happy new year. Jeff http://en.wikipedia.org/wiki/Ivory_(soap) http://en.wikipedia.org/wiki/Sterling_silver -- To unsubscribe, send mail to [EMAIL PROTECTED]
Re: Moving a list with outdated list:post header possible? ([EMAIL PROTECTED] to dev@website.openoffice.org)
On Sat, Dec 16, 2006 at 02:32:18PM -0500, Jeff Breidenbach wrote: It is possible to move a migrate a list archive, but it is kind of manual so you have to ask really nicely. Good news! The process just became semi-automated, so there should be less risk of screwups. And it is no longer strictly necessary to bribe M-A system administrators with freshly baked chocolate chip cookies. Cheers, Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
counter gymnastics
The counter on the homepage for total number of messages (which is somewhere around 40 million) is going to be a little erratic over the next week or so. No cause for alarm, just taking advantage of the quiet holiday season and moving a bunch of mail between various disks. Jeff -- To unsubscribe, send mail to [EMAIL PROTECTED]
Re: Moving a list with outdated list:post header possible? ([EMAIL PROTECTED] to dev@website.openoffice.org)
You are right, list-post tends to be the dominant header for the M-A sorting engine. It is possible to move a migrate a list archive, but it is kind of manual so you have to ask really nicely. Basically, what we do is move the archive to the new location, then insert a HTTP redirect at the old location. This keeps the gloabal search engines from getting too confused. This is done shortly after the list has switched around the headers and started the new archive. The messages received during the time gap are either discarded or poured back into M-A inbox for re-archiving. In summary, make the switch, confirm the new archive, then drop an email to our support address (address is buried in the FAQ somewhere). I'll do the rest. Jeff PS. Most lists just split their archives and deal with it. :) -- To unsubscribe, send mail to [EMAIL PROTECTED]
[Gossip] mailman import question
Does anyone have a mailman archive to mbox converter script in their back pocket? And when I say mailman archive, I'm talking about the gzip'd text like this: http://listas.asteriskbrasil.org/pipermail/asteriskbrasil/ Note the lack of headers - ugh. I have no idea what the mailman folks were thinking, but this is really inconvenient - and makes it very hard for folks who want to import their list data into M-A. Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] Re: mailman import question
I've included the simple script I use below. Thanks, Lars. I don't see the script - could you please resend? Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] Search function not working
Hi Chris, Thanks for the report, I see what the problem is. Can you wait about 24-36 hours? This problem is affecting a number of active lists (beginning with 'g' 'h' 'i' 'j' 'k' and 'l') and requires a fair amount of computer processing time to fix. Cheers, Jeff On 9/21/06, Chris McFarling [EMAIL PROTECTED] wrote: I'm attempting to search the imail_forum list but zero results are being returned for all search queries. A simple search query with just the word phrase should return hundreds if not thousands of results but I'm getting nothing. Chris ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] late night traffic jams
Around 9:20pm PST, I rebooted the primary webserver. Pretty minor reason - just adding a boot parameter to work around linux kernel bug #7068. Unfortunately, things didn't go as planned and I ended driving out to the data center and spending about an hour of quality time there. Add in driving time, and the service was down for about an hour and half tonight. By down I mean the website was inaccessible. As usual, all inbound mail was queued during the downtime and things are going just swimmingly now. There is a decent sized backlog, but that's mainly due all the CPU and disk we've diverted to search experiments recently. So archiving will be a little slow over the next day or so, but probably not ridiculously so. Anyway, that's the scoop. This particular problem won't happen again. That just means next time it will be a different one, but it really has been a while. I'd almost forgotten what the inside of the datacenter looks like. :) Anyway, my apologies. Cheers, Jeff PS. Would you believe there was a traffic jam at this time of night? Due to road construction, but still... ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] Error message while searching.
Fixed. This is a good time to mention we switched mainline search over to PyLucene yesterday afternoon. Both search engine contenders are quite excellent; we could happily go with either one. PyLucene has advantages in UTF-8 maturity and more efficient disk usage. Xapian has advantages of better upstream support and is better maintainability from a system administration perspective. For management types who like colorful spreadsheets: http://spreadsheets.google.com/ccc?key=pKHp5ItRZ0SUL0PKhN_ssfA Anyway, we had to do something as it really was time to retire HtDig 3.1 and Xapian was still rebuilding. :) We'll see how this works out in the short term and nothing prevents re-evaluation. There will undoubtedly be minor glitches to iron out, for example II still need to put in the help link I promised Mac. But search on most lists should be working really well. Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] search library battle royale
Lucene: it slices, it dices Right. That's just what I'd expect from a program whose name sounds like a brand of cheese. During the long cold war, children were inspired by Superman. In late 2003, Californians elected The Terminator to be their governor. These turbulent times demand strong, heroic sounding software. We need Xapian Omega. I am talking about a C++ thoroughbred champion. Easy to patch, trivial to apt-get install, Xapian is an administrative dream. The unconvoluted build against a 100% Free Software chain means Xapian is going to be less work, less fiddling. And therefore it may be less likely to break. When was the last time you searched Unicode characters Ok, ok, you score some serious points here, especially on Asian languages, but Xapian holds its own on European languages. I think. Check out some Brazilian Portuguese action below. And I hear the Xapian team is working hard on full UTF-8 support. http://www.mail-archive.com/cgi-bin/omega/omega?P=+jo%C3%A3oDB=flex-brasil%40yahoogrupos.com.br I hate to double team you, but how about the ability to search across two related lists at the same time? What, did I say two? How about searching across 181 related lists at the same time? Ha, ha, ha, in your face puny PyLucene! http://www.mail-archive.com/cgi-bin/omega/omega?DB=debianP=banjos But enough blustering from us - how about some comments from M-A users? Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] Re: [Team] search library battle royale [draft]
If you don't care about search, don't read further. === Sunday, Sunday, SUNDAY! Come see the data crunching, webpage hopping, free-styling search library action. Two monster libraries, titans of Free Software technology, compete to become the native search engine for The Mail Archive. Watch as Xapian Omega crushes and destroys the competition, finishing off queries in milliseconds. This probabilistic juggernaut is a battle tested, email chewing reigning champion in Europe. Honed for years and more hardened than quartz, Jeff Breidenbach will drive Xapian Omega during this Battle Royale. PyLucene is a mild mannered garbage collectin' programming library just like your mom's search index. That is, if your mom's search index could jump partitions, crush gigabytes down to tiny sements, and plow through millions of records. Forged on the anvil of a Xerox PARC alumni, brimming with black magic, Lucene will be wrought by the indomitable Jeff Marshall. We're taking these two byte belching, buffer oversized, monster libraries and pitting them head to head. Old geezer HtDig 3.1 will also make a final appearance in the arena. All three engines can run on any list, just by replacing gossip@jab.org with the listname of your choice. Who will win the monster rally? Xapian vs Lucene? Jeff vs Jeff? Yes, you decide! Send comments to gossip, or privately if you are shy, for the next week or so. Bonus points for using phrases like like slamming! spectacular or crushed like a bug. Who's got the slickest user interface? Which contender has superior data-crunching performance? How about grits, determination and the baddest sounding name? Want to see something tweaked? Have questions? Ask and it will be done if humanly possible - this is a gritty bit-for-bit battle of hotrod software and programmer ingenuity no holds barred. Ladies and gentlemen... Start your search engines! HtDig3.1 http://www.mail-archive.com/cgi-bin/htsearch?config=gossip_jab_orgwords=magically Xapian Omega http://www.mail-archive.com/cgi-bin/omega/omega?P=magicallyDB=gossip@jab.org PyLucene http://www.mail-archive.com/lucene/search.py?list=gossip@jab.org=magically ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] tidying up mbox files
Thanks! Also, one of the people with slightly-broken mbox files suggested this: perl -i -p -e '/^From / !/\d\d:\d\d:\d\d \d\d\d\d$/ s/(.+)/$1/' A_* I'm continuously amazed at both Perl, and the people whose brains are capable of understanding it. :) -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] tidying up mbox files
Hi all, When someone wants to import a bunch of messages into an archive, the provide an mbox file. The mbox file format is simple, but has at least one gotcha. In order to avoid misinterpretation of lines in message bodies which begin with the four characters From, followed by a space character, the mail delivery agent must quote any occurrence of From at the start of a body line. The majority of mbox files I've been handed do not escape From like they should, and this causes problems on M-A's end; inc from the nmh suite gets unhappy and starts trashing messages. Are there any recommendations for an mbox2mbox converter that will clean up these wayward almost-but-not-quite-mbox files? Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] Search in Mail Archive
Thanks for the problem report. Even the manual override for search indexing is acting really sluggish right now; there may be something amiss. I'm looking into it. Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] rdf vs. rss
On 19 Jul 2006 11:46:45 -0700, Jeff Marshall [EMAIL PROTECTED] wrote: I received a few private votes for keeping the .xml rss 2.0 feed, if we eliminate one of them. I think we should deprecate the RDF variant (e.g. remove it from the FAQ) then actually remove the feature once we're sure nobody is using it. Or in a year or two, whichever comes first. ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] Problem with spambot detection
Whatever happened with this? Did it get resolved? Does anyone else have this problem? On 19 Jun 2006 21:06:58 -0700, Jeff Marshall [EMAIL PROTECTED] wrote: Yikes, that shouldn't be happening at all for anything other than well-known spambot user agents. I'll look into the stats on how often this is coming up. I may follow up with you for some quick debugging help. Thanks. Jeff Marshall [EMAIL PROTECTED] -Original Message- From: Ronald Nissley Date: 6/19/06 6:14 pm To: gossip@jab.org Subj: [Gossip] Problem with spambot detection I frequently see the following when viewing (IE6 SP2) mailing list archives. It's very annoying! Spambot detected. Hi, our server has decided that you are, in fact, not a human being, but rather a sinister computer program designed to scoop up email addresses and bombard them with unsolicited advertisements. Since we find that kind of conduct reprehensible, we have decided to deny you access to our site. If we made a mistake, and you really are a human being, we apologize profusely. It just happens that your browser looks VERY similar to a known spambot. Try using a different browser and you should have no problem connecting to the site. Have a good day (and die, spambots, die) ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] slight tweak to message page layout
Hi all, I've adjusted message page layout slightly; there is now a little less artwork in the right margin. Should we keep it that way? Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] Re: change to font and formatting ??
Hmmm... I took a look at the formatting problem Marcus was talking about, which is a text reflow problem; shows up when the browser window is relatively narrow. As an experiment, I bumped the CSS entry for msgBody from width:60% to width:75% Does this break anyone? Look in particular on message pages for reflow problems, overlapping text, or the ad getting pushed into a weird position; especially for Internet Explorer with a narrow window. If no serious problems are reported, we'll make the change permanent. The reason to hem in the msgBody block is to keep it from screwing up the flow (msg on left, ad on right). Since the ad is a fixed width in pixels, what I really want is 100% - 160px, but that's not supported even in CSS2. I just checked the spec. -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] Re: green-travel resources {webliographies}
Marcus (and everyone), I want to publicly apologize for mischaracterizing the Green Travel list. Basically, I jumped to an incorrect conclusion from reading a couple of messages. The list and archive are legit. Mea culpa. Cheers, Jeff Breidenbach ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] planned downtime
I'm going to take down www.mail-archive.com in a little bit to add another small boatload of hard drives. Inbound mail will be queued. Hopefully this won't take long. Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] planned downtime
Maintenance complete; the service was offline for about half an hour. All right! The primary server is now has 3.5TB raw disk onboard, which should be enough for anyone. -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] changes this week
Hi all. Over the last week or so we * Experimented with an automatic language detection tool, and localized nearly a thousand lists. * Bumped up the setting one notch The Mail Archive's inbound spam filter. * Noticed not too many people have submitted custom logos. Maybe nobody cares? As always, please yell if you notice any problems. Thanks. -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] instant gratification, and I mean instant
The winning new feature from yesterday's coding session was... -- Custom Logos -- http://www.mail-archive.com/faq.html#customize Why logos? We're hoping that list admins will have extra warm fuzzy feelings seeing their archive customized with a particular image. Heck, maybe this will be the tipping point for some organizations running hundreds of public mailing lists. Also, when we had a slightly harder language localization feature, nobody used it. Literally nobody. Once that got moved to a simple control on the info page, hundreds of archives took advantage of the feature. Every single day, more localization requests come in. I was really shocked - I had no idea that user interface was the bottleneck for this feature. We hope logo customization will have a similar story. In other good news, Jeff I are still talking to each other. Thanks to jaf19 for his public suggestion, and others who sent in suggestions via private email. There's a good chance we'll schedule another coding jam session in the future. It almost (but not quite!) feels like performance art. -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] Re: Curious archive threading
Mac Oglesby noticed an occasional hiccup in message threading in one of the archives. I've added a FAQ entry explaining how this can occur, and with Mac's blessing I am forwarding our discussion to gossip. Cheers, Jeff -- Forwarded message -- From: Mac Oglesby Date: Nov 17, 2005 6:15 AM Subject: Re: Curious archive threading Hi Jeff, Thanks for your efforts. I had suspected that a possible cause of the strange threading might be some action taken by the message author. If fact, I had written to a few of the message originators asking them to recall if they had done anything which might possibly have involved a message outside of the main thread. In two of three cases there was no reply. The one who did respond seemed a bit annoyed at the question (perhaps thinking I was was accusing him of wrong-doing) and said he created his message in the usual way, whatever that means. Glancing back over the thread index of the Sundial Mailing List, I see numerous examples of curious threading, and in each case it would be easy to imagine that a message author had used Reply in a manner which might confuse the threading process. At a guess, I don't believe a large percentage of those who use email regularly know anything about the message ID system. Since normally only a small part of an email message is displayed for reading, most users have no idea of how large the entire message package really is, and how much information is there besides the plain text. Until recently, that was my condition also. Thanks for the courtesy of asking about posting this on Gossip. I have no problem with that, as there are no personal disclosures involved. A FAQ entry would be a good idea, IMO. Best wishes, Mac Ok, I took a look: Message #11372 is the one about the Mars Sundial. Inside the email message, it has the following header that gives it a unique identification - which is use to distinguish this email from all other emails in the world, past, present, and future. Message-Id: [EMAIL PROTECTED] Message #11373 is about helichronometers. It refers to the #11372 using a the References: header. The references header is placed by the Mail User Agent (in this case, Outlook Express), typically when someone hits Reply References: [EMAIL PROTECTED] What almost certainly happened is the author of #11373 hit reply to message #11372, then manually changed the subject line to match some other conversation. The threading subsystem had to choose whether to go with the references embedded in the message headers, or to track the subject line. It went with the references because that is usually more reliable. Essentially the system tracked who hit Reply to what in this case. Bottom line: This problem will happen whenever someone hits reply to one thread, then manually changes the subject line to match another. Most threading systems (including those built into Mail User Agents) will probably make the same mistake when this happens. I don't think it makes sense to change anything in The Mail Archive's algorithms, but it may make sense to add a FAQ entry. Let me know if this clarifies the sitaution. Also, I would like to CC this message to gossip if you are ok with that, as it is of general interest. -Jeff === Hi Jeff, When the Sundial Mailing List archive is viewed by thread, these 6 messages (URLs below) seem to be organized in an unexpected manner. Except for my message about the Mars sundial (#11372), they all deal with Southern hemisphere heliochronometers. Can you help me understand this curious threading? Thanks, Mac http://www.mail-archive.com/sundial%40rrz.uni-koeln.de/msg11374.html http://www.mail-archive.com/sundial%40rrz.uni-koeln.de/msg11375.html http://www.mail-archive.com/sundial%40rrz.uni-koeln.de/msg11377.html http://www.mail-archive.com/sundial%40rrz.uni-koeln.de/msg11372.html http://www.mail-archive.com/sundial%40rrz.uni-koeln.de/msg11373.html http://www.mail-archive.com/sundial%40rrz.uni-koeln.de/msg11376.html ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] Re: Mail-Archive Source - Policy changed?
Reply is to gossip due to topic being of general interest. I was trying to download the latest version of the mail-me GPL tarball from the /contrib directory but I found nothing there. Was the source code removed intentionally, or is it a glitch? Wow. I had no idea anyone had even looked at the software after all these years. I hope you put it to good use. The tarball in question contains the sort engine, a bunch of glue code and all The Mail Archive's branding material (i.e. the home page, the FAQ, logo, stylesheet, etc.) It came down accidentally some time during the past two years when we were moving things around and nobody seemed to notice. Then an ugly plagiarism incident [1] put a sour taste in both Jeff Marshall's and my mouth. We are now particularly wary of putting branding information in a GPL tarball lest something like that happens again. Maybe with the right packaging the code could be made into an appropriate add-water-get-archival-service but that would take a whole lot of elbow grease. In the mean time we've been pointing any inquiries along those lines to mharc. [2] Jeff and I need to talk this over and decide what we want to do. Comments from the peanut gallery are - as always - appreciated. Cheers, Jeff [1] http://www.mail-archive.com/gossip@jab.org/msg01006.html [2] http://www.mhonarc.org/mharc/doc/ ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] i18n/l10n clarification
I'm happy to report that dozens of lists so far are taking advantage of the new easier-to-use localization forms. For example, this archive made a transition to German yesterday: http://www.mail-archive.com/illustrator@domeus.de http://www.mail-archive.com/illustrator@domeus.de/info.html There's also a little confusion. Localization is per archive and everyone viewing the archive sees the same localization. So - for example - please do not request an English language list to be localized to Japanese. We really don't care who makes the localization request (either list admin or an archive reader) but we do sanity check by hand that the localization matches the language of the request. Requests will usually be processed within 48 hours and only affect new messages. I'll update the FAQ soon with this information, but wanted to get it out quickly for those who need it. -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] [Email Protected]
Every month or two the following happens. Someone does a Google search on their email address, sees a five-year-stale Google search summary and panics. We had a very light obfuscation technique back then, which you don't notice unless you look hard. Anyway, the point is lots of people are very concerned about spam, and I worry that putting up unobfuscated addresses on select lists will open the floodgates. By open the flood gates I mean lots of frantic, confused people calling me up in the middle of the night for the next ten years. I really hate that. So, The Mail Archive does blatent obfuscation with two goals. One is to prevent harvesting, the other is to keep users happy. If a list admin wants to bypass this, one approach is to put some very light obfuscation in on messages before they hit the archival service (perhaps apply at the list server). This requires some technical wizardry but I'm sure there are people on gossip capable of doing it. The other way is to have a really compelling reason why your list should have obfuscation turned off or modified. Enough to justify the risk of people calling me in the middle of the night, and complicating our migration path if we need to change obfuscation techniques some day. It's hard for me to think of a compelling enough scenario. -Jeff PS. I really hate spammers. ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] [Email Protected]
On the other [list], the mail-archive shows up with the light obfuscation, i.e. name [at] domain.com. JF, you get the award for most mysterious problems. If M-A gets a message with name [at] domain.com, it'll keep it. If M-A gets an address with an @ symbol, it'll obfuscate. Good luck figuring out the situation. Sounds like light obfuscation at the list server is what you want. Also, while I'm writing anyway, I tacked on a localization request form onto the info pages which should be of general interest, especially for non-English lists. Should be a more convenient than the previous localization method. See http://www.mail-archive.com/gossip@jab.org/info.html for an example. Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] test message, please ignore
This is a test message to see if we've fixed archiving for [EMAIL PROTECTED] -Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] scheduled maintenance
Hi all, We've been load testing a new server for the last couple of weeks, and plan to switch over tomorrow. Archiving will be put on hold at 5:30am US Pacific time (UTC -0700) and resume probably in the afternoon. Hopefully there will be very little noticeable downtime. This will be - believe it or not - The Mail Archive's seventh generation of hardware. We've been stress testing the new server for a couple of weeks now, and it's a pretty solid improvement. Some buzzwords include x86-64 dual core more memory and gigantic hulking 500GB drives but the bottom line is that a beefier primary machine should help keep the service snappy as it continues to grow. Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
[Gossip] planned downtime today (short)
Hi all, We're upgrading The Mail Archive's internal network to gigabit ethernet this afternoon, and need to take the main webserver offline to install a new network card. With luck total downtime will be well under one hour. All inbound mail will be queued during this time. Cheers, Jeff ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
Re: [Gossip] The Great UTF-8 SWITCHEROO
With Earl's help, the problem with pre-June19th subject lines in index pages has been fixed. Index pages should look pretty good now. Cheers, Jeff PS. Happy 4th of July (I'm planning to watch fireworks in a couple of hours). ___ Discussion list for The Mail Archive Gossip@jab.org http://jab.org/cgi-bin/mailman/listinfo/gossip
