Re: [tor-talk] Funded search engine for onionspace?
On Thu, 05 Mar 2015 11:27:08 +, George Kadianakis wrote: ... FWIW, none of the above will actually help against a non-experienced user that uses tor2web to connect to an onion by mistake. Even with HS authorization or HTTP auth, the onion will forever be imprinted on that public list. That's arguably a design problem of the hidden services - you can't stop this from leaking, only slow it down. I was pretty surprised once to see accesses to an unpublished hidden servive. Andreas -- Totally trivial. Famous last words. From: Linus Torvalds torvalds@*.org Date: Fri, 22 Jan 2010 07:29:21 -0800 -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
grarpamp grarp...@gmail.com writes: On Wed, Mar 4, 2015 at 12:16 PM, George Kadianakis desnac...@riseup.net wrote: I find their concern very valid Respectfully... invalid. Onions are going to be mined, shared, leaked, indexed, and copied anyways. And most certainly by your adversaries. Do we forget merely publishing an onion to the dirs results in accesses. Lists of onions just make all this abundantly and properly obvious for those who don't get the picture. Nor are you going to be able to I understand but don't really agree with your point. Mainly because I can't think of a single positive thing that can happen because of this public list. influence or censor every list. So instead of whining they should be doing something to enforce actual privacy... 1) HiddenServiceAuthorizeClient 2) HTTPS/app level auth FWIW, none of the above will actually help against a non-experienced user that uses tor2web to connect to an onion by mistake. Even with HS authorization or HTTP auth, the onion will forever be imprinted on that public list. 3) OpSec 4) Site defense 5) etc OpSec would help, but it actually relies on the human factor. -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
On Wed, Mar 4, 2015 at 12:16 PM, George Kadianakis desnac...@riseup.net wrote: I find their concern very valid Respectfully... invalid. Onions are going to be mined, shared, leaked, indexed, and copied anyways. And most certainly by your adversaries. Do we forget merely publishing an onion to the dirs results in accesses. Lists of onions just make all this abundantly and properly obvious for those who don't get the picture. Nor are you going to be able to influence or censor every list. So instead of whining they should be doing something to enforce actual privacy... 1) HiddenServiceAuthorizeClient 2) HTTPS/app level auth 3) OpSec 4) Site defense 5) etc -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
I'm not too surprised that list ruffled some feathers. Per your request, I'll take it down. :) If/when people get more used to the idea of being seen on the clear-net the Disallowed might be resurrected. Give the changes a few days to propagate. -V On Thu, Mar 5, 2015 at 1:16 AM, George Kadianakis desnac...@riseup.net wrote: Hello Virgil, I have received mails from a few people who are feeling bad about the disallowed.html list of onioncity. Some of them are afraid that it might list their private hidden service, just because an inexperienced user accidentally tried to access it over tor2web. I find their concern very valid, and I also don't see much benefit from publishing the list anyway. I think removing the list might be the responsible thing to do here. Thoughts? -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
Hello Virgil, I have received mails from a few people who are feeling bad about the disallowed.html list of onioncity. Some of them are afraid that it might list their private hidden service, just because an inexperienced user accidentally tried to access it over tor2web. I find their concern very valid, and I also don't see much benefit from publishing the list anyway. I think removing the list might be the responsible thing to do here. Thoughts? -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
Virgil Griffith i...@virgil.gr writes: I present: http://onion.city currently searching ~348,000 pages according to site:onion.city on GOOG. -V Ah, exciting! The use of a custom google search is an interesting idea. I also like the motto and the logo! (although search engine logos are supposed to be colorful right?) Some comments: - How does the custom google search thing works? Where does it get its index? You expose all the tor2web onions on your sitemap, so google crawls them and generates an index? I'm a bit concerned that clients connect directly to Google. Can this be avoided and still keep the custom google search functionality? - I don't like that the default link is through onion.city. This means that onion.city watches *both* the search query *and* the content of the communication. That's crazy. It's especially crazy if you allow your clients to submit HTTP forms over onion.city, since it basically means that onion.city gets to see *all* the usernames and passwords. I bet there are many people out there who don't really get the tor2web threat model, and it's nasty to read their passwords. There are various ways to solve or semi-solve this problem. My preference is to *always* default to the onion link (and maybe also have an option for a tor2web alternative). Combined with a nice guide on how to download Tor, this might help user education and IMO it's the responsible thing to do. - How do you crawl for more onions? - It really needs HTTPS! - Are you planning to also index non-HTTP services? - Serving onion.city as a hidden service would be nice. - Curious on the funding model here. Will there be ads? Thanks and best of luck with your project! -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
Alas no. I'm aware this is suboptimal. I see GOOG search engine as a temporary-ladder just to get the ball rolling. I am open to using any other index. For what it's worth I'm very pleased with GOOG's performance---right now it's searching an index of 650k onion pages and the number grows every day. If you instead use a google search appliance couldn't you use google engine for indexing without having to use google itself? Wouldn't that also avoid the problem of google queries being associated with the client making the request? Although we technically could read provided passwords, we don't keep logs of passed traffic. However, I understand that many users don't understand the tor2web threat model. But this is the same as all Tor2web nodes, yes? This is not at all unique to OnionCity. As far as I know all Tor2web nodes allow form submissions. What is unique to onion.city is that access to someonion.onion.city occurs using http and doesn't redirect to the .onion if Tor is in use. That the tor2web mirror might snoop is implicit--that the exit (if using tor) might also snoop is more of a concern. You mentioned it'd be better to have it randomly pick among the available Tor2web nodes instead of everything going through OnionCity. This breaks the GOOG search engine which only wants to return canonical URLs. We could talk about making OnionCity a DNS round-robin akin to how Tor2web.org currently works, but then I'm just replicating Tor2web. The ability of tor2web to provide mirrors should be optional. If you only know one mirror and that mirror cannot service the request then how are you going to get any of the other mirrors? Google engine can return related addresses in an order based on the success of loading the mirror itself. If onion.city always works it will tend to precede tor2web.org. If onion.city goes down (having search front-end separate from tor2web mirror) the search engine can reorder the result to improve the success of the first click. Right now I aggregate existing lists of onion sites and put them into the site map. * https://ahmia.fi/onions/ * http://skunksworkedp2cg.onion.city/sites.txt * http://xlmvhk3rpdux26dz.onion.city/ * http://kku5juzqh33a.onion.city/ If google is itself handling the indexing won't that cause a problem for sites in those lists, which are normally okay with being indexed, just not by googlebot? I for one couldn't care less about being indexed by ahmia.fi but it'll be a cold day in hell before I let googlebot. Precisely because of how easy it is to link the search to the requester. --leeroy -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
- How does the custom google search thing works? Where does it get its index? You expose all the tor2web onions on your sitemap, so google crawls them and generates an index? Correct :) Everything available on the Google Custom Search is also available on a regular google search with the qualifier: site:onion.city - I'm a bit concerned that clients connect directly to Google. Can this be avoided and still keep the custom google search functionality? Alas no. I'm aware this is suboptimal. I see GOOG search engine as a temporary-ladder just to get the ball rolling. I am open to using any other index. For what it's worth I'm very pleased with GOOG's performance---right now it's searching an index of 650k onion pages and the number grows every day. - I don't like that the default link is through onion.city. This means that onion.city watches *both* the search query *and* the content of the communication. That's crazy. In short, yes. However, you can prevent OnionCity from seeing the search terms by using site:onion.city on GOOG. I.e., https://www.google.com/webhp?safe=offq=site%3Aonion.cityq=site:onion.city - It's especially crazy if you allow your clients to submit HTTP forms over onion.city, since it basically means that onion.city gets to see *all* the usernames and passwords. I bet there are many people out there who don't really get the tor2web threat model, and it's nasty to read their passwords. Although we technically could read provided passwords, we don't keep logs of passed traffic. However, I understand that many users don't understand the tor2web threat model. But this is the same as all Tor2web nodes, yes? This is not at all unique to OnionCity. As far as I know all Tor2web nodes allow form submissions. - There are various ways to solve or semi-solve this problem. My preference is to *always* default to the onion link (and maybe also have an option for a tor2web alternative). Combined with a nice guide on how to download Tor, this might help user education and IMO it's the responsible thing to do. Currently the guide for downloading Tor is http://onion.city/security.html . Can you suggest something better / more explicit? You mentioned it'd be better to have it randomly pick among the available Tor2web nodes instead of everything going through OnionCity. This breaks the GOOG search engine which only wants to return canonical URLs. We could talk about making OnionCity a DNS round-robin akin to how Tor2web.org currently works, but then I'm just replicating Tor2web. We've discussed OnionCity into Tor2web, but it was discouraged because OnionCity does aggressive behind-the-scenes caching which made Tor2web uncomfortable. I respect Tor2web's collective wishes. - How do you crawl for more onions? Right now I aggregate existing lists of onion sites and put them into the site map. * https://ahmia.fi/onions/ * http://skunksworkedp2cg.onion.city/sites.txt * http://xlmvhk3rpdux26dz.onion.city/ * http://kku5juzqh33a.onion.city/ As-is GOOG has only indexed only 34% of the domains in the sitemap. This can be revisited when GOOG has indexed 90%. - It really needs HTTPS! Agreed 110%. It's already be there but unfortunately providing HTTPS for the CDN is currently out of my budget. This is me inquiring about some of that MEMEX funding :P - Are you planning to also index non-HTTP services? HTTP and HTTPS. That's probably it. Open to others, but then you get into diminishing returns per-unit-effort. - Serving onion.city as a hidden service would be nice. Open to this idea. Right now focusing on keeping response times low and legal hardening. After that rolling out HTTPS. After that however a Hidden Service would be a fine idea. - Curious on the funding model here. Will there be ads? Currently no funding model. Have considered putting ads on the search results. - Thanks and best of luck with your project! Looking forward to Tor 0.2.6! Will be able to provide much more informative error messages and diagnostics then! 3 -V -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
On 2015-02-13 15:30, l.m wrote: If you instead use a google search appliance couldn't you use google engine for indexing without having to use google itself? Wouldn't that also avoid the problem of google queries being associated with the client making the request? It might, but it's licensed based on the number of documents (pages?), starting around $20,000, so it's probably not really an ideal solution for this type of use. (Pricing from http://www.techrepublic.com/blog/google-in-the-enterprise/what-is-a-google-search-appliance/ -- You have to contact them to get a quote, which usually means the price is not reasonable to begin with) -- Dave Warren http://www.hireahit.com/ http://ca.linkedin.com/in/davejwarren -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
Are OnionCity staff reviewing and redacting those lists to protect users from themselves? Or is redaction based only on complaints? We do both. For some privacy, users can instead search https://startpage.com/ with site:onion.city, and then view using the Ixquick Proxy. Could OnionCity script that as the default? I hadn't seen this before! Thanks! I played with proxying everything through Ixquick for a bit but I personally felt the performance drop was too unpleasant to make Ixquick the default. Native HTTPS will come once funding allows. -V -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
On 02/13/2015 03:19 PM, Virgil Griffith wrote: SNIP - How do you crawl for more onions? Right now I aggregate existing lists of onion sites and put them into the site map. * https://ahmia.fi/onions/ * http://skunksworkedp2cg.onion.city/sites.txt * http://xlmvhk3rpdux26dz.onion.city/ * http://kku5juzqh33a.onion.city/ As-is GOOG has only indexed only 34% of the domains in the sitemap. This can be revisited when GOOG has indexed 90%. Are OnionCity staff reviewing and redacting those lists to protect users from themselves? Or is redaction based only on complaints? - It really needs HTTPS! Agreed 110%. It's already be there but unfortunately providing HTTPS for the CDN is currently out of my budget. This is me inquiring about some of that MEMEX funding :P For some privacy, users can instead search https://startpage.com/ with site:onion.city, and then view using the Ixquick Proxy. Could OnionCity script that as the default? SNIP -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
Leeroy, to avoid being indexed by Googlebot et al, place the appropriate /robots.txt at your root. It's described in the FAQ. Yes I'm aware of the faq. It's just that in using google you'll always be incomplete compared to ahmia.fi but thats ok by me. --leeroy -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
On 02/11/2015 11:10 PM, Virgil Griffith wrote: I present: http://onion.city currently searching ~348,000 pages according to site:onion.city on GOOG. -V Dear administrators of onion.city, please do deploy and enforce HTTPS -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
See the FAQ. It's on the roadmap. On Feb 11, 2015 2:17 PM, Alexandros irregula...@riseup.net wrote: On 02/11/2015 11:10 PM, Virgil Griffith wrote: I present: http://onion.city currently searching ~348,000 pages according to site:onion.city on GOOG. -V Dear administrators of onion.city, please do deploy and enforce HTTPS -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
+1 Nice project! Thanks for making it! And like Alexandros said: I would love to see HTTPS available! Am 11.02.2015 um 23:17 schrieb Alexandros: On 02/11/2015 11:10 PM, Virgil Griffith wrote: I present: http://onion.city currently searching ~348,000 pages according to site:onion.city on GOOG. -V Dear administrators of onion.city, please do deploy and enforce HTTPS -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
Re: [tor-talk] Funded search engine for onionspace?
On Wed, Feb 11, 2015 at 4:10 PM, Virgil Griffith i...@virgil.gr wrote: http://onion.city currently searching ~348,000 pages according to site:onion.city on GOOG. Cool. A gpg key should be posted onsite. And an auto updating list of onions indexed.txt to serve as seeds to other projects. -- tor-talk mailing list - tor-talk@lists.torproject.org To unsubscribe or change other settings go to https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk
