Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
On Wed, Jun 26, 2013 at 04:02:15PM -0700, Mike Perry wrote: > YaCY and other FOSS engines (in a sibling thread someone mentioned > another that I already forgot) are also something that I will accept > search plugins for the Omnibox, but their result quality, index depth, > and crawl frequency are no match for either StartPage or DDG. In absence of a P2P name system, even a crappy distributed crawler that indexes onionland is extremely useful. Instead of startpage TBB could bundle YaCy, which only crawls onionland. StartPage seems to be a front to Google, and as such can suffer Scroogle's fate. -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
On 27/06/13 01:02, Mike Perry wrote: > The Doctor: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> On 06/24/2013 09:16 PM, Daniel Sieradski wrote: >>> Has there ever been any effort to create an open source search >>> engine that is entirely transparent in both its software and >>> practices? (dmoz.org >> doesn't count!) >> >> ...YaCY? >> >> http://yacy.de/ > > YaCY and other FOSS engines (in a sibling thread someone mentioned > another that I already forgot) are also something that I will accept > search plugins for the Omnibox, but their result quality, index depth, > and crawl frequency are no match for either StartPage or DDG. > There's also Seeks. http://www.seeks-project.info It's “An Open Decentralized Platform for Collaborative Search, Filtering and content Curation”. From what I understand, Seeks tries to do several things at once: - Provide search results by aggregating them from different sources such as Google, Bing and other seeks nodes. To jumpstart the available results and achieve good quality, they decided the best thing to do was just to grab good results where they were, so by default nodes will ask Google for results. But more backends can and are being developed. - Keep things decentralised. The nodes share results with each other, this is the basis for the general Seeks network's crawler, if I understand correctly. - Enable users on a node to express their like or dislike for the result of a search. This means over time the node learns and will curate results for a given user. Dislikes are kept to a node while positive search results are shared between nodes to build up the general search engine's results. In terms of pure privacy, this does sound like only half a solution : if you run the node on your laptop, seeks is just querying Google for you really. But one can share a node with more people or even use a public node. There are several listed at: http://seeks-project.info/wiki/index.php/List_of_Web_Seeks_nodes In this case, a public seeks node acts like a proxy for new search requests. And for requests that have already been asked, it will give answers on its own without querying external engines. There are also instruction on how to anonymize a Seeks node on the wiki. The project is really interesting, even if a little less active today than it was 18 months ago. But it works and you run it on your server. You could probably set it up as a hidden tor service too. I've cc'd Beniz, who runs the project, he probably has far smarter things to say on the question. :) Cheers axel -- Axel Simon -- mail/jabber/gtalk: axelsi...@axelsimon.net twitter / identi.ca: @AxelSimon -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
The Doctor: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 06/24/2013 09:16 PM, Daniel Sieradski wrote: > > Has there ever been any effort to create an open source search > > engine that is entirely transparent in both its software and > > practices? (dmoz.org > doesn't count!) > > ...YaCY? > > http://yacy.de/ YaCY and other FOSS engines (in a sibling thread someone mentioned another that I already forgot) are also something that I will accept search plugins for the Omnibox, but their result quality, index depth, and crawl frequency are no match for either StartPage or DDG. -- Mike Perry -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/24/2013 09:16 PM, Daniel Sieradski wrote: > Has there ever been any effort to create an open source search > engine that is entirely transparent in both its software and > practices? (dmoz.org doesn't count!) ...YaCY? http://yacy.de/ - -- The Doctor [412/724/301/703] [ZS] Developer, Project Byzantium: http://project-byzantium.org/ PGP: 0x807B17C1 / 7960 1CDC 85C9 0B63 8D9F DD89 3BD8 FF2B 807B 17C1 WWW: https://drwho.virtadpt.net/ "These ribs might be the Buzz Rickson's jacket of _Spook Country_." - --William Gibson -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.20 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHLKBMACgkQO9j/K4B7F8EB0gCgiVrvOP48LZ6wRSpyS7KUUwRF 6SEAnjBBYIO4lOmEXCx11sQRbH6ppzIc =PvpM -END PGP SIGNATURE- -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/24/2013 10:00 PM, Mike Perry wrote: > Michael Carbone: >> On 06/24/2013 08:20 PM, Mike Perry wrote: >>> I've had a number of people tell me that they vouch for >>> DuckDuckGo. What does this even mean? Nobody seems to be >>> capable of rationally explaining it. >>> >>> Have you inspected their datacenter/server security? Have you >>> audited their logging mechanisms? >> >> The data center thing is a non-sequitur -- no third-party service >> has this type of the transparency. My understanding is that you >> don't need to trust these service providers to use them >> anonymously as they are friendly to Tor and no >> scripts/cookies/etc -- hence the difficulties you mention later >> on with Bing & Google. So it doesn't split either way between >> StartPage or DDG. They are equivalent in not allowing personal >> audits of their servers. > > I was questioning where the "vouching" comes from. "Vouch" is a > pretty strong word -- it typically suggests that you are laying > down your reputation on the line to support someone or something > else, either by oath or by evidence. > > My general point is that DuckDuckGo seems to have a lot of appeal > behind it, causing many people to endorse it in extreme ways > without any supporting evidence. > > I want to understand where that support is coming from. As you > point out, the two engines seem largely identical from the > perspective of third party "vouching"/audits wrt privacy. > >>> ** Sure, DuckDuckGo runs a hidden service, and also one of the >>> slowest Tor relays on the network (rate limited to 50KB/sec or >>> less), but it is quite debatable as to if either of these >>> things are actually helpful to Tor. In fact, such a slow Tor >>> relay probably harms Tor performance more than helps (in the >>> rare event that you actually happen to select it). >> >> The hidden service is a plus, no? They seem to be trying at >> least, does Ixquick have either? Maybe it'd be good to reach out >> to DDG about their relay. > > IxQuick has so far successfully negotiated with Google against > outright banning us. Google sees a spike in IxQuick traffic every > time we increase StartPage's prominence in TBB, and this does not > go unnoticed by Google. > > Unfortunately, Google's knee-jerk reaction to each increase so far > is to argue harder in favor of banning all Tor users from both > Startpage and Google, so we'll have to wait and see how this plays > out... > > Backchannel like that (and direct-channel refusals to work with > Tor) really makes you wonder about Google's commitment to privacy > and the freedom of access to information. Very interesting. I don't know the backchannel relationships but I'd guess Google's decision to allow or not allow Tor users doesn't depend on the levels of traffic they get from StartPage from TBB front page. And if it does then that'd be pretty sad, as you note. >> Just trying to rationally explain it. > > I would not rationally use the hidden service version in lieu of > https by default. > > As I alluded to through my questioning of the https backend link to > Bing, the transit path from Tor to DDG is not the weakest link in > an already-https search engine. Okay, so this seems to be the sticking point? Using the !g bang syntax they route Google requests through DDG (so you can search Google if you want, even though they don't seem to rely on Google for their own index). Is that reroute different than what Ixquick does? I don't know. For the index itself, I wasn't able to find anything on the technical connection between DDG and their index sources. Apparently the founder of DDG is interested in getting an external audit, so this might be the type of issue that could solve? He was looking for external audit recommendations as of two days ago ( https://duck.co/topic/we-have-to-talk-about-ddgs-honesty#2846901487421 ). I'd ping him @yegg or y...@alum.mit.edu with some recs. > Further, claims that the performance is the same or similar are > not rigorous. > > Hidden service circuits require ~4X as many Tor router traversals > as normal Tor exit circuits to set up, and unlike normal Tor exit > circuits, they are often *not* prebuilt. Once they are set up, they > still require 2X as many Tor router traversals end-to-end as normal > circuits. You could easily circle the globe several times to issue > a single search query. > > And all this is to use the Tor hidden service's 80bit-secure hash > instead of an https cert, along with all of the other issues with > Tor Hidden Services that have accumulated over the past decade due > to the lack of time for maintenance on Tor's part? I am not > convinced. This is good to know -- don't promote hidden service versions of websites (including DDG) when they have an https version, as hidden services are broken as of now. Michael - -- Michael Carbone Manager of Tech Policy & Programs Access | https://www.accessnow.org mich...@acce
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
On 2013-06-24, at 8:20 PM, Mike Perry wrote: > Nadim Kobeissi: >> I'd just like to add that I'm a DuckDuckGo user myself and that I can >> definitely vouch for the service. > > I've had a number of people tell me that they vouch for DuckDuckGo. What > does this even mean? Nobody seems to be capable of rationally explaining > it. > > Have you inspected their datacenter/server security? Have you audited > their logging mechanisms? Oh! I see my statement has been applied to a different context than the one I originally intended. I simply meant that I vouch for DuckDuckGo as a great service with good policies. I was not commenting with regards to their server security or logging mechanisms. In fact, how could I? I don't suppose it's easy or even possible to, at whim, audit the datacenter of any big search engine. Such an endeavour would require facilitation from the DuckDuckGo team. Auditing a search engine is not like auditing a git repository. NK > > Does DuckDuckGo even have an https channel to Bing on the back end? > > > Note that I don't vouch for StartPage. I merely think that StartPage > provides superior search results to DDG. > > In fact, I wish both companies the best of luck business-wise, and I'm > happy to have both of them at the two top positions in TBB's omnibox. > > This is because right now, there are only two ways to get https web > search results over Tor. Microsoft allows Tor, but has officially > refused to support https directly for Bing. Google regularly bans Tor > nodes entirely, often without the possibility of even entering a Captcha > or using a valid Gmail account (both of which are non-starters for a > default engine of course, but would be better than status quo). > > Every time Tor tries to start a conversation with either Google or > Microsoft on these two topics, they both give us a litany of excuses as > to why fixing the situation is a "hard problem", even after we present > potential cost-effective engineering solutions to both problems. > > For this reason, the loss of either DDG or Startpage would scare the > shit out of me, but right now, neither one has done enough for Tor to > warrant the default search position**, and since StartPage tends to > index more of the deep web faster, it is my opinion we should stick with > them as the top position, and have DDG in second. > > > ** Sure, DuckDuckGo runs a hidden service, and also one of the slowest > Tor relays on the network (rate limited to 50KB/sec or less), but it is > quite debatable as to if either of these things are actually helpful to > Tor. In fact, such a slow Tor relay probably harms Tor performance more > than helps (in the rare event that you actually happen to select it). > > > -- > Mike Perry > -- > Too many emails? Unsubscribe, change to digest, or change password by > emailing moderator at compa...@stanford.edu or changing your settings at > https://mailman.stanford.edu/mailman/listinfo/liberationtech -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
Michael Carbone: > On 06/24/2013 08:20 PM, Mike Perry wrote: > > I've had a number of people tell me that they vouch for DuckDuckGo. > > What does this even mean? Nobody seems to be capable of rationally > > explaining it. > > > > Have you inspected their datacenter/server security? Have you > > audited their logging mechanisms? > > The data center thing is a non-sequitur -- no third-party service has > this type of the transparency. My understanding is that you don't need > to trust these service providers to use them anonymously as they are > friendly to Tor and no scripts/cookies/etc -- hence the difficulties > you mention later on with Bing & Google. So it doesn't split either > way between StartPage or DDG. They are equivalent in not allowing > personal audits of their servers. I was questioning where the "vouching" comes from. "Vouch" is a pretty strong word -- it typically suggests that you are laying down your reputation on the line to support someone or something else, either by oath or by evidence. My general point is that DuckDuckGo seems to have a lot of appeal behind it, causing many people to endorse it in extreme ways without any supporting evidence. I want to understand where that support is coming from. As you point out, the two engines seem largely identical from the perspective of third party "vouching"/audits wrt privacy. > > Note that I don't vouch for StartPage. I merely think that > > StartPage provides superior search results to DDG. > > Since this is the only criterion you base your choice of search engine > on, then perhaps StartPage is the way to go for you. If I were to > argue for DDG, I would point to its much more friendly user > interface/experience (including the html version) and the great !bang > syntax. Maybe it also provides better results for "mainstream" things > as you alluded, I don't know. But there's certainly nothing wrong with > appealing to mainstream folks, this is TBB after all. > > I think these are the reasons why it is gaining a lot of users ( > https://duckduckgo.com/traffic.html ). Either way, users will be able > to choose the other search engine in the omnibox as you mention. That's great! I am glad they are succeeding, and hopefully are in no danger of going away! > > Every time Tor tries to start a conversation with either Google or > > Microsoft on these two topics, they both give us a litany of > > excuses as to why fixing the situation is a "hard problem", even > > after we present potential cost-effective engineering solutions to > > both problems. > > > > For this reason, the loss of either DDG or Startpage would scare > > the shit out of me, but right now, neither one has done enough for > > Tor to warrant the default search position**, and since StartPage > > tends to index more of the deep web faster, it is my opinion we > > should stick with them as the top position, and have DDG in > > second. > > > > ** Sure, DuckDuckGo runs a hidden service, and also one of the > > slowest Tor relays on the network (rate limited to 50KB/sec or > > less), but it is quite debatable as to if either of these things > > are actually helpful to Tor. In fact, such a slow Tor relay > > probably harms Tor performance more than helps (in the rare event > > that you actually happen to select it). > > The hidden service is a plus, no? They seem to be trying at least, > does Ixquick have either? Maybe it'd be good to reach out to DDG about > their relay. IxQuick has so far successfully negotiated with Google against outright banning us. Google sees a spike in IxQuick traffic every time we increase StartPage's prominence in TBB, and this does not go unnoticed by Google. Unfortunately, Google's knee-jerk reaction to each increase so far is to argue harder in favor of banning all Tor users from both Startpage and Google, so we'll have to wait and see how this plays out... Backchannel like that (and direct-channel refusals to work with Tor) really makes you wonder about Google's commitment to privacy and the freedom of access to information. > Just trying to rationally explain it. I would not rationally use the hidden service version in lieu of https by default. As I alluded to through my questioning of the https backend link to Bing, the transit path from Tor to DDG is not the weakest link in an already-https search engine. Further, claims that the performance is the same or similar are not rigorous. Hidden service circuits require ~4X as many Tor router traversals as normal Tor exit circuits to set up, and unlike normal Tor exit circuits, they are often *not* prebuilt. Once they are set up, they still require 2X as many Tor router traversals end-to-end as normal circuits. You could easily circle the globe several times to issue a single search query. And all this is to use the Tor hidden service's 80bit-secure hash instead of an https cert, along with all of the other issues with Tor Hidden Services that have accumulated over the past decade due
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/24/2013 08:20 PM, Mike Perry wrote: > I've had a number of people tell me that they vouch for DuckDuckGo. > What does this even mean? Nobody seems to be capable of rationally > explaining it. > > Have you inspected their datacenter/server security? Have you > audited their logging mechanisms? The data center thing is a non-sequitur -- no third-party service has this type of the transparency. My understanding is that you don't need to trust these service providers to use them anonymously as they are friendly to Tor and no scripts/cookies/etc -- hence the difficulties you mention later on with Bing & Google. So it doesn't split either way between StartPage or DDG. They are equivalent in not allowing personal audits of their servers. > Does DuckDuckGo even have an https channel to Bing on the back > end? Not sure the fixation on Bing, but they pull results from a lot of folks, including Yahoo!, Yandex, and others: http://help.dukgo.com/customer/portal/articles/216399 > Note that I don't vouch for StartPage. I merely think that > StartPage provides superior search results to DDG. Since this is the only criterion you base your choice of search engine on, then perhaps StartPage is the way to go for you. If I were to argue for DDG, I would point to its much more friendly user interface/experience (including the html version) and the great !bang syntax. Maybe it also provides better results for "mainstream" things as you alluded, I don't know. But there's certainly nothing wrong with appealing to mainstream folks, this is TBB after all. I think these are the reasons why it is gaining a lot of users ( https://duckduckgo.com/traffic.html ). Either way, users will be able to choose the other search engine in the omnibox as you mention. > In fact, I wish both companies the best of luck business-wise, and > I'm happy to have both of them at the two top positions in TBB's > omnibox. > > This is because right now, there are only two ways to get https > web search results over Tor. Microsoft allows Tor, but has > officially refused to support https directly for Bing. Google > regularly bans Tor nodes entirely, often without the possibility of > even entering a Captcha or using a valid Gmail account (both of > which are non-starters for a default engine of course, but would be > better than status quo). > > Every time Tor tries to start a conversation with either Google or > Microsoft on these two topics, they both give us a litany of > excuses as to why fixing the situation is a "hard problem", even > after we present potential cost-effective engineering solutions to > both problems. > > For this reason, the loss of either DDG or Startpage would scare > the shit out of me, but right now, neither one has done enough for > Tor to warrant the default search position**, and since StartPage > tends to index more of the deep web faster, it is my opinion we > should stick with them as the top position, and have DDG in > second. > > > ** Sure, DuckDuckGo runs a hidden service, and also one of the > slowest Tor relays on the network (rate limited to 50KB/sec or > less), but it is quite debatable as to if either of these things > are actually helpful to Tor. In fact, such a slow Tor relay > probably harms Tor performance more than helps (in the rare event > that you actually happen to select it). The hidden service is a plus, no? They seem to be trying at least, does Ixquick have either? Maybe it'd be good to reach out to DDG about their relay. Just trying to rationally explain it. Michael - -- Michael Carbone Manager of Tech Policy & Programs Access | https://www.accessnow.org mich...@accessnow.org | PGP: 0x81B7A13E PGP Fingerprint: 25EC 1D0F 2D44 C4F4 5BEF EF83 C471 AD94 81B7 A13E -BEGIN PGP SIGNATURE- iQIcBAEBAgAGBQJRyO/IAAoJEDH9usG3Jz33g7oQAK0ebsqOWa25tb1FysETYDCB YCsO/6Mllvuh8VYA/rGRNh+wzU0O3E9V1BGt0G95VENLm3NoT9LxxJ+eyxKuZwLV OEai1UUdnJA3fIMLHEimxsBXIPF/B4jVKZpkXE5Jm44m1g156cwJg0Wu/UeXD8VZ I+LUY8TtfPwvmBQwM87RXy18h49NDPUo26WmTraAyYDp8iDo0G9STmRqWUn+CQKl o5wSa3imMSFlCgydwfUa/RpBpmkLx9RVzjF/thyGPrsPswAG3YEC8ES0vI3QRw0I nrfIs2NufAzfQzTXHa+tWh0HbycziowHENoTY/vUL2GCNedVaYqYy0qF+hQUYnYc S66ZcnadDb9yitiaMQGZE0sqPkg9tSsrZp8XYsQ8DfUp0CmXa6LOQFvILqcd77om zyeuVau/ftO3O+t1VDTaG1k8HzAvw0RI2BYg+WIgFE+pYrVoCnTMmFZf8MqJ9USM wzlaSBo/wS47YMATnN3TeIHpiqp8lUSXI65KqxLeE3sfb4yoTQ4h04P4uSPmX1c5 gytnuRRFgvCzpoTH8+XF0k2I5h+xFSWcOtUWP6LTDeICxybKgaZp+xibggkCm651 NuT1EgQ7sXX06UJ39Ix6uKnWr4Gy6t34y8OfckHd0wwJWkA6gevXsMAW28CKmaj8 Q87mbgFEhlhARJ3nGtEj =yk6T -END PGP SIGNATURE- -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech
Re: [liberationtech] DuckDuckGo vs Startpage [was: Help test Tor Browser]
Has there ever been any effort to create an open source search engine that is entirely transparent in both its software and practices? (dmoz.org doesn't count!) -- Daniel Sieradski d...@danielsieradski.com http://danielsieradski.com 315.889.1444 Follow me at http://twitter.com/selfagency Public key http://danielsieradski.com/share/ds_public.key On Jun 24, 2013, at 8:20 PM, Mike Perry wrote: > Nadim Kobeissi: >> I'd just like to add that I'm a DuckDuckGo user myself and that I can >> definitely vouch for the service. > > I've had a number of people tell me that they vouch for DuckDuckGo. What > does this even mean? Nobody seems to be capable of rationally explaining > it. > > Have you inspected their datacenter/server security? Have you audited > their logging mechanisms? > > Does DuckDuckGo even have an https channel to Bing on the back end? > > > Note that I don't vouch for StartPage. I merely think that StartPage > provides superior search results to DDG. > > In fact, I wish both companies the best of luck business-wise, and I'm > happy to have both of them at the two top positions in TBB's omnibox. > > This is because right now, there are only two ways to get https web > search results over Tor. Microsoft allows Tor, but has officially > refused to support https directly for Bing. Google regularly bans Tor > nodes entirely, often without the possibility of even entering a Captcha > or using a valid Gmail account (both of which are non-starters for a > default engine of course, but would be better than status quo). > > Every time Tor tries to start a conversation with either Google or > Microsoft on these two topics, they both give us a litany of excuses as > to why fixing the situation is a "hard problem", even after we present > potential cost-effective engineering solutions to both problems. > > For this reason, the loss of either DDG or Startpage would scare the > shit out of me, but right now, neither one has done enough for Tor to > warrant the default search position**, and since StartPage tends to > index more of the deep web faster, it is my opinion we should stick with > them as the top position, and have DDG in second. > > > ** Sure, DuckDuckGo runs a hidden service, and also one of the slowest > Tor relays on the network (rate limited to 50KB/sec or less), but it is > quite debatable as to if either of these things are actually helpful to > Tor. In fact, such a slow Tor relay probably harms Tor performance more > than helps (in the rare event that you actually happen to select it). > > > -- > Mike Perry > -- > Too many emails? Unsubscribe, change to digest, or change password by > emailing moderator at compa...@stanford.edu or changing your settings at > https://mailman.stanford.edu/mailman/listinfo/liberationtech -- Too many emails? Unsubscribe, change to digest, or change password by emailing moderator at compa...@stanford.edu or changing your settings at https://mailman.stanford.edu/mailman/listinfo/liberationtech