Re: [tor-dev] in buffers_net.c
Hi, Thanks for this bug report and patch. > On 3 Jan 2019, at 13:27, Gisle Vanem wrote: > > Some recent change has added a: > #include > > which fails for MSVC (it doesn't have it). > > Since I fail to find any way to comment on Trac or Gitweb, > I give notice here. Patching into: > > --- a/src/lib/net/buffers_net.c 2019-01-02 18:57:42 > +++ b/src/lib/net/buffers_net.c 2019-01-03 03:16:32 > @@ -21,7 +21,10 @@ > #endif > > #include > + > +#ifndef _MSC_VER > #include > +#endif > > #ifdef PARANOIA > > > > works fine. We used a slightly different check in: https://github.com/torproject/tor/commit/4e4f93d364e33bb46e271f3b960a2cf5f38f402e T signature.asc Description: Message signed with OpenPGP ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Daemon
Hi, It looks like you're trying to run some attacks on Tor. Please don't run attacks on the live Tor network: use a test network instead. > On 3 Jan 2019, at 02:56, marziyeh latifi wrote: > > I have some question about daemon: > 1-what is the Tor daemon? A service that runs all the time: https://en.wikipedia.org/wiki/Daemon_(computing) > 2-What is the relationship between oom killer algorithm and daemon? If the daemon uses too much memory, the oom killer kills it. https://en.wikipedia.org/wiki/Out_of_memory > 3-how can I calculate memory usage of deamon by algorithm? I'm not sure what you're asking here. There is no simple algorithm to calculate memory usage, because it depends on the size of the cell queues, and other data structures. T signature.asc Description: Message signed with OpenPGP ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Multi-threading throughout
On Wed, Jan 09, 2019 at 08:17:15AM -0500, Ian Goldberg wrote: > On Wed, Jan 09, 2019 at 08:42:18PM +1100, Todd Hubers wrote: > > There are early plans to distribute crypto operations across multiple cores > > [https://trac.torproject.org/projects/tor/ticket/1749], but there might be > > a better way. > > > > (I registered, but I couldn't find a way to annotate the ticket, so I'm > > emailing for now) > > > > The ticket states the reason being to saturate the bandwidth available (by > > using all the cores as efficiently as possible). > > > > I don't understand why a relay needs to have a "main thread". Network > > traffic arrives as an async operation and can be sent back out > > asynchronously. So a final strategy shouldn't have a central thread. The > > main thread might still be needed for startup, runtime adjustment, and > > system upkeep, but not for the core network-crypto processing; that should > > never need to touch the main thread. > > > > The current proposal speaks about multi-threading crypto operations, let's > > call that "A) Speed - Speeding up processing of a single cell". Instead, I > > propose "B) Concurrency - Restructuring so multiple cells can be processed > > concurrently". > > > > A cell of data should arrive via IO-Completion thread on a random CPU core, > > have crypto transformation applied on the same one core, then be dispatched > > onward out via the network. This seems to be quite a simple approach where > > I would think crypto code can remain the same "single-threaded" > > implementation. > > > > Approach [A] will have diminishing returns as the number of cores > > increases. You can only break up a cell unit of work so much until you're > > encrypting one byte per cpu core. However, with approach [B], if you have > > millions of CPU cores (as an extreme) you can be processing millions of > > cells concurrently. Therefore, I believe approach [B] would be more > > scalable. > > > > What do you think? > > You'll have troubles if cells *on the same circuit* try to be processed > in parallel on different cores, at least with the current circuit-level > crypto. But, once circuits are established, handing each circuit to a > different thread/core (or more clever worker structure) is something > that I think at least boradly makes sense, and indeed I have been > proposing to have my students work on. (Of course, this only is even relevant for the very highest-bandwidth nodes; my own node, for example, running on 5-year-old hardware with no special configuration, was pushing 400 Mbps last month, with one core at 80%, one at 11%, one at 6%, and the rest trivially small.) -- Ian Goldberg Professor and University Research Chair Cheriton School of Computer Science University of Waterloo ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Re: [tor-dev] Multi-threading throughout
On Wed, Jan 09, 2019 at 08:42:18PM +1100, Todd Hubers wrote: > There are early plans to distribute crypto operations across multiple cores > [https://trac.torproject.org/projects/tor/ticket/1749], but there might be > a better way. > > (I registered, but I couldn't find a way to annotate the ticket, so I'm > emailing for now) > > The ticket states the reason being to saturate the bandwidth available (by > using all the cores as efficiently as possible). > > I don't understand why a relay needs to have a "main thread". Network > traffic arrives as an async operation and can be sent back out > asynchronously. So a final strategy shouldn't have a central thread. The > main thread might still be needed for startup, runtime adjustment, and > system upkeep, but not for the core network-crypto processing; that should > never need to touch the main thread. > > The current proposal speaks about multi-threading crypto operations, let's > call that "A) Speed - Speeding up processing of a single cell". Instead, I > propose "B) Concurrency - Restructuring so multiple cells can be processed > concurrently". > > A cell of data should arrive via IO-Completion thread on a random CPU core, > have crypto transformation applied on the same one core, then be dispatched > onward out via the network. This seems to be quite a simple approach where > I would think crypto code can remain the same "single-threaded" > implementation. > > Approach [A] will have diminishing returns as the number of cores > increases. You can only break up a cell unit of work so much until you're > encrypting one byte per cpu core. However, with approach [B], if you have > millions of CPU cores (as an extreme) you can be processing millions of > cells concurrently. Therefore, I believe approach [B] would be more > scalable. > > What do you think? You'll have troubles if cells *on the same circuit* try to be processed in parallel on different cores, at least with the current circuit-level crypto. But, once circuits are established, handing each circuit to a different thread/core (or more clever worker structure) is something that I think at least boradly makes sense, and indeed I have been proposing to have my students work on. -- Ian Goldberg Professor and University Research Chair Cheriton School of Computer Science University of Waterloo ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
[tor-dev] Multi-threading throughout
There are early plans to distribute crypto operations across multiple cores [https://trac.torproject.org/projects/tor/ticket/1749], but there might be a better way. (I registered, but I couldn't find a way to annotate the ticket, so I'm emailing for now) The ticket states the reason being to saturate the bandwidth available (by using all the cores as efficiently as possible). I don't understand why a relay needs to have a "main thread". Network traffic arrives as an async operation and can be sent back out asynchronously. So a final strategy shouldn't have a central thread. The main thread might still be needed for startup, runtime adjustment, and system upkeep, but not for the core network-crypto processing; that should never need to touch the main thread. The current proposal speaks about multi-threading crypto operations, let's call that "A) Speed - Speeding up processing of a single cell". Instead, I propose "B) Concurrency - Restructuring so multiple cells can be processed concurrently". A cell of data should arrive via IO-Completion thread on a random CPU core, have crypto transformation applied on the same one core, then be dispatched onward out via the network. This seems to be quite a simple approach where I would think crypto code can remain the same "single-threaded" implementation. Approach [A] will have diminishing returns as the number of cores increases. You can only break up a cell unit of work so much until you're encrypting one byte per cpu core. However, with approach [B], if you have millions of CPU cores (as an extreme) you can be processing millions of cells concurrently. Therefore, I believe approach [B] would be more scalable. What do you think? Regards, -- -- Todd Hubers ___ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
