date:20180626

[tor-commits] [translation/tails-openpgp-applet_completed] Update translations for tails-openpgp-applet_completed

2018-06-26 Thread translation

commit cf75e556e7e1ef5655e6c123a04bc2ae2f24f88a
Author: Translation commit bot 
Date:   Wed Jun 27 01:48:12 2018 +

Update translations for tails-openpgp-applet_completed
---
 zh_TW/openpgp-applet.pot | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/zh_TW/openpgp-applet.pot b/zh_TW/openpgp-applet.pot
index f04115253..dd1281456 100644
--- a/zh_TW/openpgp-applet.pot
+++ b/zh_TW/openpgp-applet.pot
@@ -11,8 +11,8 @@ msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: ta...@boum.org\n"
 "POT-Creation-Date: 2017-08-05 15:07-0400\n"
-"PO-Revision-Date: 2018-04-12 19:13+\n"
-"Last-Translator: AgustÃn Wu \n"
+"PO-Revision-Date: 2018-06-27 01:32+\n"
+"Last-Translator: Szeming T \n"
 "Language-Team: Chinese (Taiwan) 
(http://www.transifex.com/otf/torproject/language/zh_TW/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tails-openpgp-applet] Update translations for tails-openpgp-applet

2018-06-26 Thread translation

commit fd9221d06c2c243e9696378d546f311360d54b13
Author: Translation commit bot 
Date:   Wed Jun 27 01:48:05 2018 +

Update translations for tails-openpgp-applet
---
 zh_TW/openpgp-applet.pot | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/zh_TW/openpgp-applet.pot b/zh_TW/openpgp-applet.pot
index f04115253..dd1281456 100644
--- a/zh_TW/openpgp-applet.pot
+++ b/zh_TW/openpgp-applet.pot
@@ -11,8 +11,8 @@ msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: ta...@boum.org\n"
 "POT-Creation-Date: 2017-08-05 15:07-0400\n"
-"PO-Revision-Date: 2018-04-12 19:13+\n"
-"Last-Translator: AgustÃn Wu \n"
+"PO-Revision-Date: 2018-06-27 01:32+\n"
+"Last-Translator: Szeming T \n"
 "Language-Team: Chinese (Taiwan) 
(http://www.transifex.com/otf/torproject/language/zh_TW/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/tails-greeter-2] Update translations for tails-greeter-2

2018-06-26 Thread translation

commit e36f1fb6702a5f7fdbfc334d5ce800b697865ddb
Author: Translation commit bot 
Date:   Tue Jun 26 21:48:32 2018 +

Update translations for tails-greeter-2
---
 fa/fa.po | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/fa/fa.po b/fa/fa.po
index 6de3cd988..dd686f6d8 100644
--- a/fa/fa.po
+++ b/fa/fa.po
@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: PACKAGE VERSION\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2018-03-01 20:26+0100\n"
+"POT-Creation-Date: 2018-06-25 09:49+0200\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: Mehrzad, 2017\n"
 "Language-Team: Persian (https://www.transifex.com/otf/teams/1519/fa/)\n"
@@ -16,7 +16,7 @@ msgstr ""
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Language: fa\n"
-"Plural-Forms: nplurals=1; plural=0;\n"
+"Plural-Forms: nplurals=2; plural=(n > 1);\n"
 
 #: ../data/greeter.ui.h:1
 msgid "Administration Password"
@@ -283,14 +283,14 @@ msgstr "_Ø¢ØºØ§Ø² TailÙØ§"
 msgid "default:LTR"
 msgstr "Ù¾ÛØ´ ÙØ±Ø¶:LTR"
 
-#: ../tailsgreeter/persistence.py:89
+#: ../tailsgreeter/persistence.py:91
 #, python-brace-format
 msgid ""
 "live-persist failed with return code {returncode}:\n"
 "{stderr}"
 msgstr ""
 
-#: ../tailsgreeter/persistence.py:122 ../tailsgreeter/persistence.py:139
+#: ../tailsgreeter/persistence.py:124 ../tailsgreeter/persistence.py:141
 #, python-brace-format
 msgid ""
 "cryptsetup failed with return code {returncode}:\n"
@@ -299,7 +299,7 @@ msgid ""
 msgstr ""
 "cryptsetup  ÙØ§ÙÙÙÙ Ø¨ÙØ¯ Ø¨Ø§ Ú©Ø¯ Ø¨Ø§Ø²Ú¯Ø´Øª  {returncode}:  
{stdout}  {stderr}"
 
-#: ../tailsgreeter/persistence.py:156
+#: ../tailsgreeter/persistence.py:158
 #, python-brace-format
 msgid ""
 "live-persist failed with return code {returncode}:\n"
@@ -307,7 +307,7 @@ msgid ""
 "{stderr}"
 msgstr "live-persist ÙÙÙÙ ÙØ¨ÙØ¯ Ø¨Ø§ Ú©Ø¯ Ø¨Ø§Ø²Ú¯Ø´Øª {returncode}:  
{stdout}  {stderr}"
 
-#: ../tailsgreeter/persistence.py:170
+#: ../tailsgreeter/persistence.py:172
 #, python-brace-format
 msgid ""
 "umount failed with return code {returncode}:\n"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/mat-gui] Update translations for mat-gui

2018-06-26 Thread translation

commit a6d3e1ab24c9872b48ffa139f89b59802d8818df
Author: Translation commit bot 
Date:   Tue Jun 26 21:46:07 2018 +

Update translations for mat-gui
---
 fa.po | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fa.po b/fa.po
index b8b9cce27..8afa73701 100644
--- a/fa.po
+++ b/fa.po
@@ -14,14 +14,14 @@ msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: \n"
 "POT-Creation-Date: 2016-02-10 23:06+0100\n"
-"PO-Revision-Date: 2018-04-12 19:01+\n"
-"Last-Translator: Martus Translations \n"
+"PO-Revision-Date: 2018-06-26 21:24+\n"
+"Last-Translator: Saba Rostami \n"
 "Language-Team: Persian 
(http://www.transifex.com/otf/torproject/language/fa/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Language: fa\n"
-"Plural-Forms: nplurals=1; plural=0;\n"
+"Plural-Forms: nplurals=2; plural=(n > 1);\n"
 
 #: mat-gui:66 mat-gui:422 mat-gui:445
 msgid "Ready"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/mat-gui_completed] Update translations for mat-gui_completed

2018-06-26 Thread translation

commit 5993f16777e6de0bc113763be546448a1a780b3b
Author: Translation commit bot 
Date:   Tue Jun 26 21:46:12 2018 +

Update translations for mat-gui_completed
---
 fa.po | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/fa.po b/fa.po
index b8b9cce27..8afa73701 100644
--- a/fa.po
+++ b/fa.po
@@ -14,14 +14,14 @@ msgstr ""
 "Project-Id-Version: The Tor Project\n"
 "Report-Msgid-Bugs-To: \n"
 "POT-Creation-Date: 2016-02-10 23:06+0100\n"
-"PO-Revision-Date: 2018-04-12 19:01+\n"
-"Last-Translator: Martus Translations \n"
+"PO-Revision-Date: 2018-06-26 21:24+\n"
+"Last-Translator: Saba Rostami \n"
 "Language-Team: Persian 
(http://www.transifex.com/otf/torproject/language/fa/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Language: fa\n"
-"Plural-Forms: nplurals=1; plural=0;\n"
+"Plural-Forms: nplurals=2; plural=(n > 1);\n"
 
 #: mat-gui:66 mat-gui:422 mat-gui:445
 msgid "Ready"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/support-tbb] Update translations for support-tbb

2018-06-26 Thread translation

commit 4d332cb1310497dfeb501cd2ab25f88e01f8f92e
Author: Translation commit bot 
Date:   Tue Jun 26 21:19:33 2018 +

Update translations for support-tbb
---
 sv.json | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/sv.json b/sv.json
index d858073ea..364022681 100644
--- a/sv.json
+++ b/sv.json
@@ -15,7 +15,7 @@
"id": "#tbb-3",
"control": "tbb-3",
"title": "NÃ¤r jag anvÃ¤nder Tor Browser, kommer nÃ¥gon se vilka 
hemsidor jag besÃ¶ker?",
-   "description": "Tor Browser prevents people from 
knowing the websites you visit. Some entities, such as your Internet Service 
Provider (ISP), may be able to see that you're using Tor."
+   "description": "Tor Browser fÃ¶rhindrar att nÃ¥gon vet vilka sidor du 
besÃ¶ker. NÃ¥gra stÃ¤llen, till exempel din bredbandsoperatÃ¶r kan ha 
mÃ¶jlighet att se att du anvÃ¤nder Tor. "
 },
 "tbb-4": {
"id": "#tbb-4",
@@ -86,8 +86,8 @@
 "tbb-14": {
"id": "#tbb-14",
"control": "tbb-14",
-   "title": "Should I install a new add-on or extension in Tor Browser, 
like AdBlock Plus or uBlock Origin?",
-   "description": "It's strongly discouraged to install 
new add-ons in Tor Browser, because they can compromise both your privacy and 
your security. Plus, Tor Browser already comes installed with two add-ons â 
HTTPS Everywhere and NoScript â which give you added protection."
+   "title": "Borde jag installera ett nytt add-on eller tillÃ¤gg i Tor 
Browser som Adblock Plus eller uBlock Origin?",
+   "description": "Det avrÃ¥ds starkt att installera nya add-ons i Tor 
Browser fÃ¶r att de kan Ã¤ventyra bÃ¥da din integritet och din sÃ¤kerhet. 
Dessutom sÃ¥ kommer Tor Browser redan installerat med tvÃ¥ add-ons -- HTTPS 
Everywhere och NoScript-- som ger dig Ã¶kad sÃ¤kerhet. "
 },
 "tbb-15": {
"id": "#tbb-15",

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/support-faq] Update translations for support-faq

2018-06-26 Thread translation

commit 964734bd2c24dedf5002b7f4471e43a5e44af35a
Author: Translation commit bot 
Date:   Tue Jun 26 21:19:03 2018 +

Update translations for support-faq
---
 sv.json | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/sv.json b/sv.json
index 8aa9e6eaa..82e8bdc31 100644
--- a/sv.json
+++ b/sv.json
@@ -3,7 +3,7 @@
"id": "#faq-1",
"control": "faq-1",
"title": "NÃ¤r jag anvÃ¤nder Tor Browser, kommer nÃ¥gon se vilka 
hemsidor jag besÃ¶ker?",
-   "description": "Tor Browser prevents people from 
knowing the websites you visit. Some entities, such as your Internet Service 
Provider (ISP), may be able to see that you're using Tor."
+   "description": "Tor Browser fÃ¶rhindrar att nÃ¥gon vet vilka sidor du 
besÃ¶ker. NÃ¥gra stÃ¤llen, till exempel din bredbandsoperatÃ¶r kan ha 
mÃ¶jlighet att se att du anvÃ¤nder Tor. "
 },
 "faq-2": {
"id": "#faq-2",
@@ -14,8 +14,8 @@
 "faq-3": {
"id": "#faq-3",
"control": "faq-3",
-   "title": "Should I install a new add-on or extension in Tor Browser, 
like AdBlock Plus or uBlock Origin?",
-   "description": "It's strongly discouraged to install 
new add-ons in Tor Browser, because they can compromise both your privacy and 
your security. Plus, Tor Browser already comes installed with two add-ons â 
HTTPS Everywhere and NoScript â which give you added protection."
+   "title": "Borde jag installera ett nytt add-on eller tillÃ¤gg i Tor 
Browser som Adblock Plus eller uBlock Origin?",
+   "description": "Det avrÃ¥ds starkt att installera nya add-ons i Tor 
Browser fÃ¶r att de kan Ã¤ventyra bÃ¥da din integritet och din sÃ¤kerhet. 
Dessutom sÃ¥ kommer Tor Browser redan installerat med tvÃ¥ add-ons -- HTTPS 
Everywhere och NoScript-- som ger dig Ã¶kad sÃ¤kerhet. "
 },
 "faq-4": {
"id": "#faq-4",
@@ -27,6 +27,6 @@
"id": "#faq-5",
"control": "faq-5",
"title": "Kan jag anvÃ¤nda ett VPN med Tor?",
-   "description": "Generally speaking, we don't 
recommend using a VPN with Tor unless you're an advanced user who knows how to 
configure both in a way that doesn't compromise your privacy. Learn more about 
combining Tor + VPN."
+   "description": "Generellt sÃ¥ rekommenderar vi inte att anvÃ¤nda VPN 
med Tor om inte du Ã¤r en avancerad anvÃ¤ndare som vet hur man konfigurerar 
bÃ¥da pÃ¥ ett sÃ¤tt sÃ¥ att du inte Ã¤ventyrar din integritet. LÃ¤s mer om att 
kombinera Tor + VPN. "
 }
 }

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Merge branch 'ticket26439'

2018-06-26 Thread nickm

commit 5f41bc91c6817649ebcfbecd184640aad7a5c8f1
Merge: ebbb0348d 56c1fbf33
Author: Nick Mathewson 
Date:   Tue Jun 26 17:02:06 2018 -0400

Merge branch 'ticket26439'

 changes/ticket26439 | 3 +++
 configure.ac| 7 +++
 src/test/include.am | 1 -
 3 files changed, 10 insertions(+), 1 deletion(-)

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Use the "commands" argument of AC_CONFIG_FILES to make scripts +x

2018-06-26 Thread nickm

commit 56c1fbf33f34e010ec33e7660fab3ec3ecb43fc8
Author: Nick Mathewson 
Date:   Thu Jun 21 09:14:19 2018 -0400

Use the "commands" argument of AC_CONFIG_FILES to make scripts +x

Closes ticket 26439.
---
 changes/ticket26439 | 3 +++
 configure.ac| 7 +++
 src/test/include.am | 1 -
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/changes/ticket26439 b/changes/ticket26439
new file mode 100644
index 0..b813fe43e
--- /dev/null
+++ b/changes/ticket26439
@@ -0,0 +1,3 @@
+  o Minor features (build):
+- When generating scripts from autoconf, also make them executable.
+  Closes ticket 26439.
diff --git a/configure.ac b/configure.ac
index 30f8e63ec..14a1a9436 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2275,6 +2275,13 @@ AC_CONFIG_FILES([
 src/rust/.cargo/config
 scripts/maint/checkOptionDocs.pl
 scripts/maint/updateVersions.pl
+], [
+  chmod +x contrib/dist/suse/tor.sh
+  chmod +x link_rust.sh
+  chmod +x contrib/dist/tor.sh
+  chmod +x contrib/dist/torctl
+  chmod +x scripts/maint/checkOptionDocs.pl
+  chmod +x scripts/maint/updateVersions.pl
 ])
 
 if test "x$asciidoc" = "xtrue" && test "$ASCIIDOC" = "none"; then
diff --git a/src/test/include.am b/src/test/include.am
index 7e5ad4611..1080184ac 100644
--- a/src/test/include.am
+++ b/src/test/include.am
@@ -12,7 +12,6 @@ TESTS_ENVIRONMENT = \
export EXTRA_CARGO_OPTIONS="$(EXTRA_CARGO_OPTIONS)"; \
export CARGO_ONLINE="$(CARGO_ONLINE)"; \
 export CCLD="$(CCLD)"; \
-chmod +x "$(abs_top_builddir)/link_rust.sh"; \
 export RUSTFLAGS="-C linker=$(abs_top_builddir)/link_rust.sh";
 
 TESTSCRIPTS = \



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Refactor bloom filter logic not to be digest-specific.

2018-06-26 Thread nickm

commit bf89278c799e718a3b6c1ea139a6069db054ac09
Author: Nick Mathewson 
Date:   Tue Jun 26 13:18:23 2018 -0400

Refactor bloom filter logic not to be digest-specific.

Now the address-set code and the digest-set code share the same
backend.

Closes ticket 26510
---
 changes/ticket26510   |  4 ++
 src/common/address_set.c  | 85 ---
 src/common/address_set.h  | 12 ++
 src/lib/container/bloomfilt.c | 79 +---
 src/lib/container/bloomfilt.h | 71 
 src/lib/crypt_ops/digestset.c | 58 +
 src/lib/crypt_ops/digestset.h | 32 
 src/lib/crypt_ops/include.am  |  6 ++-
 src/or/entrynodes.c   |  2 +-
 src/or/routerlist.c   |  2 +-
 src/test/bench.c  |  4 +-
 src/test/test_containers.c|  2 +-
 12 files changed, 219 insertions(+), 138 deletions(-)

diff --git a/changes/ticket26510 b/changes/ticket26510
new file mode 100644
index 0..f00457964
--- /dev/null
+++ b/changes/ticket26510
@@ -0,0 +1,4 @@
+  o Code simplification and refactoring:
+- Unify our bloom filter logic. Previously we had two copies of this
+  code: one for routerlist filtering, and one for address set
+  calculations. Closes ticket 26510.
diff --git a/src/common/address_set.c b/src/common/address_set.c
index f4c5f581c..369d33e9a 100644
--- a/src/common/address_set.c
+++ b/src/common/address_set.c
@@ -14,35 +14,19 @@
 #include "common/address_set.h"
 #include "common/address.h"
 #include "common/compat.h"
-#include "lib/container/bitarray.h"
+#include "lib/container/bloomfilt.h"
 #include "lib/crypt_ops/crypto_rand.h"
 #include "common/util.h"
 #include "siphash.h"
 
-/** How many 64-bit siphash values to extract per address */
-#define N_HASHES 2
-/** How many bloom-filter bits we set per address. This is twice the N_HASHES
- * value, since we split the siphash output into two 32-bit values. */
-#define N_BITS_PER_ITEM (N_HASHES * 2)
-
-/*  This code is largely duplicated with digestset_t.  We should merge
- * them together into a common bloom-filter implementation.  I'm keeping
- * them separate for now, though, since this module needs to be backported
- * all the way to 0.2.9.
- *
- * The main difference between digestset_t and this code is that we use
- * independent siphashes rather than messing around with bit-shifts.  The
- * approach here is probably more sound, and we should prefer it if we
- * unify the implementations.
- */
-
-struct address_set_t {
-  /** siphash keys to make N_HASHES independent hashes for each address. */
-  struct sipkey key[N_HASHES];
-  int mask; /**< One less than the number of bits in ba; always one less
- * than a power of two. */
-  bitarray_t *ba; /**< A bit array to implement the Bloom filter. */
-};
+/* Wrap our hash function to have the signature that the bloom filter
+ * needs. */
+static uint64_t
+bloomfilt_addr_hash(const struct sipkey *key,
+const void *item)
+{
+  return tor_addr_keyed_hash(key, item);
+}
 
 /**
  * Allocate and return an address_set, suitable for holding up to
@@ -51,33 +35,11 @@ struct address_set_t {
 address_set_t *
 address_set_new(int max_addresses_guess)
 {
-  /* See digestset_new() for rationale on this equation. */
-  int n_bits = 1u << (tor_log2(max_addresses_guess)+5);
-
-  address_set_t *set = tor_malloc_zero(sizeof(address_set_t));
-  set->mask = n_bits - 1;
-  set->ba = bitarray_init_zero(n_bits);
-  crypto_rand((char*) set->key, sizeof(set->key));
-
-  return set;
-}
-
-/**
- * Release all storage associated with set.
- */
-void
-address_set_free(address_set_t *set)
-{
-  if (! set)
-return;
-
-  bitarray_free(set->ba);
-  tor_free(set);
+  uint8_t k[BLOOMFILT_KEY_LEN];
+  crypto_rand((void*)k, sizeof(k));
+  return bloomfilt_new(max_addresses_guess, bloomfilt_addr_hash, k);
 }
 
-/** Yield the bit index corresponding to 'val' for set. */
-#define BIT(set, val) ((val) & (set)->mask)
-
 /**
  * Add addr to set.
  *
@@ -87,14 +49,7 @@ address_set_free(address_set_t *set)
 void
 address_set_add(address_set_t *set, const struct tor_addr_t *addr)
 {
-  int i;
-  for (i = 0; i < N_HASHES; ++i) {
-uint64_t h = tor_addr_keyed_hash(>key[i], addr);
-uint32_t high_bits = (uint32_t)(h >> 32);
-uint32_t low_bits = (uint32_t)(h);
-bitarray_set(set->ba, BIT(set, high_bits));
-bitarray_set(set->ba, BIT(set, low_bits));
-  }
+  bloomfilt_add(set, addr);
 }
 
 /** As address_set_add(), but take an ipv4 address in host order. */
@@ -111,18 +66,8 @@ address_set_add_ipv4h(address_set_t *set, uint32_t addr)
  * return false if addr is not a member of set.)
  */
 int
-address_set_probably_contains(address_set_t *set,
+address_set_probably_contains(const address_set_t *set,
   const struct tor_addr_t *addr)
 {
-  int i, matches = 0;
-  for (i = 0; i < N_HASHES; ++i) {
-

[tor-commits] [tor/master] Finish renaming digestset_contains to digestset_probably_contains

2018-06-26 Thread nickm

commit ebbb0348dc3ad57f6be8208a90f3c2fd9fe2cbf7
Author: Nick Mathewson 
Date:   Tue Jun 26 13:20:54 2018 -0400

Finish renaming digestset_contains to digestset_probably_contains

Since bloom filters are probabilistic, it's nice to make it clear
that the "contains" operation can have false positives.
---
 src/lib/crypt_ops/digestset.h |  3 ---
 src/or/entrynodes.c   |  2 +-
 src/or/routerlist.c   |  7 ---
 src/test/bench.c  | 10 ++
 src/test/test_containers.c|  6 +++---
 5 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/src/lib/crypt_ops/digestset.h b/src/lib/crypt_ops/digestset.h
index 3b5d62c31..328979ae0 100644
--- a/src/lib/crypt_ops/digestset.h
+++ b/src/lib/crypt_ops/digestset.h
@@ -26,7 +26,4 @@ void digestset_add(digestset_t *set, const char *addr);
 int digestset_probably_contains(const digestset_t *set,
 const char *addr);
 
-//  to remove.
-#define digestset_contains digestset_probably_contains
-
 #endif
diff --git a/src/or/entrynodes.c b/src/or/entrynodes.c
index 9d27d80a8..47e689b5c 100644
--- a/src/or/entrynodes.c
+++ b/src/or/entrynodes.c
@@ -1065,7 +1065,7 @@ get_eligible_guards(const or_options_t *options,
   continue;
   }
   ++n_guards;
-  if (digestset_contains(sampled_guard_ids, node->identity))
+  if (digestset_probably_contains(sampled_guard_ids, node->identity))
 continue;
   smartlist_add(eligible_guards, (node_t*)node);
 } SMARTLIST_FOREACH_END(node);
diff --git a/src/or/routerlist.c b/src/or/routerlist.c
index a7dda68a5..66967f13f 100644
--- a/src/or/routerlist.c
+++ b/src/or/routerlist.c
@@ -4116,7 +4116,8 @@ routerlist_remove_old_cached_routers_with_id(time_t now,
 signed_descriptor_t *r_next;
 lifespans[i-lo].idx = i;
 if (r->last_listed_as_valid_until >= now ||
-(retain && digestset_contains(retain, r->signed_descriptor_digest))) {
+(retain && digestset_probably_contains(retain,
+   r->signed_descriptor_digest))) {
   must_keep[i-lo] = 1;
 }
 if (i < hi) {
@@ -4211,7 +4212,7 @@ routerlist_remove_old_routers(void)
   router = smartlist_get(routerlist->routers, i);
   if (router->cache_info.published_on <= cutoff &&
   router->cache_info.last_listed_as_valid_until < now &&
-  !digestset_contains(retain,
+  !digestset_probably_contains(retain,
   router->cache_info.signed_descriptor_digest)) {
 /* Too old: remove it.  (If we're a cache, just move it into
  * old_routers.) */
@@ -4232,7 +4233,7 @@ routerlist_remove_old_routers(void)
 sd = smartlist_get(routerlist->old_routers, i);
 if (sd->published_on <= cutoff &&
 sd->last_listed_as_valid_until < now &&
-!digestset_contains(retain, sd->signed_descriptor_digest)) {
+!digestset_probably_contains(retain, sd->signed_descriptor_digest)) {
   /* Too old.  Remove it. */
   routerlist_remove_old(routerlist, sd, i--);
 }
diff --git a/src/test/bench.c b/src/test/bench.c
index aa4af703f..6653f4942 100644
--- a/src/test/bench.c
+++ b/src/test/bench.c
@@ -404,18 +404,20 @@ bench_dmap(void)
  NANOCOUNT(pt3, pt4, iters*elts));
 
   for (i = 0; i < iters; ++i) {
-SMARTLIST_FOREACH(sl, const char *, cp, n += digestset_contains(ds, cp));
-SMARTLIST_FOREACH(sl2, const char *, cp, n += digestset_contains(ds, cp));
+SMARTLIST_FOREACH(sl, const char *, cp,
+  n += digestset_probably_contains(ds, cp));
+SMARTLIST_FOREACH(sl2, const char *, cp,
+  n += digestset_probably_contains(ds, cp));
   }
   end = perftime();
-  printf("digestset_contains: %.2f ns per element.\n",
+  printf("digestset_probably_contains: %.2f ns per element.\n",
  NANOCOUNT(pt4, end, iters*elts*2));
   /* We need to use this, or else the whole loop gets optimized out. */
   printf("Hits == %d\n", n);
 
   for (i = 0; i < fpostests; ++i) {
 crypto_rand(d, 20);
-if (digestset_contains(ds, d)) ++fp;
+if (digestset_probably_contains(ds, d)) ++fp;
   }
   printf("False positive rate on digestset: %.2f%%\n",
  (fp/(double)fpostests)*100);
diff --git a/src/test/test_containers.c b/src/test/test_containers.c
index acb7543b3..5c712a9b1 100644
--- a/src/test/test_containers.c
+++ b/src/test/test_containers.c
@@ -644,18 +644,18 @@ test_container_digestset(void *arg)
   }
   set = digestset_new(1000);
   SMARTLIST_FOREACH(included, const char *, cp,
-if (digestset_contains(set, cp))
+if (digestset_probably_contains(set, cp))
   ok = 0);
   tt_assert(ok);
   SMARTLIST_FOREACH(included, const char *, cp,
 digestset_add(set, cp));
   SMARTLIST_FOREACH(included, const char *, cp,
-if (!digestset_contains(set, cp))
+if

[tor-commits] [translation/support-tbb] Update translations for support-tbb

2018-06-26 Thread translation

commit 368cd78960975e5ae52cd7e08cf53eb25320a403
Author: Translation commit bot 
Date:   Tue Jun 26 20:49:34 2018 +

Update translations for support-tbb
---
 sv.json | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/sv.json b/sv.json
index ff46776a9..d858073ea 100644
--- a/sv.json
+++ b/sv.json
@@ -14,7 +14,7 @@
 "tbb-3": {
"id": "#tbb-3",
"control": "tbb-3",
-   "title": "When I use Tor Browser, will anyone be able to tell which 
websites I visit?",
+   "title": "NÃ¤r jag anvÃ¤nder Tor Browser, kommer nÃ¥gon se vilka 
hemsidor jag besÃ¶ker?",
"description": "Tor Browser prevents people from 
knowing the websites you visit. Some entities, such as your Internet Service 
Provider (ISP), may be able to see that you're using Tor."
 },
 "tbb-4": {
@@ -122,8 +122,8 @@
 "tbb-20": {
"id": "#tbb-20",
"control": "tbb-20",
-   "title": "Tor Browser won't connect, but it doesnât seem to be an 
issue with censorship.",
-   "description": "One of the most common issues that 
causes connection errors in Tor Browser is an incorrect system clock. Please 
make sure your system clock and timezone are set accurately. If this doesn't 
fix the problem, see the âTroubleshooting page on the https://tb-manual.torproject.org/en-US/bridges.html\;>âTor Browser 
manual."
+   "title": "Tor Browser ansluter inte men det verkar inte vara pÃ¥ grund 
av censur.",
+   "description": "En av de vanligaste problemen som orsakar 
anslutningsproblem till Tor Browser Ã¤r en felaktig system klocka. VÃ¤nligen se 
till att system klockan och tidszonen Ã¤r korrekt instÃ¤llda. Om det inte 
lÃ¶ser problemet, se Troubleshooting sidan i Tor Browser Manual .https://tb-manual.torproject.org/en-US/bridges.html\;> "
 },
 "tbb-21": {
"id": "#tbb-21",

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/support-faq] Update translations for support-faq

2018-06-26 Thread translation

commit b02eef8f86b10d99c0149a7e5401d5234673351a
Author: Translation commit bot 
Date:   Tue Jun 26 20:49:06 2018 +

Update translations for support-faq
---
 sv.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sv.json b/sv.json
index bf9c2b27a..8aa9e6eaa 100644
--- a/sv.json
+++ b/sv.json
@@ -2,7 +2,7 @@
 "faq-1": {
"id": "#faq-1",
"control": "faq-1",
-   "title": "When I use Tor Browser, will anyone be able to tell which 
websites I visit?",
+   "title": "NÃ¤r jag anvÃ¤nder Tor Browser, kommer nÃ¥gon se vilka 
hemsidor jag besÃ¶ker?",
"description": "Tor Browser prevents people from 
knowing the websites you visit. Some entities, such as your Internet Service 
Provider (ISP), may be able to see that you're using Tor."
 },
 "faq-2": {

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [translation/support-connecting] Update translations for support-connecting

2018-06-26 Thread translation

commit a0b5f918f52c170f26f500591245156d5e0edfde
Author: Translation commit bot 
Date:   Tue Jun 26 20:48:57 2018 +

Update translations for support-connecting
---
 sv.json | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sv.json b/sv.json
index 9f5a678cf..faf82a45b 100644
--- a/sv.json
+++ b/sv.json
@@ -2,8 +2,8 @@
 "connecting-1": {
"id": "#connecting-1",
"control": "connecting-1",
-   "title": "Tor Browser won't connect, but it doesnât seem to be an 
issue with censorship.",
-   "description": "One of the most common issues that 
causes connection errors in Tor Browser is an incorrect system clock. Please 
make sure your system clock and timezone are set accurately. If this doesn't 
fix the problem, see the âTroubleshooting page on the https://tb-manual.torproject.org/en-US/bridges.html\;>âTor Browser 
manual."
+   "title": "Tor Browser ansluter inte men det verkar inte vara pÃ¥ grund 
av censur.",
+   "description": "En av de vanligaste problemen som orsakar 
anslutningsproblem till Tor Browser Ã¤r en felaktig system klocka. VÃ¤nligen se 
till att system klockan och tidszonen Ã¤r korrekt instÃ¤llda. Om det inte 
lÃ¶ser problemet, se Troubleshooting sidan i Tor Browser Manual .https://tb-manual.torproject.org/en-US/bridges.html\;> "
 },
 "connecting-2": {
"id": "#connecting-2",

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor-browser-build/master] Bug 26489: fix .app directory name in tools/dmg2mar

2018-06-26 Thread gk

commit 1633185de3336351f47cb66b99903ef70b04d6ff
Author: Nicolas Vigier 
Date:   Mon Jun 25 14:41:39 2018 +0200

Bug 26489: fix .app directory name in tools/dmg2mar

Add the whitespace in "Tor Browser.app".
---
 tools/dmg2mar | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tools/dmg2mar b/tools/dmg2mar
index 51d6acb..9b98956 100755
--- a/tools/dmg2mar
+++ b/tools/dmg2mar
@@ -36,9 +36,10 @@ use Parallel::ForkManager;
 use Cwd;
 
 # If the application is not TorBrowser (for instance, TorMessenger)
-# set the application name in the TOR_APPNAME_BUNDLE_OSX and in
-# the TOR_APPNAME_MARFILE environment variables
-my $appname = $ENV{TOR_APPNAME_BUNDLE_OSX} // 'TorBrowser';
+# set the application name in the TOR_APPNAME_BUNDLE_OSX,
+# TOR_APPNAME_DMGFILE and TOR_APPNAME_MARFILE environment variables
+my $appname = $ENV{TOR_APPNAME_BUNDLE_OSX} // 'Tor Browser';
+my $appname_dmg = $ENV{TOR_APPNAME_DMGFILE} // 'TorBrowser';
 my $appname_mar = $ENV{TOR_APPNAME_MARFILE} // 'tor-browser';
 
 sub exit_error {
@@ -95,7 +96,7 @@ sub get_dmg_files_from_sha256sums {
 foreach my $line (read_file('sha256sums-unsigned-build.txt')) {
 my (undef, $filename) = split '  ', $line;
 chomp $filename;
-next unless $filename =~ m/^$appname-(.+)-osx64_(.+)\.dmg$/;
+next unless $filename =~ m/^$appname_dmg-(.+)-osx64_(.+)\.dmg$/;
 push @files, { filename => $filename, version => $1, lang => $2 };
 }
 return @files;



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor-browser-build/master] Merge remote-tracking branch 'boklm/bug_26489'

2018-06-26 Thread gk

commit 5d5401f7a0831702d913d5e469f70954c5796edb
Merge: ce2f13c 1633185
Author: Georg Koppen 
Date:   Tue Jun 26 19:30:03 2018 +

Merge remote-tracking branch 'boklm/bug_26489'

 tools/dmg2mar | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [stem/master] Remove redundant content-length checks

2018-06-26 Thread atagar

commit 3d3e68c05fbe84f7cf0eff45e376480ec50b48c3
Author: Dave Rolek 
Date:   Sun Jun 3 19:32:57 2018 +

Remove redundant content-length checks

The Size pop calls() following where the respective checks were will
raise an exception if the content isn't large enough for that Size.

Additionally, for the NetinfoCell payload unpacking, the removed check
exception was actually unreachable unless NetinfoCell._unpack() were
called directly, instead of through Cell.pop(), since it is a
fixed-sized cell.
---
 stem/client/cell.py | 3 ---
 stem/client/datatype.py | 5 -
 2 files changed, 8 deletions(-)

diff --git a/stem/client/cell.py b/stem/client/cell.py
index 57ccc1ab..0f9ae59e 100644
--- a/stem/client/cell.py
+++ b/stem/client/cell.py
@@ -555,9 +555,6 @@ class NetinfoCell(Cell):
 
   @classmethod
   def _unpack(cls, content, circ_id, link_protocol):
-if len(content) < Size.LONG.size:
-  raise ValueError('NETINFO cell expected to start with a timestamp')
-
 timestamp, content = Size.LONG.pop(content)
 receiver_address, content = Address.pop(content)
 
diff --git a/stem/client/datatype.py b/stem/client/datatype.py
index db9d461a..2ee624cb 100644
--- a/stem/client/datatype.py
+++ b/stem/client/datatype.py
@@ -420,11 +420,6 @@ class Address(Field):
 
   @staticmethod
   def pop(content):
-if not content:
-  raise ValueError('Payload empty where an address was expected')
-elif len(content) < 2:
-  raise ValueError('Insuffient data for address headers')
-
 addr_type, content = Size.CHAR.pop(content)
 addr_length, content = Size.CHAR.pop(content)
 

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [webwml/master] Add new Tor Browser version: 7.5.6

2018-06-26 Thread boklm

commit a56a6b5900f6f2019c5ae3d0d4af7fa7d8d94261
Author: Nicolas Vigier 
Date:   Tue Jun 26 18:41:14 2018 +0200

Add new Tor Browser version: 7.5.6
---
 include/versions.wmi   | 6 +++---
 projects/torbrowser/RecommendedTBBVersions | 4 
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/include/versions.wmi b/include/versions.wmi
index 2e159841..32fdd152 100644
--- a/include/versions.wmi
+++ b/include/versions.wmi
@@ -4,9 +4,9 @@
 maint-7.5
 
 # *** tor browser stable ***
-7.5.5
-2018-06-09
-0.3.2.10
+7.5.6
+2018-06-26
+0.3.3.7
 
 # If all platforms are on the same version, you only need to update
 # version-torbrowserbundle-all and releasedate-torbrowserbundle-all
diff --git a/projects/torbrowser/RecommendedTBBVersions 
b/projects/torbrowser/RecommendedTBBVersions
index b6dd3202..a66a1e87 100644
--- a/projects/torbrowser/RecommendedTBBVersions
+++ b/projects/torbrowser/RecommendedTBBVersions
@@ -3,6 +3,10 @@
 "7.5.5-MacOS",
 "7.5.5-Linux",
 "7.5.5-Windows",
+"7.5.6",
+"7.5.6-MacOS",
+"7.5.6-Linux",
+"7.5.6-Windows",
 "8.0a8",
 "8.0a8-MacOS",
 "8.0a8-Linux",

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Remove some accidentally committed code in checkIncludes.py

2018-06-26 Thread nickm

commit 860b9a991879c5be2b32cf98766adf5fdd349d41
Author: Nick Mathewson 
Date:   Tue Jun 26 12:04:24 2018 -0400

Remove some accidentally committed code in checkIncludes.py
---
 scripts/maint/checkIncludes.py | 15 +++
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/scripts/maint/checkIncludes.py b/scripts/maint/checkIncludes.py
index cb3354abf..3c611675e 100755
--- a/scripts/maint/checkIncludes.py
+++ b/scripts/maint/checkIncludes.py
@@ -58,21 +58,12 @@ def load_include_rules(fname):
 result.addPattern(line)
 return result
 
-MDD = "--mdd" in sys.argv[1:]
-
-if MDD:
-print(
-
 for dirpath, dirnames, fnames in os.walk("src"):
 if ".may_include" in fnames:
 rules = load_include_rules(os.path.join(dirpath, RULES_FNAME))
-if MDD:
-dp = dirpath.replace("src/","")
-rules.dump_mdd(dp)
-else:
-for fname in fnames:
-if fname_is_c(fname):
-rules.applyToFile(os.path.join(dirpath,fname))
+for fname in fnames:
+if fname_is_c(fname):
+rules.applyToFile(os.path.join(dirpath,fname))
 
 if trouble:
 err(

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Release 0.5.0

2018-06-26 Thread pastly

commit af8bb20e96b6a1bf7fab2cf3610efde335b44ce7
Author: Matt Traudt 
Date:   Tue Jun 26 11:11:35 2018 -0400

Release 0.5.0

**Important changes**:

- Result format changed, causing a version bump to 4. Updating sbws to 0.5.0
  will cause it to ignore results with version less than 4.

Summary of changes:

- Keep previously-generated v3bw files
- Allow a relay to limit its weight based on
  RelayBandwidthRate/MaxAdvertisedBandwidth
- 1 CPU usage optimization
- 1 memory usage optimization
---
 CHANGELOG.md  | 20 +++-
 sbws/__init__.py  |  2 +-
 tests/unit/lib/data/v3bw/20180425_131057.v3bw |  2 +-
 3 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4b1e1c3..a8bbd36 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,23 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+
+
+## [0.5.0] - 2018-06-26
+
+**Important changes**:
+
+- Result format changed, causing a version bump to 4. Updating sbws to 0.5.0
+  will cause it to ignore results with version less than 4.
+
+Summary of changes:
+
+- Keep previously-generated v3bw files
+- Allow a relay to limit its weight based on
+  RelayBandwidthRate/MaxAdvertisedBandwidth
+- 1 CPU usage optimization
+- 1 memory usage optimization
+
 ### Added
 
 - Use a relay's {,Relay}BandwidthRate/MaxAdvertisedBandwidth as an upper bound
@@ -67,5 +84,6 @@ So fix that.
 - `earliest_bandwidth` being the newest bw not the oldest (thanks juga0)
 - `node_id` was missing the character "$" at the beginning
 
-[Unreleased]: 
https://github.com/pastly/simple-bw-scanner/compare/v0.4.1...master
+[Unreleased]: 
https://github.com/pastly/simple-bw-scanner/compare/v0.5.0...master
+[0.5.0]: https://github.com/pastly/simple-bw-scanner/compare/v0.4.1...v0.5.0
 [0.4.1]: https://github.com/pastly/simple-bw-scanner/compare/v0.4.0...v0.4.1
diff --git a/sbws/__init__.py b/sbws/__init__.py
index 62f6024..2b8877c 100644
--- a/sbws/__init__.py
+++ b/sbws/__init__.py
@@ -1 +1 @@
-__version__ = '0.4.2-dev'
+__version__ = '0.5.0'
diff --git a/tests/unit/lib/data/v3bw/20180425_131057.v3bw 
b/tests/unit/lib/data/v3bw/20180425_131057.v3bw
index 9bb3ffd..df3dcd4 100644
--- a/tests/unit/lib/data/v3bw/20180425_131057.v3bw
+++ b/tests/unit/lib/data/v3bw/20180425_131057.v3bw
@@ -5,6 +5,6 @@ file_created=2018-04-25T13:10:57
 generator_started=2018-04-16T14:09:05
 latest_bandwidth=2018-04-17T14:09:07
 software=sbws
-software_version=0.4.2-dev
+software_version=0.5.0
 
 bw=54 error_circ=0 error_misc=0 error_stream=1 
master_key_ed25519=g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s nick=A 
node_id=$ rtt=456 success=1 
time=2018-04-17T14:09:07



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Make sure we are appropriately blocking when using our list of relays

2018-06-26 Thread pastly

commit b08694693777e4ba850589c5cf7c898e94ffe7b5
Author: Matt Traudt 
Date:   Fri Jun 22 09:55:21 2018 -0400

Make sure we are appropriately blocking when using our list of relays

I'm not sure if this is necessary, honestly. But it doesn't hurt.
---
 sbws/lib/relaylist.py | 9 +++--
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py
index 8b19a7c..6245aa1 100644
--- a/sbws/lib/relaylist.py
+++ b/sbws/lib/relaylist.py
@@ -168,16 +168,13 @@ class RelayList:
 return self._relays_with_flag(Flag.AUTHORITY)
 
 def random_relay(self):
-relays = self.relays
-return self.rng.choice(relays)
+return self.rng.choice(self.relays)
 
 def _relays_with_flag(self, flag):
-relays = self.relays
-return [r for r in relays if flag in r.flags]
+return [r for r in self.relays if flag in r.flags]
 
 def _relays_without_flag(self, flag):
-relays = self.relays
-return [r for r in relays if flag not in r.flags]
+return [r for r in self.relays if flag not in r.flags]
 
 def _init_relays(self):
 c = self._controller



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Make CircuitBuilder use the same RelayList as everyone else

2018-06-26 Thread pastly

commit ada85cb62efd5f918bd379b5b71173f24553735c
Author: Matt Traudt 
Date:   Fri Jun 22 09:51:38 2018 -0400

Make CircuitBuilder use the same RelayList as everyone else

This reduces the number of copies of relay descriptors we have to keep
in memory and how frequently we stop to refresh those descriptors.
---
 sbws/core/scanner.py   | 2 +-
 sbws/lib/circuitbuilder.py | 7 ---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py
index 6af1b0c..25c1dd3 100644
--- a/sbws/core/scanner.py
+++ b/sbws/core/scanner.py
@@ -347,8 +347,8 @@ def run_speedtest(args, conf):
 'even lead to messed up results.', conf['tor']['control_socket'])
 time.sleep(15)
 assert stem_utils.is_controller_okay(controller)
-cb = CB(args, conf, controller)
 rl = RelayList(args, conf, controller)
+cb = CB(args, conf, controller, rl)
 rd = ResultDump(args, conf, end_event)
 rp = RelayPrioritizer(args, conf, rl, rd)
 destinations, error_msg = DestinationList.from_config(
diff --git a/sbws/lib/circuitbuilder.py b/sbws/lib/circuitbuilder.py
index 33df6ec..d55b929 100644
--- a/sbws/lib/circuitbuilder.py
+++ b/sbws/lib/circuitbuilder.py
@@ -2,7 +2,7 @@ from stem import CircuitExtensionFailed, InvalidRequest, 
ProtocolError, Timeout
 from stem import InvalidArguments
 import random
 import sbws.util.stem as stem_utils
-from .relaylist import Relay, RelayList
+from .relaylist import Relay
 import logging
 
 log = logging.getLogger(__name__)
@@ -39,10 +39,11 @@ class CircuitBuilder:
 them, but CircuitBuilder will keep track of existing circuits and close
 them when it is deleted.
 '''
-def __init__(self, args, conf, controller, close_circuits_on_exit=True):
+def __init__(self, args, conf, controller, relay_list,
+ close_circuits_on_exit=True):
 self.controller = controller
 self.rng = random.SystemRandom()
-self.relay_list = RelayList(args, conf, self.controller)
+self.relay_list = relay_list
 self.built_circuits = set()
 self.close_circuits_on_exit = close_circuits_on_exit
 self.circuit_timeout = conf.getint('general', 'circuit_timeout')



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Allow up to 10% error in RelayBandwidthRate test

2018-06-26 Thread pastly

commit b7d8be9b35b6d8b7df1c974d68fac95c2f79e8fd
Author: Matt Traudt 
Date:   Wed Jun 20 14:47:04 2018 -0400

Allow up to 10% error in RelayBandwidthRate test
---
 tests/integration/core/test_scanner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/integration/core/test_scanner.py 
b/tests/integration/core/test_scanner.py
index 9f5bb8c..a9a467f 100644
--- a/tests/integration/core/test_scanner.py
+++ b/tests/integration/core/test_scanner.py
@@ -81,7 +81,7 @@ def test_measure_relay_with_relaybandwidthrate(
 result = result[0]
 assert isinstance(result, ResultSuccess)
 one_mbyte = 1 * 1024 * 1024
-allowed_error = 5  # bytes per second
+allowed_error = 0.1 * one_mbyte  # allow 10% error in either direction
 dls = result.downloads
 for dl in dls:
 assert_within(dl['amount'] / dl['duration'], one_mbyte, allowed_error)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add log.debug around relay refreshing code

2018-06-26 Thread pastly

commit 979c41e1fbb10e7390ab119f09689c095bd2ca12
Author: Matt Traudt 
Date:   Fri Jun 22 10:02:21 2018 -0400

Add log.debug around relay refreshing code
---
 sbws/lib/relaylist.py | 9 +
 1 file changed, 9 insertions(+)

diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py
index 6245aa1..6afa18a 100644
--- a/sbws/lib/relaylist.py
+++ b/sbws/lib/relaylist.py
@@ -136,14 +136,23 @@ class RelayList:
 # See if we can get the list of relays without having to do a refresh,
 # which is expensive and blocks other threads
 if self._need_refresh():
+log.debug('We need to refresh our list of relays. '
+  'Going to wait for lock.')
 # Whelp we couldn't just get the list of relays because the list is
 # stale. Wait for the lock so we can refresh it.
 with self._refresh_lock:
+log.debug('We got the lock. Now to see if we still '
+  'need to refresh.')
 # Now we have the lock ... but wait! Maybe someone else already
 # did the refreshing. So check if it still needs refreshing. If
 # not, we can do nothing.
 if self._need_refresh():
+log.debug('Yup we need to refresh our relays. Doing so.')
 self._refresh()
+else:
+log.debug('No we don\'t need to refresh our relays. '
+  'It was done by someone else.')
+log.debug('Giving back the lock for refreshing relays.')
 assert not self._need_refresh()
 return self._relays
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Catch json decode error when loading results

2018-06-26 Thread pastly

commit 991fe852683257f23d243c5f79809886adb1f7ea
Author: juga0 
Date:   Mon Jun 25 10:10:09 2018 +

Catch json decode error when loading results

log the error, ingore the line and continue
---
 sbws/lib/resultdump.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 9f4f57a..755eb2d 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -44,7 +44,11 @@ def load_result_file(fname, success_only=False):
 with open(fname, 'rt') as fd:
 for line in fd:
 num_total += 1
-r = Result.from_dict(json.loads(line.strip()))
+try:
+r = Result.from_dict(json.loads(line.strip()))
+except json.decoder.JSONDecodeError:
+log.warning('Could not decode result %s', line.strip())
+r = None
 if r is None:
 num_ignored += 1
 continue



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Check that there are results

2018-06-26 Thread pastly

commit f0ebf2702c3f1377abe6abf31afe66fa4991467f
Author: juga0 
Date:   Mon Jun 25 10:29:37 2018 +

Check that there are results

Check that there are results before trying to get the results
for an specific relay. This would avoid raising TypeError
when there are not results.
---
 sbws/lib/resultdump.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 755eb2d..2e51be9 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -497,7 +497,7 @@ class ResultDump:
 self.fresh_days = conf.getint('general', 'data_period')
 self.datadir = conf['paths']['datadir']
 self.end_event = end_event
-self.data = None
+self.data = {}
 self.data_lock = RLock()
 self.thread = Thread(target=self.enter)
 self.queue = Queue()



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add changelog entry for relay list refreshing

2018-06-26 Thread pastly

commit 17a5c9013b6e20b6fb17472b08e23f5d1b620901
Author: Matt Traudt 
Date:   Fri Jun 22 10:05:08 2018 -0400

Add changelog entry for relay list refreshing
---
 CHANGELOG.md | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 18bffbc..524ffab 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,6 +23,12 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 - Make relay priority calculations take only ~5% of the time they used to (3s
   vs 60s) by using sets instead of lists when selecting non-Authority relays.
 (GH#204)
+- Make relay list refreshing take much less time by not allowing worker threads
+  to dogpile on the CPU. Before they would all start requesting descriptors
+from Tor at roughly the same time, causing us to overload our CPU core and make
+the process take unnecessarily long. Now we let one thread do the work so it
+can peg the CPU on its own and get the refresh done ASAP.
+(GH#205)
 
 ### Changed
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add changelog entry for JSON decode exception

2018-06-26 Thread pastly

commit 58df5663cdc75d97b187f678546eec0be05adb36
Author: Matt Traudt 
Date:   Mon Jun 25 12:54:57 2018 -0400

Add changelog entry for JSON decode exception
---
 CHANGELOG.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 524ffab..4b1e1c3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -29,6 +29,8 @@ from Tor at roughly the same time, causing us to overload our 
CPU core and make
 the process take unnecessarily long. Now we let one thread do the work so it
 can peg the CPU on its own and get the refresh done ASAP.
 (GH#205)
+- Catch a JSON decode exception on malformed results so sbws can continue
+  gracefully (GH#210 GHPR#212)
 
 ### Changed
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Make RelayPrioritizer much faster by using sets

2018-06-26 Thread pastly

commit 104fbe7952357ef78b8932b38603ef38fb3dcfab
Author: Matt Traudt 
Date:   Fri Jun 22 09:19:13 2018 -0400

Make RelayPrioritizer much faster by using sets

GH: closes #204
---
 CHANGELOG.md | 6 ++
 sbws/lib/relayprioritizer.py | 5 ++---
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2f9eb22..5026290 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,12 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 - Maintenance script to help us find functions that are (probably) no longer
   being called.
 
+### Fixed
+
+- Make relay priority calculations take only ~5% of the time they used to (3s
+  vs 60s) by using sets instead of lists when selecting non-Authority relays.
+(GH#204)
+
 ### Changed
 
 - Change the path where the Bandwidth List files are generated: now they are
diff --git a/sbws/lib/relayprioritizer.py b/sbws/lib/relayprioritizer.py
index a705213..3a635f0 100644
--- a/sbws/lib/relayprioritizer.py
+++ b/sbws/lib/relayprioritizer.py
@@ -46,10 +46,9 @@ class RelayPrioritizer:
 measurement.
 '''
 fn_tstart = Decimal(time.time())
-relays = copy.deepcopy(self.relay_list.relays)
+relays = set(copy.deepcopy(self.relay_list.relays))
 if not self.measure_authorities:
-relays = [r for r in relays
-  if r not in self.relay_list.authorities]
+relays = relays.difference(set(self.relay_list.authorities))
 rd = self.result_dump
 for relay in relays:
 results = rd.results_for_relay(relay)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Cap results at generate time instead of measurement time

2018-06-26 Thread pastly

commit c7eda42e58a3fb328def64355e093440d4bca1af
Author: Matt Traudt 
Date:   Wed Jun 20 14:33:15 2018 -0400

Cap results at generate time instead of measurement time

This required a bump in the result version: we add a new field to
ResultSuccess.
---
 sbws/core/scanner.py   | 36 
 sbws/globals.py|  2 +-
 sbws/lib/resultdump.py | 14 +++---
 sbws/lib/v3bwfile.py   | 20 +++-
 4 files changed, 31 insertions(+), 41 deletions(-)

diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py
index 2a3aadc..6af1b0c 100644
--- a/sbws/core/scanner.py
+++ b/sbws/core/scanner.py
@@ -165,41 +165,6 @@ def _pick_ideal_second_hop(relay, dest, rl, cont, is_exit):
 return chosen
 
 
-def _clamp_bw_results(relay, bw_results):
-'''
-If a relay has MaxAdvertisedBandwidth set, they may be capable of some
-large amount of bandwidth but prefer if they didn't receive it. We also
-could have managed to measure them faster than their {,Relay}BandwidthRate
-somehow.
-
-See https://github.com/pastly/simple-bw-scanner/issues/155 and
-https://trac.torproject.org/projects/tor/ticket/8494
-'''
-upper_bound = relay.average_bandwidth
-if upper_bound is None:
-log.warning(
-'Could not get average bandwidth from %s\'s descriptor. Not '
-'capping the results for it to some upper bound.', relay.nickname)
-return bw_results
-capped_count = 0
-new_results = []
-for result in bw_results:
-rate = result['amount'] / result['duration']
-if rate > upper_bound:
-capped_count += 1
-new_results.append({
-'amount': int(upper_bound * result['duration']),
-'duration': result['duration']})
-else:
-new_results.append(result)
-assert len(new_results) == len(bw_results)
-if capped_count > 0:
-log.debug(
-'Capped %d results to %d for relay %s', capped_count, upper_bound,
-relay.nickname)
-return new_results
-
-
 def measure_relay(args, conf, destinations, cb, rl, relay):
 s = requests_utils.make_session(
 cb.controller, conf.getfloat('general', 'http_timeout'))
@@ -281,7 +246,6 @@ def measure_relay(args, conf, destinations, cb, rl, relay):
 ResultErrorStream(relay, circ_fps, dest.url, our_nick, msg=msg),
 ]
 cb.close_circuit(circ_id)
-bw_results = _clamp_bw_results(relay, bw_results)
 # Finally: store result
 return [
 ResultSuccess(rtts, bw_results, relay, circ_fps, dest.url, our_nick),
diff --git a/sbws/globals.py b/sbws/globals.py
index e48b378..3ae7d60 100644
--- a/sbws/globals.py
+++ b/sbws/globals.py
@@ -4,7 +4,7 @@ import socket
 
 log = logging.getLogger(__name__)
 
-RESULT_VERSION = 3
+RESULT_VERSION = 4
 WIRE_VERSION = 1
 SPEC_VERSION = '1.1.0'
 
diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 32fd1cd..21d7cc0 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -150,15 +150,18 @@ class Result:
 class Relay:
 ''' Implements just enough of a stem RouterStatusEntryV3 for this
 Result class to be happy '''
-def __init__(self, fingerprint, nickname, address, master_key_ed25519):
+def __init__(self, fingerprint, nickname, address, master_key_ed25519,
+ average_bandwidth=None):
 self.fingerprint = fingerprint
 self.nickname = nickname
 self.address = address
 self.master_key_ed25519 = master_key_ed25519
+self.average_bandwidth = average_bandwidth
 
 def __init__(self, relay, circ, dest_url, scanner_nick, t=None):
 self._relay = Result.Relay(relay.fingerprint, relay.nickname,
-   relay.address, relay.master_key_ed25519)
+   relay.address, relay.master_key_ed25519,
+   relay.average_bandwidth)
 self._circ = circ
 self._dest_url = dest_url
 self._scanner = scanner_nick
@@ -169,6 +172,10 @@ class Result:
 raise NotImplementedError()
 
 @property
+def relay_average_bandwidth(self):
+return self._relay.average_bandwidth
+
+@property
 def fingerprint(self):
 return self._relay.fingerprint
 
@@ -417,7 +424,7 @@ class ResultSuccess(Result):
 d['rtts'], d['downloads'],
 Result.Relay(
 d['fingerprint'], d['nickname'], d['address'],
-d['master_key_ed25519']),
+d['master_key_ed25519'], d['relay_average_bandwidth']),
 d['circ'], d['dest_url'], d['scanner'],
 t=d['time'])
 
@@ -426,6 +433,7 @@ class ResultSuccess(Result):
 d.update({
 'rtts': self.rtts,
 'downloads': self.downloads,
+'relay_average_bandwidth': self.relay_average_bandwidth,
 })

[tor-commits] [sbws/master] Fix tests: CircuitBuilder now needs a RelayList

2018-06-26 Thread pastly

commit e3355d1c567473822c569af8bb5e34f12cee3068
Author: Matt Traudt 
Date:   Sun Jun 24 13:53:54 2018 -0400

Fix tests: CircuitBuilder now needs a RelayList
---
 tests/integration/core/test_scanner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/integration/core/test_scanner.py 
b/tests/integration/core/test_scanner.py
index c1a8692..152e939 100644
--- a/tests/integration/core/test_scanner.py
+++ b/tests/integration/core/test_scanner.py
@@ -28,7 +28,7 @@ def get_everything_to_measure(dotsbws, cont, parser):
 conf['destinations.foo'] = {}
 conf['destinations.foo']['url'] = 'http://127.0.0.1:2/sbws.bin'
 rl = RelayList(args, conf, cont)
-cb = CB(args, conf, cont)
+cb = CB(args, conf, cont, rl)
 dests, error_msg = DestinationList.from_config(conf, cb, rl, cont)
 assert dests, error_msg
 return {



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Remove an assert on _need_refresh()

2018-06-26 Thread pastly

commit ba873130843856100db78ba204314d1be82cb2fd
Author: Matt Traudt 
Date:   Fri Jun 22 10:28:26 2018 -0400

Remove an assert on _need_refresh()

The chance of us needing a refresh at this point is TINY, but it's probably
better to return a list of relays stale by a millisecond than it is to cause
the user's program to crash.
---
 sbws/lib/relaylist.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py
index 6afa18a..ddd982a 100644
--- a/sbws/lib/relaylist.py
+++ b/sbws/lib/relaylist.py
@@ -153,7 +153,6 @@ class RelayList:
 log.debug('No we don\'t need to refresh our relays. '
   'It was done by someone else.')
 log.debug('Giving back the lock for refreshing relays.')
-assert not self._need_refresh()
 return self._relays
 
 @property



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Make flake8 happy

2018-06-26 Thread pastly

commit 1414578689fd1198fc244ba14f9cca8cb596a286
Author: Matt Traudt 
Date:   Fri Jun 22 06:14:59 2018 -0400

Make flake8 happy
---
 tests/integration/lib/test_relayprioritizer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/integration/lib/test_relayprioritizer.py 
b/tests/integration/lib/test_relayprioritizer.py
index 0267843..237a139 100644
--- a/tests/integration/lib/test_relayprioritizer.py
+++ b/tests/integration/lib/test_relayprioritizer.py
@@ -1,5 +1,5 @@
 from sbws.lib.resultdump import ResultDump
-from sbws.lib.resultdump import (Result, ResultSuccess, ResultErrorCircuit)
+from sbws.lib.resultdump import (ResultSuccess, ResultErrorCircuit)
 from sbws.lib.relaylist import RelayList
 from sbws.lib.relayprioritizer import RelayPrioritizer
 from sbws.util.config import get_config



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Bump version to 0.5.1-dev

2018-06-26 Thread pastly

commit f45706093ae3c7a2f2e923b4e7bf370df4cbb13f
Author: Matt Traudt 
Date:   Tue Jun 26 11:35:51 2018 -0400

Bump version to 0.5.1-dev
---
 sbws/__init__.py  | 2 +-
 tests/unit/lib/data/v3bw/20180425_131057.v3bw | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/sbws/__init__.py b/sbws/__init__.py
index 2b8877c..1cb6639 100644
--- a/sbws/__init__.py
+++ b/sbws/__init__.py
@@ -1 +1 @@
-__version__ = '0.5.0'
+__version__ = '0.5.1-dev'
diff --git a/tests/unit/lib/data/v3bw/20180425_131057.v3bw 
b/tests/unit/lib/data/v3bw/20180425_131057.v3bw
index df3dcd4..911d483 100644
--- a/tests/unit/lib/data/v3bw/20180425_131057.v3bw
+++ b/tests/unit/lib/data/v3bw/20180425_131057.v3bw
@@ -5,6 +5,6 @@ file_created=2018-04-25T13:10:57
 generator_started=2018-04-16T14:09:05
 latest_bandwidth=2018-04-17T14:09:07
 software=sbws
-software_version=0.5.0
+software_version=0.5.1-dev
 
 bw=54 error_circ=0 error_misc=0 error_stream=1 
master_key_ed25519=g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s nick=A 
node_id=$ rtt=456 success=1 
time=2018-04-17T14:09:07

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add more tests trimming results when IP changes

2018-06-26 Thread pastly

commit 686a5eefe1c57684deebc530445f2a8ed45f890d
Author: juga0 
Date:   Wed Jun 20 07:54:12 2018 +

Add more tests trimming results when IP changes
---
 tests/unit/lib/test_resultdump.py | 25 +
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/tests/unit/lib/test_resultdump.py 
b/tests/unit/lib/test_resultdump.py
index 26794cb..13b0db8 100644
--- a/tests/unit/lib/test_resultdump.py
+++ b/tests/unit/lib/test_resultdump.py
@@ -26,12 +26,29 @@ RESULTDICT_IP_CHANGED = {FP1: [RESULTSUCCESS1, 
RESULTSUCCESS2]}
 RESULTDICT_IP_NOT_CHANGED = {FP1: [RESULTSUCCESS1, RESULTSUCCESS1]}
 
 
-def test_trim_results_ip_changed():
+def test_trim_results_ip_changed_defaults():
+results_dict = trim_results_ip_changed(RESULTDICT_IP_NOT_CHANGED)
+assert RESULTDICT_IP_NOT_CHANGED == results_dict
+
+
+def test_trim_results_ip_changed_on_changed_ipv4_changed():
 expected_results_dict = {FP1: [RESULTSUCCESS2]}
-results_dict = trim_results_ip_changed(RESULTDICT_IP_CHANGED)
+results_dict = trim_results_ip_changed(RESULTDICT_IP_CHANGED,
+   on_changed_ipv4=True)
 assert expected_results_dict == results_dict
 
 
-def test_trim_results_ip_changed_false():
-results_dict = trim_results_ip_changed(RESULTDICT_IP_CHANGED)
+def test_trim_results_ip_changed_on_changed_ipv4_no_changed():
+results_dict = trim_results_ip_changed(RESULTDICT_IP_NOT_CHANGED,
+   on_changed_ipv4=True)
+assert RESULTDICT_IP_NOT_CHANGED == results_dict
+
+
+def test_trim_results_ip_changed_on_changed_ipv6(caplog):
+results_dict = trim_results_ip_changed(RESULTDICT_IP_NOT_CHANGED,
+   on_changed_ipv6=True)
 assert RESULTDICT_IP_NOT_CHANGED == results_dict
+for record in caplog.records:
+assert record.levelname == 'WARNING'
+assert 'Reseting bandwidth results when IPv6 changes, ' \
+   'is not yet implemented.\n' in caplog.text



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Merge branch 'master' into issue155

2018-06-26 Thread pastly

commit 467f98a97bb829a1cdcfdfe9d107f70a494ee656
Merge: c238a93 104fbe7
Author: Matt Traudt 
Date:   Sat Jun 23 14:20:18 2018 -0400

Merge branch 'master' into issue155

 CHANGELOG.md| 15 +-
 sbws/config.default.ini |  6 +++
 sbws/core/generate.py   |  6 ++-
 sbws/lib/relayprioritizer.py|  5 +-
 sbws/lib/resultdump.py  | 58 +--
 sbws/util/config.py |  8 +++-
 scripts/maint/find-dead-funcs   | 81 +
 scripts/tools/v3bw-into-xy.py   | 53 +
 scripts/tools/v3bw-into-xy.sh   | 26 ---
 tests/unit/lib/test_resultdump.py   | 54 ++
 tests/{ => unit}/util/test_timestamp.py |  4 +-
 11 files changed, 279 insertions(+), 37 deletions(-)

diff --cc CHANGELOG.md
index 6ce42bd,5026290..b3982de
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -9,8 -9,17 +9,19 @@@ and this project adheres to [Semantic V
  
  ### Added
  
 +- Use a relay's {,Relay}BandwidthRate/MaxAdvertisedBandwidth as an upper bound
 +  on the measurements we make for it. (GH#155)
+ - Ability to only consider results for a given relay valid if they came from
+   when that relay is using its most recent known IP address. Thanks Juga.
+ (GH#154 GHPR#199)
+ - Maintenance script to help us find functions that are (probably) no longer
+   being called.
+ 
+ ### Fixed
+ 
+ - Make relay priority calculations take only ~5% of the time they used to (3s
+   vs 60s) by using sets instead of lists when selecting non-Authority relays.
+ (GH#204)
  
  ### Changed
  



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] New config options were still unknown

2018-06-26 Thread pastly

commit cfbae1762a6da383a539d88444e8b6cc05cce264
Author: Matt Traudt 
Date:   Wed Jun 20 09:28:26 2018 -0400

New config options were still unknown
---
 sbws/util/config.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sbws/util/config.py b/sbws/util/config.py
index 2ea9e94..faedbd5 100644
--- a/sbws/util/config.py
+++ b/sbws/util/config.py
@@ -141,7 +141,8 @@ def _validate_general(conf):
 'reset_bw_ipv4_changes': {},
 'reset_bw_ipv6_changes': {},
 }
-all_valid_keys = list(ints.keys()) + list(floats.keys())
+all_valid_keys = list(ints.keys()) + list(floats.keys()) + \
+list(bools.keys())
 errors.extend(_validate_section_keys(conf, sec, all_valid_keys, err_tmpl))
 errors.extend(_validate_section_ints(conf, sec, ints, err_tmpl))
 errors.extend(_validate_section_floats(conf, sec, floats, err_tmpl))



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Remove tests for generate command

2018-06-26 Thread pastly

commit e03fa7224bfdf098f84b2db24a04866509df9313
Author: Matt Traudt 
Date:   Wed Jun 20 20:39:08 2018 -0400

Remove tests for generate command
---
 tests/unit/core/test_generate.py | 284 ---
 1 file changed, 284 deletions(-)

diff --git a/tests/unit/core/test_generate.py b/tests/unit/core/test_generate.py
deleted file mode 100644
index 4dff161..000
--- a/tests/unit/core/test_generate.py
+++ /dev/null
@@ -1,284 +0,0 @@
-# FIXME: all functions that depend on num lines should only use bandwith lines
-# and not whole header bandwith lines, as every time we change headers,
-# tests here would break
-# import pytest
-
-import sbws.core.generate
-from sbws.util.config import get_config
-from sbws.lib.resultdump import load_recent_results_in_datadir
-from sbws.lib.resultdump import ResultSuccess
-from sbws.lib.v3bwfile import NUM_LINES_HEADER_V110, V3BWLine
-from sbws.util.timestamp import unixts_to_isodt_str
-from statistics import median
-import logging
-
-log = logging.getLogger(__name__)
-
-
-def test_generate_no_dotsbws(tmpdir, caplog, parser):
-caplog.set_level(logging.DEBUG)
-dotsbws = tmpdir
-args = parser.parse_args(
-'-d {} --log-level DEBUG generate'.format(dotsbws).split())
-conf = get_config(args)
-try:
-sbws.core.generate.main(args, conf)
-except SystemExit as e:
-assert e.code == 1
-else:
-assert None, 'Should have failed'
-assert 'Try sbws init' in caplog.records[-1].getMessage()
-
-
-def test_generate_no_datadir(empty_dotsbws, caplog, parser):
-dotsbws = empty_dotsbws
-args = parser.parse_args(
-'-d {} --log-level DEBUG generate --output /dev/stdout'
-.format(dotsbws.name).split())
-conf = get_config(args)
-try:
-sbws.core.generate.main(args, conf)
-except SystemExit as e:
-assert e.code == 1
-else:
-assert None, 'Should have failed'
-dd = conf['paths']['datadir']
-assert '{} does not exist'.format(dd) in caplog.records[-1].getMessage()
-
-
-def test_generate_bad_scale_constant(empty_dotsbws_datadir, caplog, parser):
-dotsbws = empty_dotsbws_datadir
-args = parser.parse_args(
-'-d {} --log-level DEBUG generate --scale-constant -1 '
-'--output /dev/stdout'.format(dotsbws.name).split())
-conf = get_config(args)
-try:
-sbws.core.generate.main(args, conf)
-except SystemExit as e:
-assert e.code == 1
-else:
-assert None, 'Should have failed'
-assert '--scale-constant must be positive' in \
-caplog.records[-1].getMessage()
-
-
-def test_generate_empty_datadir(empty_dotsbws_datadir, caplog, parser):
-dotsbws = empty_dotsbws_datadir
-args = parser.parse_args(
-'-d {} --log-level DEBUG generate --output /dev/stdout'
-.format(dotsbws.name).split())
-conf = get_config(args)
-sbws.core.generate.main(args, conf)
-assert 'No recent results' in caplog.records[-1].getMessage()
-
-
-def test_generate_single_error(dotsbws_error_result, caplog, parser):
-caplog.set_level(logging.DEBUG)
-dotsbws = dotsbws_error_result
-args = parser.parse_args(
-'-d {} --log-level DEBUG generate --output /dev/stdout'
-.format(dotsbws.name).split())
-conf = get_config(args)
-sbws.core.generate.main(args, conf)
-dd = conf['paths']['datadir']
-for record in caplog.records:
-if 'Keeping 0/1 read lines from {}'.format(dd) in record.getMessage():
-break
-else:
-assert None, 'Unable to find log line indicating 0 success ' \
-'results in data file'
-assert 'No recent results' in caplog.records[-1].getMessage()
-
-
-def test_generate_single_success_noscale(dotsbws_success_result, caplog,
- parser,  capfd):
-dotsbws = dotsbws_success_result
-args = parser.parse_args(
-'-d {} --log-level DEBUG generate --output /dev/stdout'
-.format(dotsbws.name).split())
-conf = get_config(args)
-sbws.core.generate.main(args, conf)
-dd = conf['paths']['datadir']
-# Here results is a dict
-results = load_recent_results_in_datadir(1, dd, success_only=False)
-assert isinstance(results, dict)
-res_len = sum([len(results[fp]) for fp in results])
-assert res_len == 1, 'There should be one result in the datadir'
-# And here we change it to a list
-results = [r for fp in results for r in results[fp]]
-result = results[0]
-assert isinstance(result, ResultSuccess), 'The one existing result '\
-'should be a success'
-captured = capfd.readouterr()
-stdout_lines = captured.out.strip().split('\n')
-assert len(stdout_lines) == 1 + NUM_LINES_HEADER_V110
-
-bw = round(median([dl['amount'] / dl['duration'] / 1024
-   for dl in result.downloads]))
-rtt = median([round(r * 1000) for r in result.rtts])
-bw_line = V3BWLine('$' +

[tor-commits] [sbws/master] Merge branch 'issue154' from Juga, thanks!

2018-06-26 Thread pastly

commit 2dc6c8602f7fff5d0406799be4eb568fba4da54a
Merge: 5d5fa58 0799326
Author: Matt Traudt 
Date:   Wed Jun 20 09:40:16 2018 -0400

Merge branch 'issue154' from Juga, thanks!

 CHANGELOG.md  |  3 +++
 sbws/config.default.ini   |  6 +
 sbws/core/generate.py |  6 -
 sbws/lib/resultdump.py| 51 +++-
 sbws/util/config.py   |  8 +-
 tests/unit/lib/test_resultdump.py | 54 +++
 6 files changed, 125 insertions(+), 3 deletions(-)

diff --cc CHANGELOG.md
index 16045f6,f0cf0b2..2f9eb22
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -9,8 -9,9 +9,11 @@@ and this project adheres to [Semantic V
  
  ### Added
  
+ - Ability to only consider results for a given relay valid if they came from
+   when that relay is using its most recent known IP address. Thanks Juga.
+ (GH#154 GHPR#199)
 +- Maintenance script to help us find functions that are (probably) no longer
 +  being called.
  
  ### Changed
  



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] log.warn is deprecated, use log.warning

2018-06-26 Thread pastly

commit 16f1ae040709c9063c2f8b43f223b163290fc870
Author: Matt Traudt 
Date:   Wed Jun 20 09:35:15 2018 -0400

log.warn is deprecated, use log.warning
---
 sbws/lib/resultdump.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index aab27e7..feadfdb 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -116,8 +116,8 @@ def trim_results_ip_changed(result_dict, 
on_changed_ipv4=False,
 new_results_dict[fp] = results
 return new_results_dict
 if on_changed_ipv6 is True:
-log.warn("Reseting bandwidth results when IPv6 changes,"
- " is not yet implemented.")
+log.warning("Reseting bandwidth results when IPv6 changes,"
+" is not yet implemented.")
 return result_dict
 
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add integration test for RelayPrioritizer

2018-06-26 Thread pastly

commit b1d591c2cf90fe5bf70861cc289bc2b1ff5eb081
Author: Matt Traudt 
Date:   Thu Jun 21 21:13:48 2018 -0400

Add integration test for RelayPrioritizer
---
 tests/integration/conftest.py  |  4 +-
 tests/integration/lib/test_relayprioritizer.py | 91 ++
 2 files changed, 94 insertions(+), 1 deletion(-)

diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py
index e35341d..94b2433 100644
--- a/tests/integration/conftest.py
+++ b/tests/integration/conftest.py
@@ -4,6 +4,7 @@ from sbws.util.parser import create_parser
 from sbws.util.config import get_config
 from sbws.util.stem import launch_tor
 import sbws.core.init
+import os
 
 
 @pytest.fixture(scope='session')
@@ -14,13 +15,14 @@ def parser():
 @pytest.fixture(scope='session')
 def persistent_empty_dotsbws(parser):
 '''
-Creates a ~/.sbws with nothing in it but a config.ini
+Creates a ~/.sbws with nothing in it but a config.ini and a datadir/
 '''
 d = TemporaryDirectory()
 args = parser.parse_args(
 '-d {} --log-level DEBUG init'.format(d.name).split())
 conf = get_config(args)
 sbws.core.init.main(args, conf)
+os.makedirs(os.path.join(d.name, 'datadir'))
 return d
 
 
diff --git a/tests/integration/lib/test_relayprioritizer.py 
b/tests/integration/lib/test_relayprioritizer.py
new file mode 100644
index 000..0267843
--- /dev/null
+++ b/tests/integration/lib/test_relayprioritizer.py
@@ -0,0 +1,91 @@
+from sbws.lib.resultdump import ResultDump
+from sbws.lib.resultdump import (Result, ResultSuccess, ResultErrorCircuit)
+from sbws.lib.relaylist import RelayList
+from sbws.lib.relayprioritizer import RelayPrioritizer
+from sbws.util.config import get_config
+from threading import Event
+from unittest.mock import patch
+
+
+def static_time(value):
+while True:
+yield value
+
+
+def get_global_stuff(dotsbws, cont, parser):
+args = parser.parse_args(
+'-d {} --log-level DEBUG'.format(dotsbws).split())
+conf = get_config(args)
+rl = RelayList(args, conf, cont)
+return {
+'args': args,
+'conf': conf,
+'rl': rl,
+'end': Event(),
+}
+
+
+def _build_result_for_relay(relay_nick, result_type, timestamp, rl):
+relay = [r for r in rl.relays if r.nickname == relay_nick]
+assert len(relay) == 1
+relay = relay[0]
+other = [r for r in rl.relays if r.nickname != relay_nick][0]
+circ = [relay.fingerprint, other.fingerprint]
+url = 'http://example.com/sbws.bin'
+nick = 'sbws_scanner'
+if result_type == ResultSuccess:
+rtts = [0.5, 0.5, 0.5]
+dls = [
+{'amount': 1024, 'duration': 1},
+{'amount': 1024, 'duration': 1},
+{'amount': 1024, 'duration': 1},
+]
+return ResultSuccess(rtts, dls, relay, circ, url, nick, t=timestamp)
+elif result_type == ResultErrorCircuit:
+return ResultErrorCircuit(
+relay, circ, url, nick, msg='Test error circ message', t=timestamp)
+
+
+@patch('time.time')
+def test_relayprioritizer_general(
+time_mock, persistent_empty_dotsbws, parser, persistent_launch_tor):
+now = 100
+time_mock.side_effect = static_time(now)
+cont = persistent_launch_tor
+dotsbws = persistent_empty_dotsbws.name
+d = get_global_stuff(dotsbws, cont, parser)
+args = d['args']
+conf = d['conf']
+end_event = d['end']
+rl = d['rl']
+rd = ResultDump(args, conf, end_event)
+try:
+rp = RelayPrioritizer(args, conf, rl, rd)
+results = [
+_build_result_for_relay(
+'relay1', ResultSuccess, now - 100, rl),
+_build_result_for_relay(
+'relay2', ResultSuccess, now - 200, rl),
+_build_result_for_relay(
+'relay3', ResultSuccess, now - 300, rl),
+_build_result_for_relay(
+'relay4', ResultSuccess, now - 400, rl),
+_build_result_for_relay(
+'relay5', ResultSuccess, now - 500, rl),
+]
+for result in results:
+rd.store_result(result)
+best_list = [_ for _ in rp.best_priority()]
+# Of the relays for which we have added results to the ResultDump,
+# relay1 has the lowest priority (it has the most recent result) and
+# relay5 has the highest prioirty. The relays that we didn't add
+# results for will have the highest priority, but don't test the order
+# of them. Skip to the end of the list and check those guys since they
+# should have a defined order.
+for i in range(1, 5+1):
+nick = 'relay{}'.format(i)
+pos = i * -1
+relay = best_list[pos]
+assert relay.nickname == nick
+finally:
+end_event.set()



___
tor-commits mailing list
tor-commits@lists.torproject.org

[tor-commits] [sbws/master] Fixup MaxAdvertisedBandwidth test

2018-06-26 Thread pastly

commit d5659732d57089ec2325880e1ce38b997f50b620
Author: Matt Traudt 
Date:   Wed Jun 20 14:47:27 2018 -0400

Fixup MaxAdvertisedBandwidth test
---
 tests/integration/core/test_scanner.py | 11 ---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/tests/integration/core/test_scanner.py 
b/tests/integration/core/test_scanner.py
index a9a467f..c1a8692 100644
--- a/tests/integration/core/test_scanner.py
+++ b/tests/integration/core/test_scanner.py
@@ -4,6 +4,7 @@ from sbws.util.config import get_config
 from sbws.lib.relaylist import RelayList
 from sbws.lib.destination import DestinationList
 from sbws.lib.resultdump import ResultSuccess
+import logging
 
 
 def assert_within(value, target, radius):
@@ -40,7 +41,8 @@ def get_everything_to_measure(dotsbws, cont, parser):
 
 
 def test_measure_relay_with_maxadvertisedbandwidth(
-persistent_launch_tor, parser, persistent_empty_dotsbws):
+persistent_launch_tor, parser, persistent_empty_dotsbws, caplog):
+caplog.set_level(logging.DEBUG)
 cont = persistent_launch_tor
 dotsbws = persistent_empty_dotsbws.name
 d = get_everything_to_measure(dotsbws, cont, parser)
@@ -57,10 +59,13 @@ def test_measure_relay_with_maxadvertisedbandwidth(
 result = result[0]
 assert isinstance(result, ResultSuccess)
 one_mbyte = 1 * 1024 * 1024
-allowed_error = 5  # bytes per second
 dls = result.downloads
 for dl in dls:
-assert_within(dl['amount'] / dl['duration'], one_mbyte, allowed_error)
+# This relay has MaxAdvertisedBandwidth set, but should not be limited
+# to just 1 Mbyte. Assume and assert that all downloads where at least
+# more than 10% faster than 1 MBps
+assert dl['amount'] / dl['duration'] > one_mbyte * 1.1
+assert result.relay_average_bandwidth == one_mbyte
 
 
 def test_measure_relay_with_relaybandwidthrate(



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Merge pull request #191 from pastly/issue155

2018-06-26 Thread pastly

commit cf3e47851895617bfd173c9154526290a1fcd7d3
Merge: 104fbe7 467f98a
Author: juga0 
Date:   Sat Jun 23 18:22:28 2018 +

Merge pull request #191 from pastly/issue155

Implement the "MaxAdvertisedBandwidth" hack ...

 CHANGELOG.md   |   2 +
 sbws/globals.py|   2 +-
 sbws/lib/resultdump.py |  14 +-
 sbws/lib/v3bwfile.py   |  20 ++-
 scripts/tools/sbws-http-server.py  |  76 +
 tests/integration/core/test_scanner.py |  92 +++
 tests/integration/net.tar  | Bin 191488 -> 215040 bytes
 tests/unit/core/test_generate.py   | 284 -
 tests/unit/lib/data/results.txt|   4 +-
 tests/unit/lib/test_results.py |   4 +-
 tests/unit/lib/test_v3bwfile.py|   3 +-
 tox.ini|   5 +-
 12 files changed, 212 insertions(+), 294 deletions(-)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Changelog entry for integration test

2018-06-26 Thread pastly

commit d5ea67c9d99c8b0c2a98f06847204ca9a82cac65
Author: Matt Traudt 
Date:   Thu Jun 21 21:15:20 2018 -0400

Changelog entry for integration test
---
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index b3982de..18bffbc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -16,6 +16,7 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 (GH#154 GHPR#199)
 - Maintenance script to help us find functions that are (probably) no longer
   being called.
+- Integration test(s) for RelayPrioritizer (GHPR#206)
 
 ### Fixed
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Update tests to handle new result format and version

2018-06-26 Thread pastly

commit c238a9387dbd44e79a6e7cd3436d456805e0966d
Author: Matt Traudt 
Date:   Wed Jun 20 20:39:50 2018 -0400

Update tests to handle new result format and version
---
 tests/unit/lib/data/results.txt | 4 ++--
 tests/unit/lib/test_results.py  | 4 +++-
 tests/unit/lib/test_v3bwfile.py | 3 ++-
 3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/tests/unit/lib/data/results.txt b/tests/unit/lib/data/results.txt
index d2005f6..359f213 100644
--- a/tests/unit/lib/data/results.txt
+++ b/tests/unit/lib/data/results.txt
@@ -1,2 +1,2 @@
-{"version": 3, "time": 1523887747, "circ": 
["", 
""], "type": "success", "rtts": 
[0.4596822261810303, 0.44872617721557617, 0.4563450813293457, 
0.44872212409973145, 0.4561030864715576, 0.4765200614929199, 
0.4495084285736084, 0.45711588859558105, 0.45520496368408203, 
0.4635589122772217], "fingerprint": "", 
"scanner": "IDidntEditTheSBWSConfig", "downloads": [{"amount": 590009, 
"duration": 6.1014368534088135}, {"amount": 590009, "duration": 
8.391342878341675}, {"amount": 321663, "duration": 7.064587831497192}, 
{"amount": 321663, "duration": 8.266003131866455}, {"amount": 321663, 
"duration": 5.779450178146362}], "dest_url": "http://y.z;, "nickname": "A", 
"address": "111.111.111.111", "master_key_ed25519": 
"g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s"}
-{"version": 3, "time": 1523974147, "circ": 
["", 
""], "type": "error-stream", "msg": 
"Something bad happened while measuring bandwidth", "fingerprint": 
"", "scanner": 
"IDidntEditTheSBWSConfig", "dest_url": "http://y.z;, "nickname": "A", 
"address": "111.111.111.111", "master_key_ed25519": 
"g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s"}
+{"version": 4, "time": 1523887747, "circ": 
["", 
""], "type": "success", "rtts": 
[0.4596822261810303, 0.44872617721557617, 0.4563450813293457, 
0.44872212409973145, 0.4561030864715576, 0.4765200614929199, 
0.4495084285736084, 0.45711588859558105, 0.45520496368408203, 
0.4635589122772217], "fingerprint": "", 
"scanner": "IDidntEditTheSBWSConfig", "downloads": [{"amount": 590009, 
"duration": 6.1014368534088135}, {"amount": 590009, "duration": 
8.391342878341675}, {"amount": 321663, "duration": 7.064587831497192}, 
{"amount": 321663, "duration": 8.266003131866455}, {"amount": 321663, 
"duration": 5.779450178146362}], "dest_url": "http://y.z;, "nickname": "A", 
"address": "111.111.111.111", "master_key_ed25519": 
"g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s", "relay_average_bandwidth": 
10}
+{"version": 4, "time": 1523974147, "circ": 
["", 
""], "type": "error-stream", "msg": 
"Something bad happened while measuring bandwidth", "fingerprint": 
"", "scanner": 
"IDidntEditTheSBWSConfig", "dest_url": "http://y.z;, "nickname": "A", 
"address": "111.111.111.111", "master_key_ed25519": 
"g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s"}
diff --git a/tests/unit/lib/test_results.py b/tests/unit/lib/test_results.py
index 3d4770e..328d90a 100644
--- a/tests/unit/lib/test_results.py
+++ b/tests/unit/lib/test_results.py
@@ -103,7 +103,8 @@ def test_ResultSuccess_from_dict(time_mock):
 scanner_nick = 'sbwsscanner'
 nick = 'Mooo'
 relay_ip = '169.254.100.1'
-relay = Result.Relay(fp1, nick, relay_ip, ed25519)
+relay_average_bandwidth = 1 * 1024 * 1024
+relay = Result.Relay(fp1, nick, relay_ip, ed25519, relay_average_bandwidth)
 rtts = [5, 25]
 downloads = [{'duration': 4, 'amount': 40}]
 r1 = ResultSuccess(rtts, downloads, relay, circ, dest_url, scanner_nick)
@@ -113,6 +114,7 @@ def test_ResultSuccess_from_dict(time_mock):
 'dest_url': dest_url, 'scanner': scanner_nick,
 'version': RESULT_VERSION, 'type': _ResultType.Success, 'time': t,
 'master_key_ed25519': ed25519,
+'relay_average_bandwidth': relay_average_bandwidth,
 }
 r2 = Result.from_dict(d)
 assert isinstance(r1, ResultSuccess)
diff --git a/tests/unit/lib/test_v3bwfile.py b/tests/unit/lib/test_v3bwfile.py
index 2f4a631..bfe9d9b 100644
--- a/tests/unit/lib/test_v3bwfile.py
+++ b/tests/unit/lib/test_v3bwfile.py
@@ -78,7 +78,8 @@ RESULT_SUCCESS_DICT = {
 {"amount": 321663, "duration": 8.266003131866455},
 {"amount": 321663, "duration": 5.779450178146362}],
 "nickname": "A",
-"master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s"
+"master_key_ed25519": "g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s",
+"relay_average_bandwidth": 1 * 1024 * 1024,
 }
 RESULT_SUCCESS_STR = str(RESULT_SUCCESS_DICT)
 RESULT_ERROR_STREAM_STR =

[tor-commits] [sbws/master] Add changelog entry

2018-06-26 Thread pastly

commit 079932618723daf136dbcb81e70bc9dced9b6c0d
Author: Matt Traudt 
Date:   Wed Jun 20 09:39:21 2018 -0400

Add changelog entry
---
 CHANGELOG.md | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6e40965..f0cf0b2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,12 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+### Added
+
+- Ability to only consider results for a given relay valid if they came from
+  when that relay is using its most recent known IP address. Thanks Juga.
+(GH#154 GHPR#199)
+
 ### Changed
 
 - Change the path where the Bandwidth List files are generated: now they are



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Make sure we only refresh our list of relays once

2018-06-26 Thread pastly

commit 5df7396760a0703e948e97c74e2179b86aef6a90
Author: Matt Traudt 
Date:   Fri Jun 22 09:53:17 2018 -0400

Make sure we only refresh our list of relays once

GH: ref #205
---
 sbws/lib/relaylist.py | 19 +--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/relaylist.py b/sbws/lib/relaylist.py
index f87d3d0..8b19a7c 100644
--- a/sbws/lib/relaylist.py
+++ b/sbws/lib/relaylist.py
@@ -8,6 +8,7 @@ import random
 import time
 import logging
 from sbws.globals import resolve
+from threading import Lock
 
 log = logging.getLogger(__name__)
 
@@ -124,12 +125,26 @@ class RelayList:
 def __init__(self, args, conf, controller):
 self._controller = controller
 self.rng = random.SystemRandom()
+self._refresh_lock = Lock()
 self._refresh()
 
+def _need_refresh(self):
+return time.time() >= self._last_refresh + self.REFRESH_INTERVAL
+
 @property
 def relays(self):
-if time.time() >= self._last_refresh + self.REFRESH_INTERVAL:
-self._refresh()
+# See if we can get the list of relays without having to do a refresh,
+# which is expensive and blocks other threads
+if self._need_refresh():
+# Whelp we couldn't just get the list of relays because the list is
+# stale. Wait for the lock so we can refresh it.
+with self._refresh_lock:
+# Now we have the lock ... but wait! Maybe someone else already
+# did the refreshing. So check if it still needs refreshing. If
+# not, we can do nothing.
+if self._need_refresh():
+self._refresh()
+assert not self._need_refresh()
 return self._relays
 
 @property



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Move timestmap util tests, and fix them to actually pass

2018-06-26 Thread pastly

commit 3aa5444c67b4a05059c73810849d882538c6ebbb
Author: Matt Traudt 
Date:   Wed Jun 20 20:58:27 2018 -0400

Move timestmap util tests, and fix them to actually pass
---
 tests/{ => unit}/util/test_timestamp.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/util/test_timestamp.py b/tests/unit/util/test_timestamp.py
similarity index 86%
rename from tests/util/test_timestamp.py
rename to tests/unit/util/test_timestamp.py
index 0ccf2ec..8ebc2f6 100644
--- a/tests/util/test_timestamp.py
+++ b/tests/unit/util/test_timestamp.py
@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 """Test timestamp conversion util functions"""
-from datetime import datetime
+from datetime import datetime, timezone
 
 from sbws.util.timestamp import (dt_obj_to_isodt_str, unixts_to_dt_obj,
  unixts_to_isodt_str, unixts_to_str)
@@ -8,7 +8,7 @@ from sbws.util.timestamp import (dt_obj_to_isodt_str, 
unixts_to_dt_obj,
 
 isodt_str = '2018-05-23T12:55:04'
 dt_obj = datetime.strptime(isodt_str, '%Y-%m-%dT%H:%M:%S')
-unixts = int(dt_obj.timestamp())
+unixts = int(dt_obj.replace(tzinfo=timezone.utc).timestamp())
 
 
 def test_dt_obj_to_isodt_str():



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add more info to a comment

2018-06-26 Thread pastly

commit eebfd257adb10fbde1e9101aba8680442aec0360
Author: Matt Traudt 
Date:   Wed Jun 20 09:28:18 2018 -0400

Add more info to a comment
---
 sbws/config.default.ini | 10 --
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/sbws/config.default.ini b/sbws/config.default.ini
index d3abcda..80ddc8b 100644
--- a/sbws/config.default.ini
+++ b/sbws/config.default.ini
@@ -23,13 +23,11 @@ http_timeout = 10
 # CircuitBuildTimeout doesn't handle the case of a TLS connection to a relay
 # taking forever, and probably other not-yet-discovered cases.
 circuit_timeout = 10
-# Whether or not to reset the bandwidth measurements when the the relay's IPv4
-# changes
+# Whether or not to reset the bandwidth measurements when the relay's IP
+# address changes. If it changes, we only consider results for the relay that
+# we obtained while the relay was located at its most recent IP address.
+# This is NOT implemented for IPv6.
 reset_bw_ipv4_changes = on
-
-# Whether or not to reset the bandwidth measurements when the the relay's IPv6
-# changes.
-# This is currently not implemented.
 reset_bw_ipv6_changes = off
 
 [scanner]



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Don't let the web server log to stdout/err

2018-06-26 Thread pastly

commit 419f5fbbd339fb201c16819bcc6fd3a053880b01
Author: Matt Traudt 
Date:   Tue Jun 19 22:17:35 2018 -0400

Don't let the web server log to stdout/err
---
 tox.ini | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tox.ini b/tox.ini
index c905acc..2a34348 100644
--- a/tox.ini
+++ b/tox.ini
@@ -28,7 +28,7 @@ commands =
 tar -C {envtmpdir} -vxf {toxinidir}/tests/integration/net.tar
 bash {envtmpdir}/net/start.sh
 bash -c "time python3 {envtmpdir}/net/wait.py 
{envtmpdir}/net/{auth,relay,exit}*"
-bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 &"
+bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 &>/dev/null &"
 sleep 30
 wget -O/dev/null http://127.0.0.1:2/sbws.bin
 coverage run -a --rcfile={toxinidir}/.coveragerc --source=sbws -m pytest 
-s {toxinidir}/tests/integration -vv



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Low warn on trying to reset bw on IPv6 changes

2018-06-26 Thread pastly

commit 6b3718989afc57c244e11ff4d85846f1927fa01b
Author: juga0 
Date:   Tue Jun 19 19:51:45 2018 +

Low warn on trying to reset bw on IPv6 changes
---
 sbws/lib/resultdump.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 02945fa..cadc825 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -116,8 +116,8 @@ def trim_results_ip_changed(result_dict, 
on_changed_ipv4=False,
 else:
 new_results_dict[fp] = results
 if on_changed_ipv6 is True:
-# Not implemented
-pass
+log.warn("Reseting bandwidth results when IPv6 changes,"
+ " is not yet implemented.")
 return new_results_dict
 
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add relay1mbyteRBR to integration testnet

2018-06-26 Thread pastly

commit 35acd72a487f31efe495acfe2bc64e34b0ff7967
Author: Matt Traudt 
Date:   Tue Jun 19 22:19:00 2018 -0400

Add relay1mbyteRBR to integration testnet

Has 'RelayBandwidthRate 1 MByte' set
---
 tests/integration/net.tar | Bin 204800 -> 215040 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/tests/integration/net.tar b/tests/integration/net.tar
index cde1f17..2870690 100644
Binary files a/tests/integration/net.tar and b/tests/integration/net.tar differ



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Test requesting something from http server on Travis

2018-06-26 Thread pastly

commit 5d27288ae1e8bb9d5d260b790235469ae8175350
Author: Matt Traudt 
Date:   Sun Jun 17 21:33:19 2018 -0400

Test requesting something from http server on Travis
---
 tox.ini | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tox.ini b/tox.ini
index 69371bd..c905acc 100644
--- a/tox.ini
+++ b/tox.ini
@@ -23,16 +23,16 @@ whitelist_externals =
 tar
 bash
 sleep
-cat
+wget
 commands =
 tar -C {envtmpdir} -vxf {toxinidir}/tests/integration/net.tar
 bash {envtmpdir}/net/start.sh
 bash -c "time python3 {envtmpdir}/net/wait.py 
{envtmpdir}/net/{auth,relay,exit}*"
-bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 > {toxinidir}/http-server.log &"
+bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 &"
 sleep 30
+wget -O/dev/null http://127.0.0.1:2/sbws.bin
 coverage run -a --rcfile={toxinidir}/.coveragerc --source=sbws -m pytest 
-s {toxinidir}/tests/integration -vv
 bash {envtmpdir}/net/stop.sh
-cat {toxinidir}/http-server.log
 
 [testenv:lint]
 skip_install = True



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Change args' names for reseting bw on IP changes

2018-06-26 Thread pastly

commit d56ec60c41e48381da7ee3b9d52f94cb6fe2f50e
Author: juga0 
Date:   Tue Jun 19 19:48:33 2018 +

Change args' names for reseting bw on IP changes

to be more descriptive.
Also default them to False.
---
 sbws/core/generate.py  |  5 +++--
 sbws/lib/resultdump.py | 21 +
 2 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/sbws/core/generate.py b/sbws/core/generate.py
index 69c224d..827058e 100644
--- a/sbws/core/generate.py
+++ b/sbws/core/generate.py
@@ -74,8 +74,9 @@ def main(args, conf):
 reset_bw_ipv4_changes = conf.getboolean('general', 'reset_bw_ipv4_changes')
 reset_bw_ipv6_changes = conf.getboolean('general', 'reset_bw_ipv6_changes')
 results = load_recent_results_in_datadir(
-fresh_days, datadir, success_only=True, ipv4=reset_bw_ipv4_changes,
-ipv6=reset_bw_ipv6_changes)
+fresh_days, datadir, success_only=True,
+on_changed_ipv4=reset_bw_ipv4_changes,
+on_changed_ipv6=reset_bw_ipv6_changes)
 if len(results) < 1:
 log.warning('No recent results, so not generating anything. (Have you '
 'ran sbws scanner recently?)')
diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 2124cec..02945fa 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -82,20 +82,23 @@ def trim_results(fresh_days, result_dict):
 return out_results
 
 
-def trim_results_ip_changed(result_dict, ipv4=True, ipv6=False):
+def trim_results_ip_changed(result_dict, on_changed_ipv4=False,
+on_changed_ipv6=False):
 """When there are results for the same relay with different IPs,
 create a new results' dictionary without that relay's results using an
 older IP.
 
 :param dict result_dict: a dictionary of results
-:param bool ipv4: whether to trim the results when a relay's IPv4 changes
-:param bool ipv6: whether to trim the results when a relay's IPv6 changes
+:param bool on_changed_ipv4: whether to trim the results when a relay's
+IPv4 changes
+:param bool on_changed_ipv6: whether to trim the results when a relay's
+IPv6 changes
 :returns: a new results dictionary
 """
 assert isinstance(result_dict, dict)
-assert ipv4 is True or ipv6 is True
+assert on_changed_ipv4 is True or on_changed_ipv6 is True
 new_results_dict = {}
-if ipv4 is True:
+if on_changed_ipv4 is True:
 for fp in result_dict.keys():
 results = result_dict[fp]
 # find if the results for a relay have more than one ipv4
@@ -112,14 +115,15 @@ def trim_results_ip_changed(result_dict, ipv4=True, 
ipv6=False):
 new_results_dict[fp] = last_ip_results
 else:
 new_results_dict[fp] = results
-if ipv6 is True:
+if on_changed_ipv6 is True:
 # Not implemented
 pass
 return new_results_dict
 
 
 def load_recent_results_in_datadir(fresh_days, datadir, success_only=False,
-   ipv4=True, ipv6=False):
+   on_changed_ipv4=False,
+   on_changed_ipv6=False):
 ''' Given a data directory, read all results files in it that could have
 results in them that are still valid. Trim them, and return the valid
 Results as a list '''
@@ -146,7 +150,8 @@ def load_recent_results_in_datadir(fresh_days, datadir, 
success_only=False,
 results = trim_results(fresh_days, results)
 # in time fresh days is possible that a relay changed ip,
 # if that's the case, keep only the results for the last ip
-results = trim_results_ip_changed(results, ipv4, ipv6)
+results = trim_results_ip_changed(results, on_changed_ipv4,
+  on_changed_ipv6)
 num_res = sum([len(results[fp]) for fp in results])
 if num_res == 0:
 log.warning('Results files that are valid not found. '



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add find-dead-funcs maint script

2018-06-26 Thread pastly

commit 5a445c608ff4b00bb190799af2843e80ef2bbf90
Author: Matt Traudt 
Date:   Tue Jun 19 12:01:03 2018 -0400

Add find-dead-funcs maint script

GH: ref #143
---
 CHANGELOG.md  |  5 +++
 scripts/maint/find-dead-funcs | 81 +++
 2 files changed, 86 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6e40965..18708da 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,11 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+### Added
+
+- Maintenance script to help us find functions that are (probably) no longer
+  being called.
+
 ### Changed
 
 - Change the path where the Bandwidth List files are generated: now they are
diff --git a/scripts/maint/find-dead-funcs b/scripts/maint/find-dead-funcs
new file mode 100755
index 000..aa81d68
--- /dev/null
+++ b/scripts/maint/find-dead-funcs
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+
+SOURCE_DIR=sbws
+TESTS_DIR=tests
+
+UNUSED_LOG=unused-funcs.txt
+USED_LOG=used-funcs.txt
+
+cat > $USED_LOG <> $UNUSED_LOG
+else
+echo "$FUNC" >> $USED_LOG
+echo "$USAGE_INFO" >> $USED_LOG
+fi
+done
+}
+
+echo "Writing unused function names to $UNUSED_LOG"
+echo "Writing used function names and information to $USED_LOG"
+
+get_all_funcs | get_func_usage_count



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Return trimmed results only when arg is True

2018-06-26 Thread pastly

commit b2c60d0d6d4a37a75a864e8127e8761835b3f41a
Author: juga0 
Date:   Wed Jun 20 07:53:19 2018 +

Return trimmed results only when arg is True
---
 sbws/lib/resultdump.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 31a8926..aab27e7 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -96,7 +96,6 @@ def trim_results_ip_changed(result_dict, 
on_changed_ipv4=False,
 :returns: a new results dictionary
 """
 assert isinstance(result_dict, dict)
-assert on_changed_ipv4 is True or on_changed_ipv6 is True
 new_results_dict = {}
 if on_changed_ipv4 is True:
 for fp in result_dict.keys():
@@ -115,10 +114,11 @@ def trim_results_ip_changed(result_dict, 
on_changed_ipv4=False,
 new_results_dict[fp] = last_ip_results
 else:
 new_results_dict[fp] = results
+return new_results_dict
 if on_changed_ipv6 is True:
 log.warn("Reseting bandwidth results when IPv6 changes,"
  " is not yet implemented.")
-return new_results_dict
+return result_dict
 
 
 def load_recent_results_in_datadir(fresh_days, datadir, success_only=False,



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Clarify why not trimming results on IP changes

2018-06-26 Thread pastly

commit 59f6d6089d938f9530838be449c9ad5c220f983e
Author: juga0 
Date:   Tue Jun 19 19:58:36 2018 +

Clarify why not trimming results on IP changes

when storing the results.
---
 sbws/lib/resultdump.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index cadc825..31a8926 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -500,9 +500,10 @@ class ResultDump:
 self.data[fp] = []
 self.data[fp].append(result)
 self.data = trim_results(self.fresh_days, self.data)
-# we probably do not want to remove the results for a relay
-# that has changed address when storing the results
-# it will be already done when loading
+# Not calling trim_results_ip_changed here to do not remove
+# the results for a relay that has changed address.
+# It will be called when loading the results to generate a v3bw
+# file.
 
 def handle_result(self, result):
 ''' Call from ResultDump thread. If we are shutting down, ignores



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add date to changelog header

2018-06-26 Thread pastly

commit 5d5fa588c3121abe2ab056a31a7d676a82f462c4
Author: Matt Traudt 
Date:   Tue Jun 19 12:02:10 2018 -0400

Add date to changelog header
---
 CHANGELOG.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 18708da..16045f6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,7 +21,7 @@ generated ones are kept. A `latest.v3bw` symlink is updated. 
(GH#179 GHPR#190)
 - Replace v3bw-into-xy bash script with python script to handle a more complex
   v3bw file format (GH#182)
 
-## [0.4.1]
+## [0.4.1] - 2018-06-14
 
 ### Changed
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Python 3.4 doesn't have HTTPStatus

2018-06-26 Thread pastly

commit 815afcd6eacc1ba83669c1e7433e399f7a63f15b
Author: Matt Traudt 
Date:   Sun Jun 17 21:30:40 2018 -0400

Python 3.4 doesn't have HTTPStatus
---
 scripts/tools/sbws-http-server.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/tools/sbws-http-server.py 
b/scripts/tools/sbws-http-server.py
index d8575d0..f21499f 100755
--- a/scripts/tools/sbws-http-server.py
+++ b/scripts/tools/sbws-http-server.py
@@ -19,7 +19,6 @@
 # Don't breathe too hard or this script might break.
 from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
 import http.server
-from http import HTTPStatus
 # import time
 
 FILE_SIZE = 1*1024*1024*1024  # 1 GiB
@@ -39,7 +38,7 @@ class 
MyHTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
 super().__init__(*a, **kw)
 
 def send_head(self, length):
-self.send_response(HTTPStatus.OK)
+self.send_response(200)
 self.send_header('Content-Type', 'application/octet-stream')
 self.send_header('Content-Length', length)
 # self.send_header('Last-Modified', self.date_time_string(time.time()))



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Call trim_results_ip_changed when loading results

2018-06-26 Thread pastly

commit c00bf3f760d7abcafb1ab6b1a98f2ebea63f577b
Author: juga0 
Date:   Mon Jun 18 13:31:03 2018 +

Call trim_results_ip_changed when loading results
---
 sbws/lib/resultdump.py | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index d64d843..2124cec 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -144,6 +144,9 @@ def load_recent_results_in_datadir(fresh_days, datadir, 
success_only=False,
 results = merge_result_dicts(results, new_results)
 working_day += timedelta(days=1)
 results = trim_results(fresh_days, results)
+# in time fresh days is possible that a relay changed ip,
+# if that's the case, keep only the results for the last ip
+results = trim_results_ip_changed(results, ipv4, ipv6)
 num_res = sum([len(results[fp]) for fp in results])
 if num_res == 0:
 log.warning('Results files that are valid not found. '
@@ -492,6 +495,9 @@ class ResultDump:
 self.data[fp] = []
 self.data[fp].append(result)
 self.data = trim_results(self.fresh_days, self.data)
+# we probably do not want to remove the results for a relay
+# that has changed address when storing the results
+# it will be already done when loading
 
 def handle_result(self, result):
 ''' Call from ResultDump thread. If we are shutting down, ignores



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Remove v3bw file link if it exist

2018-06-26 Thread pastly

commit 75c37b818138d74ce5174dad4112619b797375ed
Author: juga0 
Date:   Thu Jun 14 16:30:54 2018 +

Remove v3bw file link if it exist

also fix link path
---
 sbws/lib/v3bwfile.py | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index 1294d99..40a8fd5 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -402,14 +402,21 @@ class V3BwFile(object):
 
 def write(self, output):
 log.info('Writing v3bw file to %s', output)
+out_dir = os.path.dirname(output)
+out_link = os.path.join(out_dir, 'latest.v3bw')
+if os.path.exists(out_link):
+log.debug('Deleting existing symlink before creating a new one.')
+os.remove(out_link)
 # to keep test_generate.py working
 if output != '/dev/stdout':
-with DirectoryLock(os.path.dirname(output)):
+with DirectoryLock(out_dir):
 with open(output, 'wt') as fd:
 fd.write(str(self.header))
 for line in self.bw_lines:
 fd.write(str(line))
-os.symlink(output, 'latest.v3bw')
+log.debug('Creating symlink from {} to {}.'
+  .format(output, out_link))
+os.symlink(output, out_link)
 else:
 with open(output, 'wt') as fd:
 fd.write(str(self.header))



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Improve tests for reseting bw on IP changes

2018-06-26 Thread pastly

commit 91d9e90f099ff94e18d350697e8872a4ab0c61cc
Author: juga0 
Date:   Tue Jun 19 20:09:51 2018 +

Improve tests for reseting bw on IP changes
---
 tests/unit/lib/test_resultdump.py | 18 --
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/tests/unit/lib/test_resultdump.py 
b/tests/unit/lib/test_resultdump.py
index 05a21ae..26794cb 100644
--- a/tests/unit/lib/test_resultdump.py
+++ b/tests/unit/lib/test_resultdump.py
@@ -11,10 +11,10 @@ ED25519 = 'g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s'
 CIRC = [FP1, FP2]
 DEST_URL = 'http://example.com/sbws.bin'
 NICK = 'A'
-RELAY_ip1 = '169.254.100.1'
-RELAY_ip2 = '169.254.100.2'
-RELAY1 = Result.Relay(FP1, NICK, RELAY_ip1, ED25519)
-RELAY2 = Result.Relay(FP1, NICK, RELAY_ip2, ED25519)
+RELAY_IP1 = '169.254.100.1'
+RELAY_IP2 = '169.254.100.2'
+RELAY1 = Result.Relay(FP1, NICK, RELAY_IP1, ED25519)
+RELAY2 = Result.Relay(FP1, NICK, RELAY_IP2, ED25519)
 RTTS = [5, 25]
 DOWNLOADS = [{'duration': 4, 'amount': 40}]
 
@@ -22,10 +22,16 @@ RESULTSUCCESS1 = ResultSuccess(RTTS, DOWNLOADS, RELAY1, 
CIRC, DEST_URL,
'sbws', t=TIME1)
 RESULTSUCCESS2 = ResultSuccess(RTTS, DOWNLOADS, RELAY2, CIRC, DEST_URL,
'sbws', t=TIME2)
-RESULTDICT = {FP1: [RESULTSUCCESS1, RESULTSUCCESS2]}
+RESULTDICT_IP_CHANGED = {FP1: [RESULTSUCCESS1, RESULTSUCCESS2]}
+RESULTDICT_IP_NOT_CHANGED = {FP1: [RESULTSUCCESS1, RESULTSUCCESS1]}
 
 
 def test_trim_results_ip_changed():
 expected_results_dict = {FP1: [RESULTSUCCESS2]}
-results_dict = trim_results_ip_changed(RESULTDICT)
+results_dict = trim_results_ip_changed(RESULTDICT_IP_CHANGED)
 assert expected_results_dict == results_dict
+
+
+def test_trim_results_ip_changed_false():
+results_dict = trim_results_ip_changed(RESULTDICT_IP_CHANGED)
+assert RESULTDICT_IP_NOT_CHANGED == results_dict



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Implement the "MaxAdvertisedBandwidth" hack ...

2018-06-26 Thread pastly

commit 068d521002e0858d91c2fc65d4c16b1c226d8a1b
Author: Matt Traudt 
Date:   Thu Jun 14 19:49:19 2018 -0400

Implement the "MaxAdvertisedBandwidth" hack ...

Use a relay's {,Relay}BandwidthRate/MaxAdvertisedBandwidth as an
upper bound on the measurements we make for it.
---
 CHANGELOG.md |  5 +
 sbws/core/scanner.py | 36 
 2 files changed, 41 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3a79c02..6ce42bd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,11 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+### Added
+
+- Use a relay's {,Relay}BandwidthRate/MaxAdvertisedBandwidth as an upper bound
+  on the measurements we make for it. (GH#155)
+
 ### Changed
 
 - Change the path where the Bandwidth List files are generated: now they are
diff --git a/sbws/core/scanner.py b/sbws/core/scanner.py
index 6af1b0c..2a3aadc 100644
--- a/sbws/core/scanner.py
+++ b/sbws/core/scanner.py
@@ -165,6 +165,41 @@ def _pick_ideal_second_hop(relay, dest, rl, cont, is_exit):
 return chosen
 
 
+def _clamp_bw_results(relay, bw_results):
+'''
+If a relay has MaxAdvertisedBandwidth set, they may be capable of some
+large amount of bandwidth but prefer if they didn't receive it. We also
+could have managed to measure them faster than their {,Relay}BandwidthRate
+somehow.
+
+See https://github.com/pastly/simple-bw-scanner/issues/155 and
+https://trac.torproject.org/projects/tor/ticket/8494
+'''
+upper_bound = relay.average_bandwidth
+if upper_bound is None:
+log.warning(
+'Could not get average bandwidth from %s\'s descriptor. Not '
+'capping the results for it to some upper bound.', relay.nickname)
+return bw_results
+capped_count = 0
+new_results = []
+for result in bw_results:
+rate = result['amount'] / result['duration']
+if rate > upper_bound:
+capped_count += 1
+new_results.append({
+'amount': int(upper_bound * result['duration']),
+'duration': result['duration']})
+else:
+new_results.append(result)
+assert len(new_results) == len(bw_results)
+if capped_count > 0:
+log.debug(
+'Capped %d results to %d for relay %s', capped_count, upper_bound,
+relay.nickname)
+return new_results
+
+
 def measure_relay(args, conf, destinations, cb, rl, relay):
 s = requests_utils.make_session(
 cb.controller, conf.getfloat('general', 'http_timeout'))
@@ -246,6 +281,7 @@ def measure_relay(args, conf, destinations, cb, rl, relay):
 ResultErrorStream(relay, circ_fps, dest.url, our_nick, msg=msg),
 ]
 cb.close_circuit(circ_id)
+bw_results = _clamp_bw_results(relay, bw_results)
 # Finally: store result
 return [
 ResultSuccess(rtts, bw_results, relay, circ_fps, dest.url, our_nick),



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Generate bandwidth file without relays' old IPs

2018-06-26 Thread pastly

commit 64a33f63f91549a427ba16c52c75dd419c6079e7
Author: juga0 
Date:   Mon Jun 18 13:32:58 2018 +

Generate bandwidth file without relays' old IPs
---
 sbws/core/generate.py | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sbws/core/generate.py b/sbws/core/generate.py
index 25653b1..69c224d 100644
--- a/sbws/core/generate.py
+++ b/sbws/core/generate.py
@@ -71,8 +71,11 @@ def main(args, conf):
 fail_hard('--scale-constant must be positive')
 
 fresh_days = conf.getint('general', 'data_period')
+reset_bw_ipv4_changes = conf.getboolean('general', 'reset_bw_ipv4_changes')
+reset_bw_ipv6_changes = conf.getboolean('general', 'reset_bw_ipv6_changes')
 results = load_recent_results_in_datadir(
-fresh_days, datadir, success_only=True)
+fresh_days, datadir, success_only=True, ipv4=reset_bw_ipv4_changes,
+ipv6=reset_bw_ipv6_changes)
 if len(results) < 1:
 log.warning('No recent results, so not generating anything. (Have you '
 'ran sbws scanner recently?)')



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Replace v3bw-into-xy bash script with python script

2018-06-26 Thread pastly

commit 09691a0fe7b3809f2cdacd7713c2b37668c6c93b
Author: Matt Traudt 
Date:   Thu Jun 14 22:23:22 2018 -0400

Replace v3bw-into-xy bash script with python script

GH: ref #182
---
 CHANGELOG.md  |  2 ++
 scripts/tools/v3bw-into-xy.py | 53 +++
 scripts/tools/v3bw-into-xy.sh | 26 -
 3 files changed, 55 insertions(+), 26 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3a79c02..6e40965 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,8 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
   stored in `v3bw` directory, named `YYmmdd_HHMMSS.v3bw`, and previously
 generated ones are kept. A `latest.v3bw` symlink is updated. (GH#179 GHPR#190)
 - Code refactoring in the v3bw classes and generation area
+- Replace v3bw-into-xy bash script with python script to handle a more complex
+  v3bw file format (GH#182)
 
 ## [0.4.1]
 
diff --git a/scripts/tools/v3bw-into-xy.py b/scripts/tools/v3bw-into-xy.py
new file mode 100755
index 000..fda5d74
--- /dev/null
+++ b/scripts/tools/v3bw-into-xy.py
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+import sys
+import re
+# File: v3bw-into-xy.py
+# Author: Matt Traudt
+# License: CC0
+#
+# Takes one or more v3bw files as arguments.
+#
+# Looks for lines that contain actual data. That means most of them, since most
+# of them contain "node_id=" and those are the ones that are interesting.
+#
+# Extract the fingerprint and bandwidth values for each of those lines and put
+# them on stdout, one per line. Effectively, after ignoring other lines, this:
+# node_id=$... bw=12345
+# becomes this:
+# ... 12345
+#
+# NOTE: If you specify more than v3bw file, this will do NOTHING to tell you
+# when the output from one file stops and the next begins
+#
+# With v1.1.0 of the v3bw file format, we no longer know if node_id or bw is
+# first in the line. Hence two regular expresions and searching for the matched
+# item that has 40 chars (the fingerprint)
+
+
+def main():
+re1 = re.compile('.*node_id=\$?([\w]+).* bw=([\d]+).*')  # noqa
+re2 = re.compile('.*bw=([\d]+).* node_id=\$?([\w]+)')  # noqa
+for fname in sys.argv[1:]:
+with open(fname, 'rt') as fd:
+for line in fd:
+if 'node_id' not in line:
+continue
+match = re1.match(line) or re2.match(line)
+if not match:
+continue
+items = match.groups()
+assert len(items) == 2
+s = '{} {}\n'
+if len(items[0]) == 40:
+s = s.format(*items)
+else:
+s = s.format(*items[::-1])
+sys.stdout.write(s)
+return 0
+
+
+if __name__ == '__main__':
+try:
+exit(main())
+except (KeyboardInterrupt, BrokenPipeError):
+pass
diff --git a/scripts/tools/v3bw-into-xy.sh b/scripts/tools/v3bw-into-xy.sh
deleted file mode 100755
index 059739d..000
--- a/scripts/tools/v3bw-into-xy.sh
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/usr/bin/env bash
-# File: v3bw-into-xy.sh
-# Author: Matt Traudt
-# License: CC0
-#
-# Takes one or more v3bw files as arguments.
-#
-# Looks for lines that contain actual data. That means most of them, since most
-# of them start with "node_id=" and those are the ones that are interesting.
-#
-# Extract the fingerprint and bandwidth values for each of those lines and put
-# them on stdout, one per line. Effectively, after ignoring other lines, this:
-# node_id=$... bw=12345
-# becomes this:
-# ... 12345
-#
-# NOTE: If you specify more than v3bw file, this will do NOTHING to tell you
-# when the output from one file stops and the next begins
-set -e
-while [ "$1" != "" ]
-do
-grep '^node_id=' "$1" |
-   sed -r 's|^node_id=([$A-Z0-9]+) bw=([0-9]+).*$|\1 \2|' |
-   sed 's|\$||g'
-shift
-done



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add config options for trimming results when ip changes

2018-06-26 Thread pastly

commit d4d4aab79254171e73237bc0571ed3a7899331e5
Author: juga0 
Date:   Mon Jun 18 13:32:17 2018 +

Add config options for trimming results when ip changes
---
 sbws/config.default.ini | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/sbws/config.default.ini b/sbws/config.default.ini
index faa0ab1..ff72097 100644
--- a/sbws/config.default.ini
+++ b/sbws/config.default.ini
@@ -23,6 +23,8 @@ http_timeout = 10
 # CircuitBuildTimeout doesn't handle the case of a TLS connection to a relay
 # taking forever, and probably other not-yet-discovered cases.
 circuit_timeout = 10
+reset_bw_ipv4_changes = on
+reset_bw_ipv6_changes = off
 
 [scanner]
 # A human-readable string with chars in a-zA-Z0-9 to identify your scanner



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add comment to config vars for reseting bw on IP

2018-06-26 Thread pastly

commit d25799f08378ed66615098946fbca100fe2df67d
Author: juga0 
Date:   Tue Jun 19 19:34:15 2018 +

Add comment to config vars for reseting bw on IP

changes.
---
 sbws/config.default.ini | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/sbws/config.default.ini b/sbws/config.default.ini
index ff72097..d3abcda 100644
--- a/sbws/config.default.ini
+++ b/sbws/config.default.ini
@@ -23,7 +23,13 @@ http_timeout = 10
 # CircuitBuildTimeout doesn't handle the case of a TLS connection to a relay
 # taking forever, and probably other not-yet-discovered cases.
 circuit_timeout = 10
+# Whether or not to reset the bandwidth measurements when the the relay's IPv4
+# changes
 reset_bw_ipv4_changes = on
+
+# Whether or not to reset the bandwidth measurements when the the relay's IPv6
+# changes.
+# This is currently not implemented.
 reset_bw_ipv6_changes = off
 
 [scanner]



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Some integration tests for measuring relays (probably will fail)

2018-06-26 Thread pastly

commit c6c7ed1a6c5db840a8ca59ede892874ae41e6d6d
Author: Matt Traudt 
Date:   Tue Jun 19 22:19:53 2018 -0400

Some integration tests for measuring relays (probably will fail)
---
 tests/integration/core/test_scanner.py | 87 ++
 1 file changed, 87 insertions(+)

diff --git a/tests/integration/core/test_scanner.py 
b/tests/integration/core/test_scanner.py
new file mode 100644
index 000..9f5bb8c
--- /dev/null
+++ b/tests/integration/core/test_scanner.py
@@ -0,0 +1,87 @@
+from sbws.core.scanner import measure_relay
+from sbws.lib.circuitbuilder import GapsCircuitBuilder as CB
+from sbws.util.config import get_config
+from sbws.lib.relaylist import RelayList
+from sbws.lib.destination import DestinationList
+from sbws.lib.resultdump import ResultSuccess
+
+
+def assert_within(value, target, radius):
+'''
+Assert that **value** is within **radius** of **target**
+
+If target is 10 and radius is 2, value can be anywhere between 8 and 12
+inclusive
+'''
+assert target - radius < value, 'Value is too small. {} is not within '\
+'{} of {}'.format(value, radius, target)
+assert target + radius > value, 'Value is too big. {} is not within '\
+'{} of {}'.format(value, radius, target)
+
+
+def get_everything_to_measure(dotsbws, cont, parser):
+args = parser.parse_args(
+'-d {} --log-level DEBUG scanner'.format(dotsbws).split())
+conf = get_config(args)
+conf['destinations']['foo'] = 'on'
+conf['destinations.foo'] = {}
+conf['destinations.foo']['url'] = 'http://127.0.0.1:2/sbws.bin'
+rl = RelayList(args, conf, cont)
+cb = CB(args, conf, cont)
+dests, error_msg = DestinationList.from_config(conf, cb, rl, cont)
+assert dests, error_msg
+return {
+'args': args,
+'conf': conf,
+'rl': rl,
+'cb': cb,
+'dests': dests
+}
+
+
+def test_measure_relay_with_maxadvertisedbandwidth(
+persistent_launch_tor, parser, persistent_empty_dotsbws):
+cont = persistent_launch_tor
+dotsbws = persistent_empty_dotsbws.name
+d = get_everything_to_measure(dotsbws, cont, parser)
+args = d['args']
+conf = d['conf']
+rl = d['rl']
+dests = d['dests']
+cb = d['cb']
+# 117A456C94076BEB4E757AC48B16CC0CCC5F is relay1mbyteMAB
+relay = [r for r in rl.relays
+ if r.nickname == 'relay1mbyteMAB'][0]
+result = measure_relay(args, conf, dests, cb, rl, relay)
+assert len(result) == 1
+result = result[0]
+assert isinstance(result, ResultSuccess)
+one_mbyte = 1 * 1024 * 1024
+allowed_error = 5  # bytes per second
+dls = result.downloads
+for dl in dls:
+assert_within(dl['amount'] / dl['duration'], one_mbyte, allowed_error)
+
+
+def test_measure_relay_with_relaybandwidthrate(
+persistent_launch_tor, parser, persistent_empty_dotsbws):
+cont = persistent_launch_tor
+dotsbws = persistent_empty_dotsbws.name
+d = get_everything_to_measure(dotsbws, cont, parser)
+args = d['args']
+conf = d['conf']
+rl = d['rl']
+dests = d['dests']
+cb = d['cb']
+# relay1mbyteRBR 934E06F38A391CB71DF83ECDE05DFF5CDE3AC49D
+relay = [r for r in rl.relays
+ if r.nickname == 'relay1mbyteRBR'][0]
+result = measure_relay(args, conf, dests, cb, rl, relay)
+assert len(result) == 1
+result = result[0]
+assert isinstance(result, ResultSuccess)
+one_mbyte = 1 * 1024 * 1024
+allowed_error = 5  # bytes per second
+dls = result.downloads
+for dl in dls:
+assert_within(dl['amount'] / dl['duration'], one_mbyte, allowed_error)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Validate new config options for reseting bw

2018-06-26 Thread pastly

commit 6a55ae5a4f3edb1af0710a253ad1d2b1883465f5
Author: juga0 
Date:   Tue Jun 19 19:38:38 2018 +

Validate new config options for reseting bw
---
 sbws/util/config.py | 5 +
 1 file changed, 5 insertions(+)

diff --git a/sbws/util/config.py b/sbws/util/config.py
index 1425dda..2ea9e94 100644
--- a/sbws/util/config.py
+++ b/sbws/util/config.py
@@ -137,10 +137,15 @@ def _validate_general(conf):
 floats = {
 'http_timeout': {'minimum': 0.0, 'maximum': None},
 }
+bools = {
+'reset_bw_ipv4_changes': {},
+'reset_bw_ipv6_changes': {},
+}
 all_valid_keys = list(ints.keys()) + list(floats.keys())
 errors.extend(_validate_section_keys(conf, sec, all_valid_keys, err_tmpl))
 errors.extend(_validate_section_ints(conf, sec, ints, err_tmpl))
 errors.extend(_validate_section_floats(conf, sec, floats, err_tmpl))
+errors.extend(_validate_section_bools(conf, sec, bools, err_tmpl))
 return errors
 
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] This sleep can probably be shorter

2018-06-26 Thread pastly

commit 3791316a251ce9f9c5fea02abdb9de3fd7aefe51
Author: Matt Traudt 
Date:   Tue Jun 19 22:17:48 2018 -0400

This sleep can probably be shorter
---
 tox.ini | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tox.ini b/tox.ini
index 2a34348..36d9283 100644
--- a/tox.ini
+++ b/tox.ini
@@ -29,7 +29,7 @@ commands =
 bash {envtmpdir}/net/start.sh
 bash -c "time python3 {envtmpdir}/net/wait.py 
{envtmpdir}/net/{auth,relay,exit}*"
 bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 &>/dev/null &"
-sleep 30
+sleep 15
 wget -O/dev/null http://127.0.0.1:2/sbws.bin
 coverage run -a --rcfile={toxinidir}/.coveragerc --source=sbws -m pytest 
-s {toxinidir}/tests/integration -vv
 bash {envtmpdir}/net/stop.sh



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add test for trim_results_ip_changed

2018-06-26 Thread pastly

commit f6a0876eebddc04d9158ca2f6f2fe80132c3def9
Author: juga0 
Date:   Mon Jun 18 13:29:58 2018 +

Add test for trim_results_ip_changed
---
 tests/unit/lib/test_resultdump.py | 31 +++
 1 file changed, 31 insertions(+)

diff --git a/tests/unit/lib/test_resultdump.py 
b/tests/unit/lib/test_resultdump.py
new file mode 100644
index 000..05a21ae
--- /dev/null
+++ b/tests/unit/lib/test_resultdump.py
@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+"""Unit tests for resultdump."""
+from sbws.lib.resultdump import trim_results_ip_changed, Result, ResultSuccess
+
+
+TIME1 = 1529232277.9028733
+TIME2 = 1529232278.9028733
+FP1 = 'A' * 40
+FP2 = 'Z' * 40
+ED25519 = 'g+Shk00y9Md0hg1S6ptnuc/wWKbADBgdjT0Kg+TSF3s'
+CIRC = [FP1, FP2]
+DEST_URL = 'http://example.com/sbws.bin'
+NICK = 'A'
+RELAY_ip1 = '169.254.100.1'
+RELAY_ip2 = '169.254.100.2'
+RELAY1 = Result.Relay(FP1, NICK, RELAY_ip1, ED25519)
+RELAY2 = Result.Relay(FP1, NICK, RELAY_ip2, ED25519)
+RTTS = [5, 25]
+DOWNLOADS = [{'duration': 4, 'amount': 40}]
+
+RESULTSUCCESS1 = ResultSuccess(RTTS, DOWNLOADS, RELAY1, CIRC, DEST_URL,
+   'sbws', t=TIME1)
+RESULTSUCCESS2 = ResultSuccess(RTTS, DOWNLOADS, RELAY2, CIRC, DEST_URL,
+   'sbws', t=TIME2)
+RESULTDICT = {FP1: [RESULTSUCCESS1, RESULTSUCCESS2]}
+
+
+def test_trim_results_ip_changed():
+expected_results_dict = {FP1: [RESULTSUCCESS2]}
+results_dict = trim_results_ip_changed(RESULTDICT)
+assert expected_results_dict == results_dict



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Remove unused member function

2018-06-26 Thread pastly

commit 7f085e23137b0420e2ec485639411fd7b0bbbea5
Author: Matt Traudt 
Date:   Thu Jun 14 18:35:11 2018 -0400

Remove unused member function
---
 sbws/lib/v3bwfile.py | 9 -
 1 file changed, 9 deletions(-)

diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index 3cff29d..e09392e 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -379,15 +379,6 @@ class V3BwFile(object):
 return self.total_bw / self.num_lines
 
 @classmethod
-def from_results(cls, conf, output, results):
-bw_lines = [V3BWLine.from_results(results[fp]) for fp in results]
-bw_lines = sorted(bw_lines, key=lambda d: d.bw, reverse=True)
-header = V3BwHeader.from_results(conf, results)
-f = cls(header, bw_lines)
-f.write(output)
-return f
-
-@classmethod
 def from_arg_results(cls, args, conf, results):
 bw_lines = [V3BWLine.from_results(results[fp]) for fp in results]
 bw_lines = sorted(bw_lines, key=lambda d: d.bw, reverse=True)



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add changelog entry for v3bw refactoring

2018-06-26 Thread pastly

commit 6440da403c36c7eba2e3d736ff3ef7874a2d76bc
Author: Matt Traudt 
Date:   Thu Jun 14 18:40:47 2018 -0400

Add changelog entry for v3bw refactoring
---
 CHANGELOG.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3e838ac..3a79c02 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,7 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 - Change the path where the Bandwidth List files are generated: now they are
   stored in `v3bw` directory, named `YYmmdd_HHMMSS.v3bw`, and previously
 generated ones are kept. A `latest.v3bw` symlink is updated. (GH#179 GHPR#190)
+- Code refactoring in the v3bw classes and generation area
 
 ## [0.4.1]
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Stop giving exception when threads can not start

2018-06-26 Thread pastly

commit b1dcfe4a448dfe515e82d1ccdc4ea719b1551cab
Author: juga0 
Date:   Mon Jun 18 11:12:45 2018 +

Stop giving exception when threads can not start

instead log the error and exit.
Fixes #195
---
 sbws/lib/resultdump.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 32fd1cd..977a9e5 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -11,7 +11,7 @@ from queue import Empty
 from datetime import datetime
 from datetime import timedelta
 from enum import Enum
-from sbws.globals import RESULT_VERSION
+from sbws.globals import RESULT_VERSION, fail_hard
 from sbws.util.filelock import DirectoryLock
 from sbws.lib.relaylist import Relay
 
@@ -444,7 +444,10 @@ class ResultDump:
 self.data_lock = RLock()
 self.thread = Thread(target=self.enter)
 self.queue = Queue()
-self.thread.start()
+try:
+self.thread.start()
+except RuntimeError as e:
+fail_hard(e)
 
 def store_result(self, result):
 ''' Call from ResultDump thread '''



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add simple HTTP server script

2018-06-26 Thread pastly

commit f55b84aca876a9c72ce4d364695457f8a0fd124d
Author: Matt Traudt 
Date:   Sun Jun 17 21:16:43 2018 -0400

Add simple HTTP server script
---
 scripts/tools/sbws-http-server.py | 77 +++
 1 file changed, 77 insertions(+)

diff --git a/scripts/tools/sbws-http-server.py 
b/scripts/tools/sbws-http-server.py
new file mode 100755
index 000..d8575d0
--- /dev/null
+++ b/scripts/tools/sbws-http-server.py
@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+# File: sbws-http-server.py
+# Author: Matt Traudt
+# License: CC0
+#
+# This script implements just enough of the HTTP protocol to work with Simple
+# Bandwidth Scanner.
+#
+# All requested URLs exist. All return 1 GiB of garbage data. We always speak
+# HTTP/1.1 because that's necessary for Keep-Alive request headers
+# (used by sbws scanners) to work.
+#
+# HEAD and GET requests are supported to the minimum extent necessary.
+# This essentially means that if the client sends Range request headers just
+# like sbws does, then we'll only send back the number of bytes they requested.
+# Indeed, this was the motivating reason for the complexity of this script;
+# normally I would have used SimpleHTTPRequestHandler unmodified.
+#
+# Don't breathe too hard or this script might break.
+from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
+import http.server
+from http import HTTPStatus
+# import time
+
+FILE_SIZE = 1*1024*1024*1024  # 1 GiB
+
+
+def _get_resp_size_from_range(range_str):
+assert range_str.startswith('bytes=')
+range_str = range_str[len('bytes='):]
+start_byte, end_byte = range_str.split('-')
+return int(end_byte) - int(start_byte) + 1
+
+
+class MyHTTPRequestHandler(http.server.SimpleHTTPRequestHandler):
+protocol_version = 'HTTP/1.1'
+
+def __init__(self, *a, **kw):
+super().__init__(*a, **kw)
+
+def send_head(self, length):
+self.send_response(HTTPStatus.OK)
+self.send_header('Content-Type', 'application/octet-stream')
+self.send_header('Content-Length', length)
+# self.send_header('Last-Modified', self.date_time_string(time.time()))
+self.end_headers()
+
+def do_GET(self):
+range_hdr = self.headers['Range']
+if not range_hdr:
+num_bytes = FILE_SIZE
+else:
+assert range_hdr.startswith('bytes=')
+num_bytes = _get_resp_size_from_range(range_hdr)
+self.send_head(num_bytes)
+self.wfile.write(b'A' * num_bytes)
+
+def do_HEAD(self):
+self.send_head(FILE_SIZE)
+
+
+def main(args):
+addr = ('', args.port)
+print('Listening on', addr)
+httpd = http.server.HTTPServer(addr, MyHTTPRequestHandler)
+httpd.serve_forever()
+
+
+if __name__ == '__main__':
+parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
+parser.add_argument(
+'-p', '--port', default=8000, type=int, help='Port on which to listen')
+args = parser.parse_args()
+try:
+exit(main(args))
+except KeyboardInterrupt:
+pass



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Sart an http server during integration tests

2018-06-26 Thread pastly

commit 313eeae2fb805bda6ee8ca46e36f0b0b9998d8d7
Author: Matt Traudt 
Date:   Sun Jun 17 21:22:35 2018 -0400

Sart an http server during integration tests
---
 tox.ini | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tox.ini b/tox.ini
index a3af1ef..bfe4e97 100644
--- a/tox.ini
+++ b/tox.ini
@@ -23,13 +23,16 @@ whitelist_externals =
 tar
 bash
 sleep
+cat
 commands =
 tar -C {envtmpdir} -vxf {toxinidir}/tests/integration/net.tar
 bash {envtmpdir}/net/start.sh
 bash -c "time python3 {envtmpdir}/net/wait.py 
{envtmpdir}/net/{auth,relay,exit}*"
+bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 > {envtmpdir}/http-server.log &"
 sleep 30
 coverage run -a --rcfile={toxinidir}/.coveragerc --source=sbws -m pytest 
-s {toxinidir}/tests/integration -vv
 bash {envtmpdir}/net/stop.sh
+cat {envtmpdir}/http-server.log
 
 [testenv:lint]
 skip_install = True



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Hopefully get http server logs if put in {toxinidir}

2018-06-26 Thread pastly

commit 60ef71bcd51427fed6fc4f2d80aeb8c128dd0c38
Author: Matt Traudt 
Date:   Sun Jun 17 21:32:57 2018 -0400

Hopefully get http server logs if put in {toxinidir}
---
 tox.ini | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tox.ini b/tox.ini
index bfe4e97..69371bd 100644
--- a/tox.ini
+++ b/tox.ini
@@ -28,11 +28,11 @@ commands =
 tar -C {envtmpdir} -vxf {toxinidir}/tests/integration/net.tar
 bash {envtmpdir}/net/start.sh
 bash -c "time python3 {envtmpdir}/net/wait.py 
{envtmpdir}/net/{auth,relay,exit}*"
-bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 > {envtmpdir}/http-server.log &"
+bash -c "python3 {toxinidir}/scripts/tools/sbws-http-server.py --port 
2 > {toxinidir}/http-server.log &"
 sleep 30
 coverage run -a --rcfile={toxinidir}/.coveragerc --source=sbws -m pytest 
-s {toxinidir}/tests/integration -vv
 bash {envtmpdir}/net/stop.sh
-cat {envtmpdir}/http-server.log
+cat {toxinidir}/http-server.log
 
 [testenv:lint]
 skip_install = True



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Add relay1mbyteMAB relay to integration test network

2018-06-26 Thread pastly

commit 57006a438190d8a76aed13a53c36eb86fedbe4cd
Author: Matt Traudt 
Date:   Sun Jun 17 19:51:16 2018 -0400

Add relay1mbyteMAB relay to integration test network

This relay has MaxAdvertisedBandwidth 1 MByte set in its torrc
---
 tests/integration/net.tar | Bin 191488 -> 204800 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/tests/integration/net.tar b/tests/integration/net.tar
index 2d8f1b9..cde1f17 100644
Binary files a/tests/integration/net.tar and b/tests/integration/net.tar differ



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Trim bandwidth results for relays that change IP

2018-06-26 Thread pastly

commit 4e7bc3e5b3f4f17d9a5848f21176f4c60b96c098
Author: juga0 
Date:   Mon Jun 18 13:22:43 2018 +

Trim bandwidth results for relays that change IP
---
 sbws/lib/resultdump.py | 39 ++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/sbws/lib/resultdump.py b/sbws/lib/resultdump.py
index 32fd1cd..d64d843 100644
--- a/sbws/lib/resultdump.py
+++ b/sbws/lib/resultdump.py
@@ -82,7 +82,44 @@ def trim_results(fresh_days, result_dict):
 return out_results
 
 
-def load_recent_results_in_datadir(fresh_days, datadir, success_only=False):
+def trim_results_ip_changed(result_dict, ipv4=True, ipv6=False):
+"""When there are results for the same relay with different IPs,
+create a new results' dictionary without that relay's results using an
+older IP.
+
+:param dict result_dict: a dictionary of results
+:param bool ipv4: whether to trim the results when a relay's IPv4 changes
+:param bool ipv6: whether to trim the results when a relay's IPv6 changes
+:returns: a new results dictionary
+"""
+assert isinstance(result_dict, dict)
+assert ipv4 is True or ipv6 is True
+new_results_dict = {}
+if ipv4 is True:
+for fp in result_dict.keys():
+results = result_dict[fp]
+# find if the results for a relay have more than one ipv4
+# address
+ipv4s = set([result.address for result in results])
+if len(ipv4s) > 1:
+# keep only the results for the last ip used
+# probably we should not just discard all the results for
+# a relay that change address
+ordered_results = sorted(results, key=lambda r: r.time)
+latest_address = ordered_results[-1].address
+last_ip_results = [result for result in results
+   if result.address == latest_address]
+new_results_dict[fp] = last_ip_results
+else:
+new_results_dict[fp] = results
+if ipv6 is True:
+# Not implemented
+pass
+return new_results_dict
+
+
+def load_recent_results_in_datadir(fresh_days, datadir, success_only=False,
+   ipv4=True, ipv6=False):
 ''' Given a data directory, read all results files in it that could have
 results in them that are still valid. Trim them, and return the valid
 Results as a list '''



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Standardize capitalization of v3bw class names

2018-06-26 Thread pastly

commit ee80724056521a4aa3b0741ab53f68b0448189bb
Author: Matt Traudt 
Date:   Thu Jun 14 18:37:25 2018 -0400

Standardize capitalization of v3bw class names
---
 sbws/core/generate.py   |  4 ++--
 sbws/lib/v3bwfile.py| 10 +-
 tests/unit/lib/test_v3bwfile.py | 24 
 3 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/sbws/core/generate.py b/sbws/core/generate.py
index 3f0df18..25653b1 100644
--- a/sbws/core/generate.py
+++ b/sbws/core/generate.py
@@ -1,5 +1,5 @@
 from sbws.globals import (fail_hard, is_initted, SCALE_CONSTANT)
-from sbws.lib.v3bwfile import V3BwFile
+from sbws.lib.v3bwfile import V3BWFile
 from sbws.lib.resultdump import load_recent_results_in_datadir
 from argparse import ArgumentDefaultsHelpFormatter
 import os
@@ -77,7 +77,7 @@ def main(args, conf):
 log.warning('No recent results, so not generating anything. (Have you '
 'ran sbws scanner recently?)')
 return
-bw_file = V3BwFile.from_arg_results(args, conf, results)
+bw_file = V3BWFile.from_arg_results(args, conf, results)
 output = args.output or conf['paths']['v3bw_fname'].format(now_fname())
 _write_v3bw_file(bw_file, output)
 log.info('Mean bandwidth per line: %f "KiB"', bw_file.avg_bw)
diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index e09392e..5b07f80 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -95,7 +95,7 @@ def result_type_to_key(type_str):
 return type_str.replace('-', '_')
 
 
-class V3BwHeader(object):
+class V3BWHeader(object):
 """
 Create a bandwidth measurements (V3bw) header
 following bandwidth measurements document spec version 1.1.0.
@@ -178,7 +178,7 @@ class V3BwHeader(object):
 def from_lines_v110(cls, lines):
 """
 :param list lines: list of lines to parse
-:returns: tuple of V3BwHeader object and non-header lines
+:returns: tuple of V3BWHeader object and non-header lines
 """
 assert isinstance(lines, list)
 try:
@@ -198,7 +198,7 @@ class V3BwHeader(object):
 def from_text_v110(self, text):
 """
 :param str text: text to parse
-:returns: tuple of V3BwHeader object and non-header lines
+:returns: tuple of V3BWHeader object and non-header lines
 """
 assert isinstance(text, str)
 return self.from_lines_v110(text.split(LINE_SEP))
@@ -351,7 +351,7 @@ class V3BWLine(object):
 return cls.from_results(data[fingerprint])
 
 
-class V3BwFile(object):
+class V3BWFile(object):
 """
 Create a Bandwidth List file following spec version 1.1.0
 
@@ -384,6 +384,6 @@ class V3BwFile(object):
 bw_lines = sorted(bw_lines, key=lambda d: d.bw, reverse=True)
 if args.scale:
 bw_lines = scale_lines(bw_lines, args.scale_constant)
-header = V3BwHeader.from_results(conf, results)
+header = V3BWHeader.from_results(conf, results)
 f = cls(header, bw_lines)
 return f
diff --git a/tests/unit/lib/test_v3bwfile.py b/tests/unit/lib/test_v3bwfile.py
index e18278d..2f4a631 100644
--- a/tests/unit/lib/test_v3bwfile.py
+++ b/tests/unit/lib/test_v3bwfile.py
@@ -5,9 +5,9 @@ import json
 from sbws import __version__ as version
 from sbws.globals import SPEC_VERSION, RESULT_VERSION
 from sbws.lib.resultdump import Result, load_result_file
-from sbws.lib.v3bwfile import (V3BwHeader, V3BWLine, TERMINATOR, LINE_SEP,
+from sbws.lib.v3bwfile import (V3BWHeader, V3BWLine, TERMINATOR, LINE_SEP,
KEYVALUE_SEP_V110, num_results_of_type,
-   V3BwFile)
+   V3BWFile)
 
 timestamp = 1523974147
 timestamp_l = str(timestamp)
@@ -86,13 +86,13 @@ RESULT_ERROR_STREAM_STR = str(RESULT_ERROR_STREAM_DICT)
 
 def test_v3bwheader_str():
 """Test header str"""
-header = V3BwHeader(timestamp_l, file_created=file_created)
+header = V3BWHeader(timestamp_l, file_created=file_created)
 assert header_str == str(header)
 
 
 def test_v3bwheader_extra_str():
 """Test header str with additional headers"""
-header = V3BwHeader(timestamp_l,
+header = V3BWHeader(timestamp_l,
 file_created=file_created,
 generator_started=generator_started,
 earliest_bandwidth=earliest_bandwidth)
@@ -101,33 +101,33 @@ def test_v3bwheader_extra_str():
 
 def test_v3bwheader_from_lines():
 ""
-header_obj = V3BwHeader(timestamp_l,
+header_obj = V3BWHeader(timestamp_l,
 file_created=file_created,
 generator_started=generator_started,
 earliest_bandwidth=earliest_bandwidth)
-header, _ = V3BwHeader.from_lines_v110(header_extra_ls)
+header, _ = V3BWHeader.from_lines_v110(header_extra_ls)
 assert str(header_obj) == str(header)
 
 
 def

[tor-commits] [sbws/master] Update changelog: v3bw files are stored in v3bw dir

2018-06-26 Thread pastly

commit a46bac3d3268ce5e2ebbb16794ef197065fc2d17
Author: juga0 
Date:   Thu Jun 14 15:07:21 2018 +

Update changelog: v3bw files are stored in v3bw dir
---
 CHANGELOG.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 79d781f..d3255c8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,6 +21,8 @@ is at least 2.00, 1.75, 1.50, 1.25, or 1.00 times as fast. If 
that fails,
 instead of giving up, (2) pick the fastest relay in the network instead of
 giving up. This compliments the previous change about measuring target exits in
 the exit position.
+- Change the path where the Bandwidth List files are generated: now they are
+  stored in `v3bw` directory and named `YYmmdd_HHMMSS.v3bw`
 
 ### Fixed
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Generate v3bw files in a dir and named with timestamp

2018-06-26 Thread pastly

commit b94f7ae022d7de761235fa80f69a21c9ba99aa7b
Author: juga0 
Date:   Thu Jun 14 14:27:59 2018 +

Generate v3bw files in a dir and named with timestamp
---
 sbws/config.default.ini |  3 ++-
 sbws/core/generate.py   |  2 ++
 sbws/lib/v3bwfile.py|  4 ++--
 sbws/util/config.py |  2 +-
 sbws/util/timestamp.py  | 10 ++
 5 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/sbws/config.default.ini b/sbws/config.default.ini
index 15ab014..faa0ab1 100644
--- a/sbws/config.default.ini
+++ b/sbws/config.default.ini
@@ -1,6 +1,7 @@
 [paths]
 datadir = ${sbws_home}/datadir
-v3bw_fname = ${sbws_home}/v3bw.txt
+v3bw_dname = ${sbws_home}/v3bw
+v3bw_fname = ${v3bw_dname}/{}.v3bw
 started_filepath = ${sbws_home}/started_at
 log_filepath = ${sbws_home}/sbws.log
 
diff --git a/sbws/core/generate.py b/sbws/core/generate.py
index 4921f57..61f807d 100644
--- a/sbws/core/generate.py
+++ b/sbws/core/generate.py
@@ -35,6 +35,8 @@ def main(args, conf):
 if not is_initted(args.directory):
 fail_hard('Sbws isn\'t initialized.  Try sbws init')
 
+os.makedirs(conf['paths']['v3bw_dname'], exist_ok=True)
+
 datadir = conf['paths']['datadir']
 if not os.path.isdir(datadir):
 fail_hard('%s does not exist', datadir)
diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index c6ba9af..ba15d14 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -9,7 +9,7 @@ from sbws import __version__
 from sbws.globals import SPEC_VERSION, BW_LINE_SIZE
 from sbws.lib.resultdump import ResultSuccess, _ResultType
 from sbws.util.filelock import FileLock
-from sbws.util.timestamp import now_isodt_str, unixts_to_isodt_str
+from sbws.util.timestamp import now_isodt_str, unixts_to_isodt_str, now_fname
 
 log = logging.getLogger(__name__)
 
@@ -395,7 +395,7 @@ class V3BwFile(object):
 bw_lines = scale_lines(bw_lines, args.scale_constant)
 header = V3BwHeader.from_results(conf, results)
 f = cls(header, bw_lines)
-output = args.output or conf['paths']['v3bw_fname']
+output = args.output or conf['paths']['v3bw_fname'].format(now_fname())
 f.write(output)
 return f
 
diff --git a/sbws/util/config.py b/sbws/util/config.py
index cdf8601..1425dda 100644
--- a/sbws/util/config.py
+++ b/sbws/util/config.py
@@ -149,7 +149,7 @@ def _validate_paths(conf):
 sec = 'paths'
 err_tmpl = Template('$sec/$key ($val): $e')
 unvalidated_keys = [
-'datadir', 'sbws_home', 'v3bw_fname',
+'datadir', 'sbws_home', 'v3bw_fname', 'v3bw_dname',
 'started_filepath', 'log_filepath']
 all_valid_keys = unvalidated_keys
 allow_missing = ['sbws_home']
diff --git a/sbws/util/timestamp.py b/sbws/util/timestamp.py
index 0de7551..d4629a3 100644
--- a/sbws/util/timestamp.py
+++ b/sbws/util/timestamp.py
@@ -47,6 +47,16 @@ def now_isodt_str():
 return dt_obj_to_isodt_str(datetime.utcnow())
 
 
+def now_fname():
+"""
+Return now timestamp in UTC formatted as %Y%m%d_%H%M%S string for file
+names.
+
+:returns: now timestamp in UTC formatted as %Y%m%d_%H%M%S string
+"""
+return datetime.utcnow().strftime("%Y%m%d_%H%M%S")
+
+
 def unixts_to_str(unixts):
 """Convert unix timestamp integer or float to string"""
 # even if it is only converting to str, ensure that input is nothing else



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Refactor V3BwFile ...

2018-06-26 Thread pastly

commit 873530454b07d99674f805b8652888b866cb9683
Author: Matt Traudt 
Date:   Thu Jun 14 18:34:37 2018 -0400

Refactor V3BwFile ...

It doesn't need to know how to write to a file. It store and organizes
data.
---
 sbws/core/generate.py | 29 +
 sbws/lib/v3bwfile.py  | 31 ++-
 2 files changed, 31 insertions(+), 29 deletions(-)

diff --git a/sbws/core/generate.py b/sbws/core/generate.py
index 61f807d..3f0df18 100644
--- a/sbws/core/generate.py
+++ b/sbws/core/generate.py
@@ -4,6 +4,8 @@ from sbws.lib.resultdump import load_recent_results_in_datadir
 from argparse import ArgumentDefaultsHelpFormatter
 import os
 import logging
+from sbws.util.filelock import DirectoryLock
+from sbws.util.timestamp import now_fname
 
 log = logging.getLogger(__name__)
 
@@ -31,6 +33,31 @@ def gen_parser(sub):
'out, and we do so proportionally')
 
 
+def _write_v3bw_file(v3bwfile, output):
+log.info('Writing v3bw file to %s', output)
+out_dir = os.path.dirname(output)
+out_link = os.path.join(out_dir, 'latest.v3bw')
+if os.path.exists(out_link):
+log.debug('Deleting existing symlink before creating a new one.')
+os.remove(out_link)
+# to keep test_generate.py working
+if output != '/dev/stdout':
+with DirectoryLock(out_dir):
+with open(output, 'wt') as fd:
+fd.write(str(v3bwfile.header))
+for line in v3bwfile.bw_lines:
+fd.write(str(line))
+output_basename = os.path.basename(output)
+log.debug('Creating symlink from {} to {}.'
+  .format(output_basename, out_link))
+os.symlink(output_basename, out_link)
+else:
+with open(output, 'wt') as fd:
+fd.write(str(v3bwfile.header))
+for line in v3bwfile.bw_lines:
+fd.write(str(line))
+
+
 def main(args, conf):
 if not is_initted(args.directory):
 fail_hard('Sbws isn\'t initialized.  Try sbws init')
@@ -51,4 +78,6 @@ def main(args, conf):
 'ran sbws scanner recently?)')
 return
 bw_file = V3BwFile.from_arg_results(args, conf, results)
+output = args.output or conf['paths']['v3bw_fname'].format(now_fname())
+_write_v3bw_file(bw_file, output)
 log.info('Mean bandwidth per line: %f "KiB"', bw_file.avg_bw)
diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index 9b8e2e8..3cff29d 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -3,14 +3,13 @@
 (v3bw) used by bandwidth authorities."""
 
 import logging
-import os
 from statistics import median
 
 from sbws import __version__
 from sbws.globals import SPEC_VERSION, BW_LINE_SIZE
 from sbws.lib.resultdump import ResultSuccess, _ResultType
-from sbws.util.filelock import FileLock, DirectoryLock
-from sbws.util.timestamp import now_isodt_str, unixts_to_isodt_str, now_fname
+from sbws.util.filelock import FileLock
+from sbws.util.timestamp import now_isodt_str, unixts_to_isodt_str
 
 log = logging.getLogger(__name__)
 
@@ -396,30 +395,4 @@ class V3BwFile(object):
 bw_lines = scale_lines(bw_lines, args.scale_constant)
 header = V3BwHeader.from_results(conf, results)
 f = cls(header, bw_lines)
-output = args.output or conf['paths']['v3bw_fname'].format(now_fname())
-f.write(output)
 return f
-
-def write(self, output):
-log.info('Writing v3bw file to %s', output)
-out_dir = os.path.dirname(output)
-out_link = os.path.join(out_dir, 'latest.v3bw')
-if os.path.exists(out_link):
-log.debug('Deleting existing symlink before creating a new one.')
-os.remove(out_link)
-# to keep test_generate.py working
-if output != '/dev/stdout':
-with DirectoryLock(out_dir):
-with open(output, 'wt') as fd:
-fd.write(str(self.header))
-for line in self.bw_lines:
-fd.write(str(line))
-output_basename = os.path.basename(output)
-log.debug('Creating symlink from {} to {}.'
-  .format(output_basename, out_link))
-os.symlink(output_basename, out_link)
-else:
-with open(output, 'wt') as fd:
-fd.write(str(self.header))
-for line in self.bw_lines:
-fd.write(str(line))



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Fix two linter warnings

2018-06-26 Thread pastly

commit ef6e152b3c94f3e258575521e6d0887d7b68debc
Author: Matt Traudt 
Date:   Thu Jun 14 12:44:49 2018 -0400

Fix two linter warnings
---
 sbws/lib/v3bwfile.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index 40a8fd5..ddfefe7 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -75,7 +75,7 @@ def read_started_ts(conf):
 """
 try:
 filepath = conf['paths']['started_filepath']
-except TypeError as e:
+except TypeError:
 return ''
 try:
 with FileLock(filepath):
@@ -184,7 +184,7 @@ class V3BwHeader(object):
 assert isinstance(lines, list)
 try:
 index_terminator = lines.index(TERMINATOR)
-except ValueError as e:
+except ValueError:
 # is not a bw file or is v100
 log.warn('Terminator is not in lines')
 return None



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Fix header level

2018-06-26 Thread pastly

commit 0402bdc483313927108539663d956f2d97b9400d
Author: Matt Traudt 
Date:   Thu Jun 14 14:00:21 2018 -0400

Fix header level
---
 CHANGELOG.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8981cb2..3e838ac 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,7 +7,7 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
-## Changed
+### Changed
 
 - Change the path where the Bandwidth List files are generated: now they are
   stored in `v3bw` directory, named `YYmmdd_HHMMSS.v3bw`, and previously



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Make symlink relative ...

2018-06-26 Thread pastly

commit e59349c8c95b5fe749aaeda32333515aa1122e6d
Author: Matt Traudt 
Date:   Thu Jun 14 18:25:21 2018 -0400

Make symlink relative ...

I think this is the more common way to do things. It allows you to move
the files around without the symlink breaking.
---
 sbws/lib/v3bwfile.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index ddfefe7..9b8e2e8 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -414,9 +414,10 @@ class V3BwFile(object):
 fd.write(str(self.header))
 for line in self.bw_lines:
 fd.write(str(line))
+output_basename = os.path.basename(output)
 log.debug('Creating symlink from {} to {}.'
-  .format(output, out_link))
-os.symlink(output, out_link)
+  .format(output_basename, out_link))
+os.symlink(output_basename, out_link)
 else:
 with open(output, 'wt') as fd:
 fd.write(str(self.header))



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Create symlink to latest v3bw file

2018-06-26 Thread pastly

commit 0db6329319ec9ea2a9486838257d5c1c3e210cc1
Author: juga0 
Date:   Thu Jun 14 15:54:24 2018 +

Create symlink to latest v3bw file

also lock the directory except when writing to the standard output,
needed for test_generate.py
---
 sbws/lib/v3bwfile.py | 20 +++-
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/sbws/lib/v3bwfile.py b/sbws/lib/v3bwfile.py
index ba15d14..1294d99 100644
--- a/sbws/lib/v3bwfile.py
+++ b/sbws/lib/v3bwfile.py
@@ -3,12 +3,13 @@
 (v3bw) used by bandwidth authorities."""
 
 import logging
+import os
 from statistics import median
 
 from sbws import __version__
 from sbws.globals import SPEC_VERSION, BW_LINE_SIZE
 from sbws.lib.resultdump import ResultSuccess, _ResultType
-from sbws.util.filelock import FileLock
+from sbws.util.filelock import FileLock, DirectoryLock
 from sbws.util.timestamp import now_isodt_str, unixts_to_isodt_str, now_fname
 
 log = logging.getLogger(__name__)
@@ -401,7 +402,16 @@ class V3BwFile(object):
 
 def write(self, output):
 log.info('Writing v3bw file to %s', output)
-with open(output, 'wt') as fd:
-fd.write(str(self.header))
-for line in self.bw_lines:
-fd.write(str(line))
+# to keep test_generate.py working
+if output != '/dev/stdout':
+with DirectoryLock(os.path.dirname(output)):
+with open(output, 'wt') as fd:
+fd.write(str(self.header))
+for line in self.bw_lines:
+fd.write(str(line))
+os.symlink(output, 'latest.v3bw')
+else:
+with open(output, 'wt') as fd:
+fd.write(str(self.header))
+for line in self.bw_lines:
+fd.write(str(line))



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Change the path of the v3bw files to a dir too

2018-06-26 Thread pastly

commit 3de8eeda75c25530ae05d4e28b6b2f633bbde15a
Author: juga0 
Date:   Thu Jun 14 15:00:45 2018 +

Change the path of the v3bw files to a dir too

It does not change functionality right now, but probably needed
for future tests
---
 tests/unit/lib/data/{v3bw.txt => v3bw/20180425_131057.v3bw} | 0
 tests/unit/lib/test_v3bwfile.py | 6 --
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/tests/unit/lib/data/v3bw.txt 
b/tests/unit/lib/data/v3bw/20180425_131057.v3bw
similarity index 100%
rename from tests/unit/lib/data/v3bw.txt
rename to tests/unit/lib/data/v3bw/20180425_131057.v3bw
diff --git a/tests/unit/lib/test_v3bwfile.py b/tests/unit/lib/test_v3bwfile.py
index ca47a2b..e18278d 100644
--- a/tests/unit/lib/test_v3bwfile.py
+++ b/tests/unit/lib/test_v3bwfile.py
@@ -125,7 +125,8 @@ def test_v3bwheader_from_file(datadir):
 file_created=file_created,
 generator_started=generator_started,
 earliest_bandwidth=earliest_bandwidth)
-text = datadir.read('v3bw.txt')
+# at some point this should be read from conftest
+text = datadir.read('v3bw/20180425_131057.v3bw')
 h, _ = V3BwHeader.from_text_v110(text)
 assert str(h) == str(header)
 
@@ -156,7 +157,8 @@ def test_v3bwline_from_results_file(datadir):
 
 def test_v3bwfile(datadir, tmpdir):
 """Test generate v3bw file (including relay_lines)."""
-v3bw = datadir.read('v3bw.txt')
+# at some point this should be obtained from conftest
+v3bw = datadir.read('v3bw/20180425_131057.v3bw')
 results = load_result_file(str(datadir.join("results.txt")))
 header = V3BwHeader(timestamp_l,
 file_created=file_created,



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [sbws/master] Changelog entry belongs in Unreleased; add more details

2018-06-26 Thread pastly

commit c47238dd07d890ad09ada9f078e5425d44ba539e
Author: Matt Traudt 
Date:   Thu Jun 14 12:44:02 2018 -0400

Changelog entry belongs in Unreleased; add more details
---
 CHANGELOG.md | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index d3255c8..8981cb2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,12 @@ and this project adheres to [Semantic 
Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## Changed
+
+- Change the path where the Bandwidth List files are generated: now they are
+  stored in `v3bw` directory, named `YYmmdd_HHMMSS.v3bw`, and previously
+generated ones are kept. A `latest.v3bw` symlink is updated. (GH#179 GHPR#190)
+
 ## [0.4.1]
 
 ### Changed
@@ -21,8 +27,6 @@ is at least 2.00, 1.75, 1.50, 1.25, or 1.00 times as fast. If 
that fails,
 instead of giving up, (2) pick the fastest relay in the network instead of
 giving up. This compliments the previous change about measuring target exits in
 the exit position.
-- Change the path where the Bandwidth List files are generated: now they are
-  stored in `v3bw` directory and named `YYmmdd_HHMMSS.v3bw`
 
 ### Fixed
 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Fix comments in lib/container/*.c

2018-06-26 Thread nickm

commit 58fc42fdcedc9c0446e709d98eaf7c53c464750a
Author: Nick Mathewson 
Date:   Tue Jun 26 11:33:53 2018 -0400

Fix comments in lib/container/*.c
---
 src/lib/container/bloomfilt.c | 7 ++-
 src/lib/container/map.c   | 4 +---
 src/lib/container/order.c | 7 ++-
 src/lib/container/smartlist.c | 4 +---
 4 files changed, 6 insertions(+), 16 deletions(-)

diff --git a/src/lib/container/bloomfilt.c b/src/lib/container/bloomfilt.c
index cbb4d13e5..2133c280a 100644
--- a/src/lib/container/bloomfilt.c
+++ b/src/lib/container/bloomfilt.c
@@ -4,11 +4,8 @@
 /* See LICENSE for licensing information */
 
 /**
- * \file container.c
- * \brief Implements a smartlist (a resizable array) along
- * with helper functions to use smartlists.  Also includes
- * hash table implementations of a string-to-void* map, and of
- * a digest-to-void* map.
+ * \file bloomfilt.c
+ * \brief Uses bitarray_t to implement a bloom filter.
  **/
 
 #include 
diff --git a/src/lib/container/map.c b/src/lib/container/map.c
index 5f280b316..3d84356cc 100644
--- a/src/lib/container/map.c
+++ b/src/lib/container/map.c
@@ -5,9 +5,7 @@
 
 /**
  * \file container.c
- * \brief Implements a smartlist (a resizable array) along
- * with helper functions to use smartlists.  Also includes
- * hash table implementations of a string-to-void* map, and of
+ * \brief Hash table implementations of a string-to-void* map, and of
  * a digest-to-void* map.
  **/
 
diff --git a/src/lib/container/order.c b/src/lib/container/order.c
index 4fdd51d99..1efef2c73 100644
--- a/src/lib/container/order.c
+++ b/src/lib/container/order.c
@@ -4,11 +4,8 @@
 /* See LICENSE for licensing information */
 
 /**
- * \file container.c
- * \brief Implements a smartlist (a resizable array) along
- * with helper functions to use smartlists.  Also includes
- * hash table implementations of a string-to-void* map, and of
- * a digest-to-void* map.
+ * \file order.c
+ * \brief Functions for finding the n'th element of an array.
  **/
 
 #include 
diff --git a/src/lib/container/smartlist.c b/src/lib/container/smartlist.c
index 852a32810..3c0844384 100644
--- a/src/lib/container/smartlist.c
+++ b/src/lib/container/smartlist.c
@@ -6,9 +6,7 @@
 /**
  * \file container.c
  * \brief Implements a smartlist (a resizable array) along
- * with helper functions to use smartlists.  Also includes
- * hash table implementations of a string-to-void* map, and of
- * a digest-to-void* map.
+ * with helper functions to use smartlists.
  **/
 
 #include "lib/malloc/util_malloc.h"

___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Make an inline static so we can build with coverage enabled.

2018-06-26 Thread nickm

commit 1b93b065fc3eb52fe5674d6df05c71c505ed1ef3
Author: Nick Mathewson 
Date:   Fri Jun 22 14:11:37 2018 -0400

Make an inline static so we can build with coverage enabled.
---
 src/lib/string/compat_ctype.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/string/compat_ctype.h b/src/lib/string/compat_ctype.h
index 3fde6eaf8..530a10270 100644
--- a/src/lib/string/compat_ctype.h
+++ b/src/lib/string/compat_ctype.h
@@ -31,10 +31,10 @@ extern const uint8_t TOR_TOLOWER_TABLE[];
 #define TOR_TOLOWER(c) (TOR_TOLOWER_TABLE[(uint8_t)c])
 #define TOR_TOUPPER(c) (TOR_TOUPPER_TABLE[(uint8_t)c])
 
-inline int hex_decode_digit(char c);
+static inline int hex_decode_digit(char c);
 
 /** Helper: given a hex digit, return its value, or -1 if it isn't hex. */
-inline int
+static inline int
 hex_decode_digit(char c)
 {
   switch (c) {



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Another windows include

2018-06-26 Thread nickm

commit 405fa42e8a835de394503231a85258972a65bd89
Author: Nick Mathewson 
Date:   Fri Jun 22 13:52:30 2018 -0400

Another windows include
---
 scripts/maint/checkIncludes.py | 15 ---
 src/lib/wallclock/tm_cvt.c |  1 +
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/scripts/maint/checkIncludes.py b/scripts/maint/checkIncludes.py
index 3c611675e..cb3354abf 100755
--- a/scripts/maint/checkIncludes.py
+++ b/scripts/maint/checkIncludes.py
@@ -58,12 +58,21 @@ def load_include_rules(fname):
 result.addPattern(line)
 return result
 
+MDD = "--mdd" in sys.argv[1:]
+
+if MDD:
+print(
+
 for dirpath, dirnames, fnames in os.walk("src"):
 if ".may_include" in fnames:
 rules = load_include_rules(os.path.join(dirpath, RULES_FNAME))
-for fname in fnames:
-if fname_is_c(fname):
-rules.applyToFile(os.path.join(dirpath,fname))
+if MDD:
+dp = dirpath.replace("src/","")
+rules.dump_mdd(dp)
+else:
+for fname in fnames:
+if fname_is_c(fname):
+rules.applyToFile(os.path.join(dirpath,fname))
 
 if trouble:
 err(
diff --git a/src/lib/wallclock/tm_cvt.c b/src/lib/wallclock/tm_cvt.c
index 63f92bff2..987b0ffeb 100644
--- a/src/lib/wallclock/tm_cvt.c
+++ b/src/lib/wallclock/tm_cvt.c
@@ -13,6 +13,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #if !defined(_WIN32)
 /** Defined iff we need to add locks when defining fake versions of reentrant



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Merge branch 'log_dependencies'

2018-06-26 Thread nickm

commit 92d8284a9792267514cbe21f3ce1564f6ad0e10b
Merge: b4e23dba9 1b93b065f
Author: Nick Mathewson 
Date:   Tue Jun 26 11:27:33 2018 -0400

Merge branch 'log_dependencies'

 .gitignore |   16 +
 Makefile.am|   16 +
 scripts/maint/checkIncludes.py |   15 +-
 src/common/.may_include|3 -
 src/common/address.c   |6 +-
 src/common/address.h   |   26 +-
 src/common/address_set.c   |3 +-
 src/common/buffers.c   |2 +-
 src/common/compat.c|  488 +
 src/common/compat.h|   73 +-
 src/common/compat_libevent.c   |2 +-
 src/common/compat_pthreads.c   |   82 +-
 src/common/compat_threads.c|   30 +-
 src/common/compat_threads.h|   42 +-
 src/common/compat_time.c   |   56 +-
 src/common/compat_time.h   |5 +-
 src/common/compat_winthreads.c |   30 +-
 src/common/confline.c  |5 +-
 src/common/confline.h  |5 +-
 src/common/container.h |  742 -
 src/common/include.am  |6 -
 src/common/memarea.c   |3 +-
 src/common/procmon.h   |2 +-
 src/common/sandbox.c   |6 +-
 src/common/storagedir.c|5 +-
 src/common/timers.c|2 +-
 src/common/token_bucket.c  |4 +-
 src/common/util.c  | 1122 +---
 src/common/util.h  |  207 +---
 src/common/util_format.c   |   41 +-
 src/common/util_format.h   |2 -
 src/common/util_process.c  |2 +-
 src/common/workqueue.c |2 +-
 src/include.am |9 +
 src/lib/cc/compat_compiler.h   |   14 +
 src/lib/compress/.may_include  |1 +
 src/lib/compress/compress.c|2 +-
 src/lib/compress/compress_lzma.c   |2 +-
 src/lib/compress/compress_none.c   |2 +-
 src/lib/compress/compress_zlib.c   |2 +-
 src/lib/compress/compress_zstd.c   |2 +-
 src/lib/container/.may_include |   16 +
 src/lib/container/bitarray.h   |   80 ++
 src/lib/container/bloomfilt.c  |   49 +
 src/lib/container/bloomfilt.h  |   58 +
 src/lib/container/include.am   |   24 +
 src/lib/container/map.c|  414 
 src/lib/container/map.h|  255 +
 src/lib/container/order.c  |   51 +
 src/lib/container/order.h  |   54 +
 .../container.c => lib/container/smartlist.c}  |  508 +
 src/lib/container/smartlist.h  |  360 +++
 src/lib/crypt_ops/.may_include |8 +
 src/lib/crypt_ops/aes.c|2 +-
 src/lib/crypt_ops/crypto.c |5 +-
 src/lib/crypt_ops/crypto_curve25519.c  |3 +-
 src/lib/crypt_ops/crypto_dh.c  |2 +-
 src/lib/crypt_ops/crypto_digest.c  |5 +-
 src/lib/crypt_ops/crypto_digest.h  |   16 +-
 src/lib/crypt_ops/crypto_ed25519.c |2 +-
 src/lib/crypt_ops/crypto_format.c  |6 +-
 src/lib/crypt_ops/crypto_openssl_mgt.c |2 +-
 src/lib/crypt_ops/crypto_pwbox.c   |2 +-
 src/lib/crypt_ops/crypto_rand.c|5 +-
 src/lib/crypt_ops/crypto_rsa.c |2 +-
 src/lib/crypt_ops/crypto_rsa.h |2 +-
 src/lib/crypt_ops/crypto_util.c|2 +-
 src/lib/ctime/.may_include |5 +-
 src/lib/ctime/di_ops.c |   17 +-
 src/lib/defs/.may_include  |1 +
 src/lib/defs/digest_sizes.h|   18 +
 src/lib/defs/include.am|3 +
 src/lib/fdio/.may_include  |4 +
 src/lib/fdio/fdio.c|  109 ++
 src/lib/fdio/fdio.h|   17 +
 src/lib/fdio/include.am|   17 +
 src/lib/intmath/.may_include   |4 +

[tor-commits] [tor/master] We also need torerr in tm_cvt.

2018-06-26 Thread nickm

commit 4b32446a4c52f9ac91cf230b90ee5b72331793be
Author: Nick Mathewson 
Date:   Fri Jun 22 13:34:35 2018 -0400

We also need torerr in tm_cvt.
---
 src/lib/wallclock/tm_cvt.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/lib/wallclock/tm_cvt.c b/src/lib/wallclock/tm_cvt.c
index cf73cc542..63f92bff2 100644
--- a/src/lib/wallclock/tm_cvt.c
+++ b/src/lib/wallclock/tm_cvt.c
@@ -8,6 +8,7 @@
 #include "lib/cc/compat_compiler.h"
 #include "lib/wallclock/tm_cvt.h"
 #include "lib/string/printf.h"
+#include "lib/err/torerr.h"
 
 #include 
 #include 



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Fix up the rust build script library list.

2018-06-26 Thread nickm

commit b0adf2fc9b079e530494328c8c1aa4528b649080
Author: Nick Mathewson 
Date:   Fri Jun 22 12:04:11 2018 -0400

Fix up the rust build script library list.
---
 src/rust/build.rs | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/rust/build.rs b/src/rust/build.rs
index 8bb9eb295..b43fbdceb 100644
--- a/src/rust/build.rs
+++ b/src/rust/build.rs
@@ -151,10 +151,13 @@ pub fn main() {
 // moving forward!
 cfg.component("tor-crypt-ops-testing");
 cfg.component("or-testing");
-cfg.component("libtor-lock");
+cfg.component("tor-log");
+cfg.component("tor-lock");
+cfg.component("tor-fdio");
 cfg.component("tor-container-testing");
 cfg.component("tor-string-testing");
 cfg.component("tor-malloc");
+cfg.component("tor-wallclock");
 cfg.component("tor-err-testing");
 cfg.component("or-event-testing");
 cfg.component("tor-intmath-testing");



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Finally extract the log library and make it build.

2018-06-26 Thread nickm

commit 79f73ab330c0e643ba1ec2eb0633f0f80525a083
Author: Nick Mathewson 
Date:   Fri Jun 22 11:40:20 2018 -0400

Finally extract the log library and make it build.

This patch:
  - introduces an fdio module for low-level fd functions that don't
need to log.
  - moves the responsibility for opening files outside of torlog.c,
so it won't need to call tor_open_cloexec.
---
 Makefile.am   |   2 +
 src/common/compat.c   |  75 ---
 src/common/compat.h   |   5 ---
 src/common/util.c |   1 +
 src/include.am|   1 +
 src/lib/fdio/.may_include |   4 ++
 src/lib/fdio/fdio.c   | 109 ++
 src/lib/fdio/fdio.h   |  17 
 src/lib/fdio/include.am   |  17 
 src/lib/log/torlog.c  |  54 ---
 src/lib/log/torlog.h  |   6 ++-
 src/or/config.c   |  20 -
 src/or/config.h   |   4 +-
 src/or/keypin.c   |   2 +-
 src/or/microdesc.c|   4 +-
 src/test/test_logging.c   |   5 ++-
 src/test/test_util.c  |   1 +
 17 files changed, 204 insertions(+), 123 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index b6200c6a2..61451ed26 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -42,6 +42,7 @@ TOR_UTIL_LIBS = \
src/common/libor.a \
 src/lib/libtor-log.a \
 src/lib/libtor-lock.a \
+src/lib/libtor-fdio.a \
src/lib/libtor-container.a \
 src/lib/libtor-string.a \
src/lib/libtor-malloc.a \
@@ -56,6 +57,7 @@ TOR_UTIL_TESTING_LIBS = \
src/common/libor-testing.a \
 src/lib/libtor-log-testing.a \
 src/lib/libtor-lock-testing.a \
+src/lib/libtor-fdio-testing.a \
src/lib/libtor-container-testing.a \
 src/lib/libtor-string-testing.a \
src/lib/libtor-malloc-testing.a \
diff --git a/src/common/compat.c b/src/common/compat.c
index 1dcd06581..467c51d6e 100644
--- a/src/common/compat.c
+++ b/src/common/compat.c
@@ -733,80 +733,6 @@ tor_lockfile_unlock(tor_lockfile_t *lockfile)
   tor_free(lockfile);
 }
 
-/** @{ */
-/** Some old versions of Unix didn't define constants for these values,
- * and instead expect you to say 0, 1, or 2. */
-#ifndef SEEK_SET
-#define SEEK_SET 0
-#endif
-#ifndef SEEK_CUR
-#define SEEK_CUR 1
-#endif
-#ifndef SEEK_END
-#define SEEK_END 2
-#endif
-/** @} */
-
-/** Return the position of fd with respect to the start of the file. */
-off_t
-tor_fd_getpos(int fd)
-{
-#ifdef _WIN32
-  return (off_t) _lseek(fd, 0, SEEK_CUR);
-#else
-  return (off_t) lseek(fd, 0, SEEK_CUR);
-#endif
-}
-
-/** Move fd to the end of the file. Return -1 on error, 0 on success.
- * If the file is a pipe, do nothing and succeed.
- **/
-int
-tor_fd_seekend(int fd)
-{
-#ifdef _WIN32
-  return _lseek(fd, 0, SEEK_END) < 0 ? -1 : 0;
-#else
-  off_t rc = lseek(fd, 0, SEEK_END) < 0 ? -1 : 0;
-#ifdef ESPIPE
-  /* If we get an error and ESPIPE, then it's a pipe or a socket of a fifo:
-   * no need to worry. */
-  if (rc < 0 && errno == ESPIPE)
-rc = 0;
-#endif /* defined(ESPIPE) */
-  return (rc < 0) ? -1 : 0;
-#endif /* defined(_WIN32) */
-}
-
-/** Move fd to position pos in the file. Return -1 on error, 0
- * on success. */
-int
-tor_fd_setpos(int fd, off_t pos)
-{
-#ifdef _WIN32
-  return _lseek(fd, pos, SEEK_SET) < 0 ? -1 : 0;
-#else
-  return lseek(fd, pos, SEEK_SET) < 0 ? -1 : 0;
-#endif
-}
-
-/** Replacement for ftruncate(fd, 0): move to the front of the file and remove
- * all the rest of the file. Return -1 on error, 0 on success. */
-int
-tor_ftruncate(int fd)
-{
-  /* Rumor has it that some versions of ftruncate do not move the file pointer.
-   */
-  if (tor_fd_setpos(fd, 0) < 0)
-return -1;
-
-#ifdef _WIN32
-  return _chsize(fd, 0);
-#else
-  return ftruncate(fd, 0);
-#endif
-}
-
 #undef DEBUG_SOCKET_COUNTING
 #ifdef DEBUG_SOCKET_COUNTING
 /** A bitarray of all fds that should be passed to tor_socket_close(). Only
@@ -2641,7 +2567,6 @@ compute_num_cpus(void)
   return num_cpus;
 }
 
-
 /** As localtime_r, but defined for platforms that don't have it:
  *
  * Convert *timep to a struct tm in local time, and store the value in
diff --git a/src/common/compat.h b/src/common/compat.h
index 605438cc6..f91c22425 100644
--- a/src/common/compat.h
+++ b/src/common/compat.h
@@ -160,11 +160,6 @@ tor_lockfile_t *tor_lockfile_lock(const char *filename, 
int blocking,
   int *locked_out);
 void tor_lockfile_unlock(tor_lockfile_t *lockfile);
 
-off_t tor_fd_getpos(int fd);
-int tor_fd_setpos(int fd, off_t pos);
-int tor_fd_seekend(int fd);
-int tor_ftruncate(int fd);
-
 int64_t tor_get_avail_disk_space(const char *path);
 
 #ifdef _WIN32
diff --git a/src/common/util.c b/src/common/util.c
index 986792ff2..fa95f933c 100644
--- a/src/common/util.c
+++ b/src/common/util.c
@@ -19,6 +19,7 @@
 #include "lib/crypt_ops/crypto_digest.h"
 #include "lib/cc/torint.h"
 #include

[tor-commits] [tor/master] Move util_bug into libtor-log

2018-06-26 Thread nickm

commit bcf3e546d196910f53beb79814833c96f279e987
Author: Nick Mathewson 
Date:   Fri Jun 22 11:54:38 2018 -0400

Move util_bug into libtor-log
---
 src/common/address.h   | 2 +-
 src/common/include.am  | 2 --
 src/common/token_bucket.c  | 2 +-
 src/common/util.h  | 2 +-
 src/lib/container/.may_include | 2 +-
 src/lib/container/map.c| 2 +-
 src/lib/container/order.c  | 2 +-
 src/lib/container/smartlist.c  | 2 +-
 src/lib/log/include.am | 6 --
 src/{common => lib/log}/util_bug.c | 4 ++--
 src/{common => lib/log}/util_bug.h | 0
 11 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/src/common/address.h b/src/common/address.h
index 8abff072c..b95ee364a 100644
--- a/src/common/address.h
+++ b/src/common/address.h
@@ -15,7 +15,7 @@
 #include "orconfig.h"
 #include "lib/cc/torint.h"
 #include "common/compat.h"
-#include "common/util_bug.h"
+#include "lib/log/util_bug.h"
 
 #ifdef ADDRESS_PRIVATE
 
diff --git a/src/common/include.am b/src/common/include.am
index d4404fb1b..4927f5742 100644
--- a/src/common/include.am
+++ b/src/common/include.am
@@ -40,7 +40,6 @@ LIBOR_A_SRC = \
   src/common/confline.c\
   src/common/memarea.c \
   src/common/util.c\
-  src/common/util_bug.c\
   src/common/util_format.c \
   src/common/util_process.c\
   src/common/sandbox.c \
@@ -94,7 +93,6 @@ COMMONHEADERS = \
   src/common/timers.h  \
   src/common/token_bucket.h\
   src/common/util.h\
-  src/common/util_bug.h\
   src/common/util_format.h \
   src/common/util_process.h\
   src/common/workqueue.h
diff --git a/src/common/token_bucket.c b/src/common/token_bucket.c
index 502e8eac5..3a16720d7 100644
--- a/src/common/token_bucket.c
+++ b/src/common/token_bucket.c
@@ -20,7 +20,7 @@
 
 #include "common/token_bucket.h"
 #include "common/compat.h"
-#include "common/util_bug.h"
+#include "lib/log/util_bug.h"
 
 /**
  * Set the rate and burst value in a token_bucket_cfg.
diff --git a/src/common/util.h b/src/common/util.h
index 3f6476d75..d1b4e8fe9 100644
--- a/src/common/util.h
+++ b/src/common/util.h
@@ -32,7 +32,7 @@
 #include "lib/intmath/muldiv.h"
 #include "lib/intmath/cmp.h"
 #include "lib/log/ratelim.h"
-#include "common/util_bug.h"
+#include "lib/log/util_bug.h"
 
 #ifndef O_BINARY
 #define O_BINARY 0
diff --git a/src/lib/container/.may_include b/src/lib/container/.may_include
index 42458b295..1114ad845 100644
--- a/src/lib/container/.may_include
+++ b/src/lib/container/.may_include
@@ -13,4 +13,4 @@ ht.h
 siphash.h
 
 # XXX I'd like to remove this.
-common/util_bug.h
+lib/log/util_bug.h
diff --git a/src/lib/container/map.c b/src/lib/container/map.c
index ba59e37d6..5f280b316 100644
--- a/src/lib/container/map.c
+++ b/src/lib/container/map.c
@@ -17,7 +17,7 @@
 #include "lib/string/util_string.h"
 #include "lib/malloc/util_malloc.h"
 
-#include "common/util_bug.h"
+#include "lib/log/util_bug.h"
 
 #include 
 #include 
diff --git a/src/lib/container/order.c b/src/lib/container/order.c
index 599f83300..4fdd51d99 100644
--- a/src/lib/container/order.c
+++ b/src/lib/container/order.c
@@ -14,7 +14,7 @@
 #include 
 
 #include "lib/container/order.h"
-#include "common/util_bug.h"
+#include "lib/log/util_bug.h"
 
 /** Declare a function called funcname that acts as a find_nth_FOO
  * function for an array of type elt_t*.
diff --git a/src/lib/container/smartlist.c b/src/lib/container/smartlist.c
index 9d405085b..852a32810 100644
--- a/src/lib/container/smartlist.c
+++ b/src/lib/container/smartlist.c
@@ -21,7 +21,7 @@
 #include "lib/string/util_string.h"
 #include "lib/string/printf.h"
 
-#include "common/util_bug.h"
+#include "lib/log/util_bug.h"
 
 #include 
 #include 
diff --git a/src/lib/log/include.am b/src/lib/log/include.am
index f9f327e2d..e3041ed78 100644
--- a/src/lib/log/include.am
+++ b/src/lib/log/include.am
@@ -7,7 +7,8 @@ endif
 
 src_lib_libtor_log_a_SOURCES = \
src/lib/log/ratelim.c   \
-   src/lib/log/torlog.c
+   src/lib/log/torlog.c\
+   src/lib/log/util_bug.c
 
 src_lib_libtor_log_testing_a_SOURCES = \
$(src_lib_libtor_log_a_SOURCES)
@@ -16,4 +17,5 @@ src_lib_libtor_log_testing_a_CFLAGS = $(AM_CFLAGS) 
$(TEST_CFLAGS)
 
 noinst_HEADERS +=  \
src/lib/log/ratelim.h   \
-   src/lib/log/torlog.h
+   src/lib/log/torlog.h\
+   src/lib/log/util_bug.h
diff --git a/src/common/util_bug.c b/src/lib/log/util_bug.c
similarity index 98%
rename from src/common/util_bug.c

[tor-commits] [tor/master] Update the micro-revision.i dependencies, and add a stdlib.h

2018-06-26 Thread nickm

commit 4118ba67dbdaecdd93d6ba02584b19dc62e28916
Author: Nick Mathewson 
Date:   Fri Jun 22 13:25:52 2018 -0400

Update the micro-revision.i dependencies, and add a stdlib.h
---
 src/lib/container/smartlist.h | 1 +
 src/lib/log/include.am| 3 +++
 src/or/include.am | 4 ++--
 3 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/lib/container/smartlist.h b/src/lib/container/smartlist.h
index 7b1d06d7b..c202e134f 100644
--- a/src/lib/container/smartlist.h
+++ b/src/lib/container/smartlist.h
@@ -65,6 +65,7 @@ void smartlist_subtract(smartlist_t *sl1, const smartlist_t 
*sl2);
 /* smartlist_choose() is defined in crypto.[ch] */
 #ifdef DEBUG_SMARTLIST
 #include "lib/err/torerr.h"
+#include 
 /** Return the number of items in sl.
  */
 static inline int smartlist_len(const smartlist_t *sl);
diff --git a/src/lib/log/include.am b/src/lib/log/include.am
index e3041ed78..bbe345de7 100644
--- a/src/lib/log/include.am
+++ b/src/lib/log/include.am
@@ -15,6 +15,9 @@ src_lib_libtor_log_testing_a_SOURCES = \
 src_lib_libtor_log_testing_a_CPPFLAGS = $(AM_CPPFLAGS) $(TEST_CPPFLAGS)
 src_lib_libtor_log_testing_a_CFLAGS = $(AM_CFLAGS) $(TEST_CFLAGS)
 
+src/lib/log/torlog.$(OBJEXT) \
+  src/lib/log/src_lib_libtor_log_testing_a-torlog.$(OBJEXT): micro-revision.i
+
 noinst_HEADERS +=  \
src/lib/log/ratelim.h   \
src/lib/log/torlog.h\
diff --git a/src/or/include.am b/src/or/include.am
index a2532cd4d..9e80de1f5 100644
--- a/src/or/include.am
+++ b/src/or/include.am
@@ -143,8 +143,8 @@ endif
 
 src_or_tor_SOURCES = src/or/tor_main.c
 
-src/or/tor_main.$(OBJEXT) \
-  src/or/src_or_tor_cov-tor_main.$(OBJEXT): micro-revision.i
+src/or/git_revision.$(OBJEXT) \
+  src/or/src_or_libtor_app_testing_a-git_revision.$(OBJEXT): micro-revision.i
 
 AM_CPPFLAGS += -DSHARE_DATADIR="\"$(datadir)\""\
-DLOCALSTATEDIR="\"$(localstatedir)\""  \



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Update .gitignore and .may_include files

2018-06-26 Thread nickm

commit 7a93ce8f6382a2fe8687e7e42dc398e08480c901
Author: Nick Mathewson 
Date:   Fri Jun 22 11:46:44 2018 -0400

Update .gitignore and .may_include files
---
 .gitignore | 4 
 src/common/.may_include| 1 -
 src/lib/log/.may_include   | 2 ++
 src/lib/wallclock/.may_include | 2 ++
 4 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/.gitignore b/.gitignore
index f141fee17..025d7202d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -173,10 +173,14 @@ uptime-*.json
 /src/lib/libtor-ctime-testing.a
 /src/lib/libtor-err.a
 /src/lib/libtor-err-testing.a
+/src/lib/libtor-fdio.a
+/src/lib/libtor-fdio-testing.a
 /src/lib/libtor-intmath.a
 /src/lib/libtor-intmath-testing.a
 /src/lib/libtor-lock.a
 /src/lib/libtor-lock-testing.a
+/src/lib/libtor-log.a
+/src/lib/libtor-log-testing.a
 /src/lib/libtor-malloc.a
 /src/lib/libtor-malloc-testing.a
 /src/lib/libtor-string.a
diff --git a/src/common/.may_include b/src/common/.may_include
index 484b57e10..fab9ad0da 100644
--- a/src/common/.may_include
+++ b/src/common/.may_include
@@ -5,7 +5,6 @@ lib/*/*.h
 #  These all belong somewhere else
 ht.h
 linux_syscalls.inc
-micro-revision.i
 siphash.h
 src/ext/timeouts/timeout.c
 tor_queue.h
diff --git a/src/lib/log/.may_include b/src/lib/log/.may_include
index 4f6244c2d..36a164cce 100644
--- a/src/lib/log/.may_include
+++ b/src/lib/log/.may_include
@@ -3,7 +3,9 @@ orconfig.h
 lib/cc/*.h
 lib/container/smartlist.h
 lib/err/*.h
+lib/fdio/*.h
 lib/intmath/*.h
+lib/lock/*.h
 lib/log/*.h
 lib/malloc/*.h
 lib/string/*.h
diff --git a/src/lib/wallclock/.may_include b/src/lib/wallclock/.may_include
index 686d9196f..dc010da06 100644
--- a/src/lib/wallclock/.may_include
+++ b/src/lib/wallclock/.may_include
@@ -1,4 +1,6 @@
 orconfig.h
+lib/cc/*.h
 lib/err/*.h
 lib/wallclock/*.h
+lib/string/*.h
 lib/testsupport/*.h



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

[tor-commits] [tor/master] Remove an "m" that did not belong.

2018-06-26 Thread nickm

commit 76a717890e6c491d0ecd696687a9ba842f143729
Author: Nick Mathewson 
Date:   Fri Jun 22 13:32:47 2018 -0400

Remove an "m" that did not belong.
---
 src/lib/wallclock/tm_cvt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lib/wallclock/tm_cvt.c b/src/lib/wallclock/tm_cvt.c
index bb23d1bef..cf73cc542 100644
--- a/src/lib/wallclock/tm_cvt.c
+++ b/src/lib/wallclock/tm_cvt.c
@@ -145,7 +145,7 @@ tor_localtime_r_msg(const time_t *timep, struct tm *result, 
char **err_out)
   r = localtime(timep);
   if (r)
 memcpy(result, r, sizeof(struct tm));
-  return correct_tm(1, timep, result, rm, err_out);
+  return correct_tm(1, timep, result, r, err_out);
 }
 #endif /* defined(HAVE_LOCALTIME_R) || ... */
 /** @} */



___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

1 2 3 4 >

1 - 100 of 321 matches

Mail list logo