[tor-commits] [translation/support-faq] Update translations for support-faq
commit e807097d86252ba084529045ae51b3fe0e1416cd
Author: Translation commit bot
Date: Thu May 31 05:19:04 2018 +
Update translations for support-faq
---
el.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/el.json b/el.json
index a4a631779..985abfdab 100644
--- a/el.json
+++ b/el.json
@@ -26,7 +26,7 @@
"faq-5": {
"id": "#faq-5",
"control": "faq-5",
- "title": "Can I use a VPN with Tor?",
+ "title": "ÎÏÎ¿Ï Î½Î± ÏÏηÏιμοÏοιήÏÏ VPN με Ïο Tor;",
"description": "Generally speaking, we don't
recommend using a VPN with Tor unless you're an advanced user who knows how to
configure both in a way that doesn't compromise your privacy. Learn more about
combining Tor + VPN."
}
}
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [translation/support-miscellaneous] Update translations for support-miscellaneous
commit 36d71fb5ae3f0226703a45aa557120d922baa5aa
Author: Translation commit bot
Date: Thu May 31 05:19:28 2018 +
Update translations for support-miscellaneous
---
el.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/el.json b/el.json
index c29993d09..94d5b8fc1 100644
--- a/el.json
+++ b/el.json
@@ -74,7 +74,7 @@
"misc-13": {
"id": "#misc-13",
"control": "misc-13",
- "title": "Can I use a VPN with Tor?",
+ "title": "ÎÏÎ¿Ï Î½Î± ÏÏηÏιμοÏοιήÏÏ VPN με Ïο Tor;",
"description": "Generally speaking, we don't
recommend using a VPN with Tor unless you're an advanced user who knows how to
configure both in a way that doesn't compromise your privacy. You can find more
detailed information about Tor + VPN at https://trac.torproject.org/projects/tor/wiki/doc/TorPlusVPN\;>âour
wiki."
},
"misc-14": {
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [torspec/master] Add a new proposal to help us move forward with 275.
commit a323d84e7c8b83676010dd78554a455ad2f869ce Author: Nick Mathewson Date: Wed May 30 17:19:54 2018 -0700 Add a new proposal to help us move forward with 275. --- proposals/000-index.txt| 2 ++ proposals/293-know-when-to-publish.txt | 58 ++ 2 files changed, 60 insertions(+) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index b70e998..2d7a595 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -213,6 +213,7 @@ Proposals by number: 290 Continuously update consensus methods [OPEN] 291 The move to two guard nodes [OPEN] 292 Mesh-based vanguards [OPEN] +293 Other ways for relays to know when to publish [OPEN] Proposals by status: @@ -273,6 +274,7 @@ Proposals by status: 290 Continuously update consensus methods 291 The move to two guard nodes 292 Mesh-based vanguards + 293 Other ways for relays to know when to publish [for 0.3.5] ACCEPTED: 172 GETINFO controller option for circuit information 173 GETINFO Option Expansion diff --git a/proposals/293-know-when-to-publish.txt b/proposals/293-know-when-to-publish.txt new file mode 100644 index 000..4a46730 --- /dev/null +++ b/proposals/293-know-when-to-publish.txt @@ -0,0 +1,58 @@ +Filename: 293-know-when-to-publish.txt +Title: Other ways for relays to know when to publish +Author: Nick Mathewson +Created: 30-May-2018 +Status: Open +Target: 0.3.5 + + +1. Motivation + + In proposal 275, we give reasons for dropping the published-on + field from consensus documents, to improve the performance of + consensus diffs. We've already changed Tor (as of 0.2.9.11) to + allow us to set those fields far in the future -- but + unfortunately, there is still one use case that requires them: + relays use the published-on field to tell if they are about to fall + out of the consensus and need to make new descriptors. + + Here we propose two alternative mechanisms for relays to know that + they should publish descriptors, so we can enact proposal 275 and + set the published-on field to some time in the distant future. + + +2. Mechanism One: The StaleDesc flag + + Authorities should begin voting aon a new StaleDesc flag. + + When authorities vote, if the most recent published_on date for + a descriptor has over DESC_IS_STALE_INTERVAL in the past, the + authorities should vote to give the StaleDesc flag to that relay. + + If any relay sees that it has the StaleDesc flag, it should upload + some time in the first half of the voting interval. (Implementors + should take care not to re-upload over and over, though: Relays won't + lose the flag until the next voting interval is reached.) + + (Define DESC_IS_STALE_INTERVAL as equal to + FORCE_REGENERATE_DESCRIPTOR_INTERVAL.) + + +3. Mechanism Two: Uploading more frequently when rejected. + + Tor relays should remember the last time at which they uploaded a + descriptor that was accepted by a majority of dirauths. If this + time is more than FAST_RETRY_DESCRIPTOR_INTERVAL in the past, we + mark our descriptor as dirty from + mark_my_descriptor_dirty_if_too_old(). + + +4. Implications for proposal 275 + + Once most relays are running verions that support the features + above, and once authorities are generating consensuses with the + StaleDesc flag, there will no longer be a need to keep the + published time in consensus documents accurate -- we can start + setting it to some time in the distant future, per proposal 275. + + ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [webwml/master] Update Colin's people page description
commit 2c3728c4926da05662760f39a47d5048f759f277 Author: Damian Johnson Date: Wed May 30 10:03:09 2018 -0700 Update Colin's people page description Adjusting Colin's description per his request. --- about/en/corepeople.wml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/about/en/corepeople.wml b/about/en/corepeople.wml index 3e298128..5010e17e 100644 --- a/about/en/corepeople.wml +++ b/about/en/corepeople.wml @@ -156,7 +156,7 @@ https://db.torproject.org/fetchkey.cgi?fingerprint=726824BE9D8D5CF009C5F039FF9D1C57779FB883;> Colin Childs IRC: Phoul - Support and translation coordinator, GSoC administrator, member of the community team and a director of https://www.coldhak.ca;>Coldhak. + Relay advocate, Outreachy mentor, member of the community team and a director of https://www.coldhak.ca/;>Coldhak. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [tor/master] Fix various typos
commit d7bbfd0f6217181fc7f4a4a71d23874873a5262a
Author: Nick Mathewson
Date: Wed May 30 07:57:22 2018 -0700
Fix various typos
---
ReleaseNotes| 2 +-
scripts/test/appveyor-irc-notify.py | 2 +-
src/common/crypto_rand.h| 2 +-
src/test/test_controller.c | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/ReleaseNotes b/ReleaseNotes
index d36f87eea..5db488592 100644
--- a/ReleaseNotes
+++ b/ReleaseNotes
@@ -529,7 +529,7 @@ Changes in version 0.3.3.6 - 2018-05-22
would call the Rust implementation of
protover_get_supported_protocols(). This was due to the C version
returning a static string, whereas the Rust version newly allocated
- a CString to pass accross the FFI boundary. Consequently, the C
+ a CString to pass across the FFI boundary. Consequently, the C
code was not expecting to need to free() what it was given. Fixes
bug 25127; bugfix on 0.3.2.1-alpha.
diff --git a/scripts/test/appveyor-irc-notify.py
b/scripts/test/appveyor-irc-notify.py
index 44ef4c96f..4ffea5268 100644
--- a/scripts/test/appveyor-irc-notify.py
+++ b/scripts/test/appveyor-irc-notify.py
@@ -169,7 +169,7 @@ def notify():
response = line.split()
if response[0] == 'PING':
-irc_file.send('PONG {}\r\n'.format(reponse[1]).encode())
+irc_file.send('PONG {}\r\n'.format(response[1]).encode())
elif response[1] == '433':
irc_sock.send('NICK {}\r\n'.format(irc_nick).encode())
diff --git a/src/common/crypto_rand.h b/src/common/crypto_rand.h
index 4eac94f57..bb02e5100 100644
--- a/src/common/crypto_rand.h
+++ b/src/common/crypto_rand.h
@@ -7,7 +7,7 @@
/**
* \file crypto_rand.h
*
- * \brief Common functions for using (psuedo-)random number generators.
+ * \brief Common functions for using (pseudo-)random number generators.
**/
#ifndef TOR_CRYPTO_RAND_H
diff --git a/src/test/test_controller.c b/src/test/test_controller.c
index 3b58a78e1..1a350f66c 100644
--- a/src/test/test_controller.c
+++ b/src/test/test_controller.c
@@ -1470,7 +1470,7 @@ test_download_status_bridge(void *arg)
return;
}
-/** Set timeval to a mock date and time. This is neccessary
+/** Set timeval to a mock date and time. This is necessary
* to make tor_gettimeofday() mockable. */
static void
mock_tor_gettimeofday(struct timeval *timeval)
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [onionoo/master] Bump version to 6.0-1.14.0-dev.
commit 13d53601b1ab2a769f888f00548ab6b692aa9d10
Author: Karsten Loesing
Date: Wed May 30 15:58:13 2018 +0200
Bump version to 6.0-1.14.0-dev.
---
build.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build.xml b/build.xml
index 596fb84..e1fb1ab 100644
--- a/build.xml
+++ b/build.xml
@@ -10,7 +10,7 @@
+value="${onionoo.protocol.version}-1.14.0-dev"/>
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [onionoo/master] Replace Gson with Jackson.
commit 73f8dedfc994dd6476570798c576060bcffc6fa6
Author: Karsten Loesing
Date: Wed May 23 10:15:36 2018 +0200
Replace Gson with Jackson.
Update metrics-lib dependency to 2.4.0.
Adapt touched logging statements to our standards.
Implements #25848.
---
CHANGELOG.md | 6 +++
build.xml | 6 ++-
.../org/torproject/onionoo/docs/DetailsStatus.java | 6 +--
.../org/torproject/onionoo/docs/DocumentStore.java | 57 +---
.../torproject/onionoo/docs/SummaryDocument.java | 63 ++
.../torproject/onionoo/server/ResponseBuilder.java | 26 ++---
.../onionoo/server/ResourceServletTest.java| 20 +--
.../onionoo/writer/GraphHistoryCompilerTest.java | 17 --
8 files changed, 126 insertions(+), 75 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 199cf6f..7a792fc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,9 @@
+# Changes in version 6.0-1.??.? - 2018-05-??
+
+ * Medium changes
+ - Replace Gson with Jackson.
+
+
# Changes in version 6.0-1.13.0 - 2018-04-17
* Medium changes
diff --git a/build.xml b/build.xml
index 557297d..7d752e2 100644
--- a/build.xml
+++ b/build.xml
@@ -11,7 +11,7 @@
-
+
@@ -47,7 +47,9 @@
-
+
+
+
diff --git a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
index f838ec0..7ee0bb6 100644
--- a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
+++ b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
@@ -34,12 +34,12 @@ public class DetailsStatus extends Document {
private String desc_published;
public void setDescPublished(Long descPublished) {
-this.desc_published = DateTimeHelper.format(descPublished);
+this.desc_published = null == descPublished ? null
+: DateTimeHelper.format(descPublished);
}
public Long getDescPublished() {
-return this.desc_published == null ? null :
-DateTimeHelper.parse(this.desc_published);
+return DateTimeHelper.parse(this.desc_published);
}
private String last_restarted;
diff --git a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
index f1f3803..3fac5c9 100644
--- a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
+++ b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
@@ -5,9 +5,12 @@ package org.torproject.onionoo.docs;
import org.torproject.onionoo.util.FormattingUtils;
-import com.google.gson.Gson;
-import com.google.gson.GsonBuilder;
-import com.google.gson.JsonParseException;
+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.PropertyAccessor;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.PropertyNamingStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -41,6 +44,12 @@ public class DocumentStore {
private static Logger log = LoggerFactory.getLogger(
DocumentStore.class);
+ private static ObjectMapper objectMapper = new ObjectMapper()
+ .setPropertyNamingStrategy(PropertyNamingStrategy.SNAKE_CASE)
+ .setSerializationInclusion(JsonInclude.Include.NON_EMPTY)
+ .setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.NONE)
+ .setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);
+
private final File statusDir = new File("status");
private File outDir = null;
@@ -162,12 +171,11 @@ public class DocumentStore {
String line = null;
try (BufferedReader br = new BufferedReader(new FileReader(
summaryFile))) {
- Gson gson = new Gson();
while ((line = br.readLine()) != null) {
if (line.length() == 0) {
continue;
}
-SummaryDocument summaryDocument = gson.fromJson(line,
+SummaryDocument summaryDocument = objectMapper.readValue(line,
SummaryDocument.class);
if (summaryDocument != null) {
parsedSummaryDocuments.put(summaryDocument.getFingerprint(),
@@ -178,11 +186,8 @@ public class DocumentStore {
this.listedFiles += parsedSummaryDocuments.size();
this.listOperations++;
} catch (IOException e) {
- log.error("Could not read file '"
- + summaryFile.getAbsolutePath() + "'.", e);
-} catch (JsonParseException e) {
- log.error("Could not parse summary document '" + line
- + "' in file '" + summaryFile.getAbsolutePath() + "'.", e);
+ log.error("Could not parse summary document '{}' from file '{}'.",
+ line,
[tor-commits] [onionoo/release] Prepare for 6.0-1.14.0 release.
commit fac35760987b60168abb6afc6029a6bd50d8ce04
Author: Karsten Loesing
Date: Tue May 29 09:20:13 2018 +0200
Prepare for 6.0-1.14.0 release.
---
CERT | 18 +-
CHANGELOG.md | 2 +-
build.xml| 2 +-
3 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/CERT b/CERT
index 43be56e..61fa7c6 100644
--- a/CERT
+++ b/CERT
@@ -1,8 +1,8 @@
-BEGIN CERTIFICATE-
-MIIDaTCCAlGgAwIBAgIEIk6NnzANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV
+MIIDaTCCAlGgAwIBAgIENNGkczANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV
UzELMAkGA1UECBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBU
b3IgUHJvamVjdCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwHhcNMTgw
-MjI2MTQwMzUzWhcNMTgwNTI3MTQwMzUzWjBlMQswCQYDVQQGEwJVUzELMAkGA1UE
+NTIzMTUxNDU2WhcNMTgwODIxMTUxNDU2WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE
CBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBUb3IgUHJvamVj
dCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQChXn+IUp+o6G+k4ffxk3TkxZb3iXfiG7byNsG63olU
@@ -11,11 +11,11 @@
Qw+VAhKTcEIv4yiR0BWapQyR07pgmKirYVjN6s6ef8NJzUptpxLlaYJ3ZfQfc4aE
MXzScgaccwDFIWQ661lzLGCfeSxxa3Xy4wWsGwzNzLITYrrABcbg7yogLo2btNvD
oEwGL3/baQdhl0dra6biVCZr9ydn3Hg57S55pUU0rBY25id78zUO8xrfNHw54wwX
lOblGt75OOkahP/ZZSBxxoiknJ6y5VQV8y+noA4vigXFAgMBAAGjITAfMB0GA1Ud
-DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAlUkU
-qqf+4yfXwAWFr2q6iijr54NDDEQwybCblIzVnsuGHPUDuie3ZWSHirtblBs/uJ9x
-RxmwkBrJr9IGMmGhN2GKXIPeUH0EZBYo7bsgo5d+E61OCnd/O+1JZzdG9dK+0kfq
-MLfo6ltFZZouHIIXfvOm8sLLRrdkXPrLQ/E8fTHB7dL6T8Hqg6pHRrRZDtuSM9CO
-zSYropxqlFzzlzciOdTU05D8Cnx2j/RtaycxHxFS7QtriDB0uOfqvyiVeqpr72wG
-qetlu3h46fXj3ALGVSXy+YZpYxcRNZsQyiBXdlXbgY0OfOVPFOH3HiZuv3zhfRJW
-2DiJiA8BLxZToe2XDA==
+DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAJtEQ
+B2AVpVtjTGU7uujUtEjSn1ICv7QLW/37TmHn+m1SWvXOVlSG81eb5JwX449Pn3w5
+K1Bx0JJmJ6Kec2kHtoR1r5/DUlTMtRqKQ5ZHQbvYMx+Ifq8TRALHjcw8p5Vw+gep
+0zYbSQycklzFaFqLB8Cus0ICb+UY54HTddpezQ3IXS/vc4Vy07ocm8rUCz1s8L/r
+ehTOxSym7G7+kcRzNplFOLL5iO8o6uHPyLR1TIAIXa0XZ41ogNl2q+6CvXG+UI/j
+qkzSXD6FJK1Px2nxNFIe/w9NL+chSytIGnV3CImyiORuV+1OxMglyQspSGEbl1XP
+OfiTEYHnp12BYMeRyw==
-END CERTIFICATE-
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7a792fc..7bc8985 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,4 @@
-# Changes in version 6.0-1.??.? - 2018-05-??
+# Changes in version 6.0-1.14.0 - 2018-05-29
* Medium changes
- Replace Gson with Jackson.
diff --git a/build.xml b/build.xml
index 7d752e2..596fb84 100644
--- a/build.xml
+++ b/build.xml
@@ -10,7 +10,7 @@
+value="${onionoo.protocol.version}-1.14.0"/>
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [onionoo/release] Bump version to 6.0-1.13.0-dev.
commit 960ded0162b99d3ed8b7af976d92eb422c5969d0
Author: Karsten Loesing
Date: Wed Apr 18 16:42:38 2018 +0200
Bump version to 6.0-1.13.0-dev.
---
build.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build.xml b/build.xml
index 315989b..557297d 100644
--- a/build.xml
+++ b/build.xml
@@ -10,7 +10,7 @@
+value="${onionoo.protocol.version}-1.13.0-dev"/>
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [onionoo/release] Replace Gson with Jackson.
commit 73f8dedfc994dd6476570798c576060bcffc6fa6
Author: Karsten Loesing
Date: Wed May 23 10:15:36 2018 +0200
Replace Gson with Jackson.
Update metrics-lib dependency to 2.4.0.
Adapt touched logging statements to our standards.
Implements #25848.
---
CHANGELOG.md | 6 +++
build.xml | 6 ++-
.../org/torproject/onionoo/docs/DetailsStatus.java | 6 +--
.../org/torproject/onionoo/docs/DocumentStore.java | 57 +---
.../torproject/onionoo/docs/SummaryDocument.java | 63 ++
.../torproject/onionoo/server/ResponseBuilder.java | 26 ++---
.../onionoo/server/ResourceServletTest.java| 20 +--
.../onionoo/writer/GraphHistoryCompilerTest.java | 17 --
8 files changed, 126 insertions(+), 75 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 199cf6f..7a792fc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,9 @@
+# Changes in version 6.0-1.??.? - 2018-05-??
+
+ * Medium changes
+ - Replace Gson with Jackson.
+
+
# Changes in version 6.0-1.13.0 - 2018-04-17
* Medium changes
diff --git a/build.xml b/build.xml
index 557297d..7d752e2 100644
--- a/build.xml
+++ b/build.xml
@@ -11,7 +11,7 @@
-
+
@@ -47,7 +47,9 @@
-
+
+
+
diff --git a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
index f838ec0..7ee0bb6 100644
--- a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
+++ b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java
@@ -34,12 +34,12 @@ public class DetailsStatus extends Document {
private String desc_published;
public void setDescPublished(Long descPublished) {
-this.desc_published = DateTimeHelper.format(descPublished);
+this.desc_published = null == descPublished ? null
+: DateTimeHelper.format(descPublished);
}
public Long getDescPublished() {
-return this.desc_published == null ? null :
-DateTimeHelper.parse(this.desc_published);
+return DateTimeHelper.parse(this.desc_published);
}
private String last_restarted;
diff --git a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
index f1f3803..3fac5c9 100644
--- a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
+++ b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java
@@ -5,9 +5,12 @@ package org.torproject.onionoo.docs;
import org.torproject.onionoo.util.FormattingUtils;
-import com.google.gson.Gson;
-import com.google.gson.GsonBuilder;
-import com.google.gson.JsonParseException;
+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.annotation.PropertyAccessor;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.PropertyNamingStrategy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -41,6 +44,12 @@ public class DocumentStore {
private static Logger log = LoggerFactory.getLogger(
DocumentStore.class);
+ private static ObjectMapper objectMapper = new ObjectMapper()
+ .setPropertyNamingStrategy(PropertyNamingStrategy.SNAKE_CASE)
+ .setSerializationInclusion(JsonInclude.Include.NON_EMPTY)
+ .setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.NONE)
+ .setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);
+
private final File statusDir = new File("status");
private File outDir = null;
@@ -162,12 +171,11 @@ public class DocumentStore {
String line = null;
try (BufferedReader br = new BufferedReader(new FileReader(
summaryFile))) {
- Gson gson = new Gson();
while ((line = br.readLine()) != null) {
if (line.length() == 0) {
continue;
}
-SummaryDocument summaryDocument = gson.fromJson(line,
+SummaryDocument summaryDocument = objectMapper.readValue(line,
SummaryDocument.class);
if (summaryDocument != null) {
parsedSummaryDocuments.put(summaryDocument.getFingerprint(),
@@ -178,11 +186,8 @@ public class DocumentStore {
this.listedFiles += parsedSummaryDocuments.size();
this.listOperations++;
} catch (IOException e) {
- log.error("Could not read file '"
- + summaryFile.getAbsolutePath() + "'.", e);
-} catch (JsonParseException e) {
- log.error("Could not parse summary document '" + line
- + "' in file '" + summaryFile.getAbsolutePath() + "'.", e);
+ log.error("Could not parse summary document '{}' from file '{}'.",
+ line,
[tor-commits] [onionoo/master] Prepare for 6.0-1.14.0 release.
commit fac35760987b60168abb6afc6029a6bd50d8ce04
Author: Karsten Loesing
Date: Tue May 29 09:20:13 2018 +0200
Prepare for 6.0-1.14.0 release.
---
CERT | 18 +-
CHANGELOG.md | 2 +-
build.xml| 2 +-
3 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/CERT b/CERT
index 43be56e..61fa7c6 100644
--- a/CERT
+++ b/CERT
@@ -1,8 +1,8 @@
-BEGIN CERTIFICATE-
-MIIDaTCCAlGgAwIBAgIEIk6NnzANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV
+MIIDaTCCAlGgAwIBAgIENNGkczANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV
UzELMAkGA1UECBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBU
b3IgUHJvamVjdCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwHhcNMTgw
-MjI2MTQwMzUzWhcNMTgwNTI3MTQwMzUzWjBlMQswCQYDVQQGEwJVUzELMAkGA1UE
+NTIzMTUxNDU2WhcNMTgwODIxMTUxNDU2WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE
CBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBUb3IgUHJvamVj
dCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQChXn+IUp+o6G+k4ffxk3TkxZb3iXfiG7byNsG63olU
@@ -11,11 +11,11 @@
Qw+VAhKTcEIv4yiR0BWapQyR07pgmKirYVjN6s6ef8NJzUptpxLlaYJ3ZfQfc4aE
MXzScgaccwDFIWQ661lzLGCfeSxxa3Xy4wWsGwzNzLITYrrABcbg7yogLo2btNvD
oEwGL3/baQdhl0dra6biVCZr9ydn3Hg57S55pUU0rBY25id78zUO8xrfNHw54wwX
lOblGt75OOkahP/ZZSBxxoiknJ6y5VQV8y+noA4vigXFAgMBAAGjITAfMB0GA1Ud
-DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAlUkU
-qqf+4yfXwAWFr2q6iijr54NDDEQwybCblIzVnsuGHPUDuie3ZWSHirtblBs/uJ9x
-RxmwkBrJr9IGMmGhN2GKXIPeUH0EZBYo7bsgo5d+E61OCnd/O+1JZzdG9dK+0kfq
-MLfo6ltFZZouHIIXfvOm8sLLRrdkXPrLQ/E8fTHB7dL6T8Hqg6pHRrRZDtuSM9CO
-zSYropxqlFzzlzciOdTU05D8Cnx2j/RtaycxHxFS7QtriDB0uOfqvyiVeqpr72wG
-qetlu3h46fXj3ALGVSXy+YZpYxcRNZsQyiBXdlXbgY0OfOVPFOH3HiZuv3zhfRJW
-2DiJiA8BLxZToe2XDA==
+DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAJtEQ
+B2AVpVtjTGU7uujUtEjSn1ICv7QLW/37TmHn+m1SWvXOVlSG81eb5JwX449Pn3w5
+K1Bx0JJmJ6Kec2kHtoR1r5/DUlTMtRqKQ5ZHQbvYMx+Ifq8TRALHjcw8p5Vw+gep
+0zYbSQycklzFaFqLB8Cus0ICb+UY54HTddpezQ3IXS/vc4Vy07ocm8rUCz1s8L/r
+ehTOxSym7G7+kcRzNplFOLL5iO8o6uHPyLR1TIAIXa0XZ41ogNl2q+6CvXG+UI/j
+qkzSXD6FJK1Px2nxNFIe/w9NL+chSytIGnV3CImyiORuV+1OxMglyQspSGEbl1XP
+OfiTEYHnp12BYMeRyw==
-END CERTIFICATE-
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7a792fc..7bc8985 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,4 @@
-# Changes in version 6.0-1.??.? - 2018-05-??
+# Changes in version 6.0-1.14.0 - 2018-05-29
* Medium changes
- Replace Gson with Jackson.
diff --git a/build.xml b/build.xml
index 7d752e2..596fb84 100644
--- a/build.xml
+++ b/build.xml
@@ -10,7 +10,7 @@
+value="${onionoo.protocol.version}-1.14.0"/>
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/master] Bump version to 1.1.0-dev.
commit 9ff54625841590904b0198dbac0b6e12d041938e Author: Karsten Loesing Date: Wed May 30 15:46:39 2018 +0200 Bump version to 1.1.0-dev. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index 8baa764..acf7f51 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ - + https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Make all parameters in write_* functions optional.
commit 167e72b5a06cec3753f7f952fb3e3247bae943a5
Author: Karsten Loesing
Date: Fri May 11 11:54:45 2018 +0200
Make all parameters in write_* functions optional.
We now permit parameters in write_* functions to be omitted. The
effect is that we're not filtering if a parameter is missing, thus
producing a CSV file with more rows.
At the same time we're adding columns for data that was previously
pre-determined by parameter values. For example, if a user specified a
given country in a parameter, we didn't have to include a country
column containing only that country. Now we need to put that column
back.
Implements #25383.
---
src/main/R/rserver/graphs.R| 369 -
.../torproject/metrics/web/RObjectGenerator.java | 4 +
2 files changed, 222 insertions(+), 151 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index ebb8c80..a9b7fc7 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -351,8 +351,13 @@ robust_call <- function(wrappee, filename) {
prepare_networksize <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), flag == "",
- country == "", version == "", platform == "", ec2bridge == "") %>%
+filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
+filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(flag == "") %>%
+filter(country == "") %>%
+filter(version == "") %>%
+filter(platform == "") %>%
+filter(ec2bridge == "") %>%
select(date, relays, bridges)
}
@@ -373,16 +378,21 @@ plot_networksize <- function(start, end, path) {
ggsave(filename = path, width = 8, height = 5, dpi = 150)
}
-write_networksize <- function(start, end, path) {
+write_networksize <- function(start = NULL, end = NULL, path) {
prepare_networksize(start, end) %>%
-write.csv(path, quote = FALSE, row.names = FALSE)
+write.csv(path, quote = FALSE, row.names = FALSE, na = "")
}
prepare_versions <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), flag == "",
- country == "", version != "", platform == "", ec2bridge == "") %>%
+filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
+filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(flag == "") %>%
+filter(country == "") %>%
+filter(version != "") %>%
+filter(platform == "") %>%
+filter(ec2bridge == "") %>%
select(date, version, relays)
}
@@ -411,17 +421,22 @@ plot_versions <- function(start, end, path) {
ggsave(filename = path, width = 8, height = 5, dpi = 150)
}
-write_versions <- function(start, end, path) {
+write_versions <- function(start = NULL, end = NULL, path) {
prepare_versions(start, end) %>%
spread(key = "version", value = "relays", fill = 0) %>%
-write.csv(path, quote = FALSE, row.names = FALSE)
+write.csv(path, quote = FALSE, row.names = FALSE, na = "")
}
prepare_platforms <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), flag == "",
- country == "", version == "", platform != "", ec2bridge == "") %>%
+filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
+filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(flag == "") %>%
+filter(country == "") %>%
+filter(version == "") %>%
+filter(platform != "") %>%
+filter(ec2bridge == "") %>%
select(date, platform, relays) %>%
mutate(platform = ifelse(platform == "Darwin", "macOS",
as.character(platform)))
@@ -442,17 +457,19 @@ plot_platforms <- function(start, end, path) {
ggsave(filename = path, width = 8, height = 5, dpi = 150)
}
-write_platforms <- function(start, end, path) {
+write_platforms <- function(start = NULL, end = NULL, path) {
prepare_platforms(start, end) %>%
spread(platform, relays) %>%
-write.csv(path, quote = FALSE, row.names = FALSE)
+write.csv(path, quote = FALSE, row.names = FALSE, na = "")
}
prepare_bandwidth <- function(start, end) {
read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), isexit != "",
- isguard != "") %>%
+filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
+filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(isexit != "") %>%
+filter(isguard != "") %>%
group_by(date) %>%
summarize(advbw = sum(advbw) * 8 / 1e9,
bwhist = sum(bwread + bwwrite) * 8 / 2e9) %>%
@@ -477,16 +494,18 @@ plot_bandwidth <-
[tor-commits] [metrics-web/release] Avoid underscores in write_* function parameters.
commit 3f1cafdb5c4dffecbeab2a6688218d91f35c79ce
Author: Karsten Loesing
Date: Thu May 10 21:30:54 2018 +0200
Avoid underscores in write_* function parameters.
We added underscores to parameters in write_* functions when they
would otherwise conflict with columns in the processed data. For
example, if a graph supports a `country` parameter and the data also
contains a `country` column, dplyr/tidyr won't know which `country` we
mean. That's why we renamed the parameter to `country_`.
However, we're soon going to make parameters optional, and if R
receives a couple of parameters of which one has the name `country`,
it can't match that to its `country_` parameter. We need to change the
parameter back to `country` for this to work, which conflicts with the
issue we were fixing earlier.
Turns out there's a way to use the same name for parameter and data
column: whenever we want to use the parameter, we use the
quasiquotation operator `!!` which evaluates its argument early and
inlines the result; and whenever we want to refer to the data column,
we just refer to it by name, without that operator.
Prepares #25383.
---
src/main/R/rserver/graphs.R | 26 +-
1 file changed, 13 insertions(+), 13 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 2ac2756..ebb8c80 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -629,14 +629,14 @@ plot_torperf <- function(start, end, source, server,
filesize, path) {
# harder than for other functions, because plot_torperf uses different
# colours based on which sources exist, unrelated to which source is
# plotted. Left as future work.
-write_torperf <- function(start, end, source_, server_, filesize_, path) {
+write_torperf <- function(start, end, source, server, filesize, path) {
read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
filter(date >= as.Date(start), date <= as.Date(end),
- filesize == ifelse(filesize_ == "50kb", 50 * 1024,
-ifelse(filesize_ == "1mb", 1024 * 1024, 5 * 1024 * 1024)),
- source == ifelse(source_ == "all", "", source_),
- server == server_) %>%
+ filesize == ifelse(!!filesize == "50kb", 50 * 1024,
+ifelse(!!filesize == "1mb", 1024 * 1024, 5 * 1024 * 1024)),
+ source == ifelse(!!source == "all", "", !!source),
+ server == !!server) %>%
transmute(date, q1 = q1 / 1e3, md = md / 1e3, q3 = q3 / 1e3) %>%
write.csv(path, quote = FALSE, row.names = FALSE)
}
@@ -921,18 +921,18 @@ plot_userstats_bridge_version <- function(start, end,
version, path) {
plot_userstats(start, end, "bridge", "version", version, "off", path)
}
-write_userstats_relay_country <- function(start, end, country_, events,
+write_userstats_relay_country <- function(start, end, country, events,
path) {
load(paste(rdata_dir, "clients-relay.RData", sep = ""))
u <- data %>%
filter(date >= as.Date(start), date <= as.Date(end),
- country == ifelse(country_ == "all", "", country_), transport == "",
+ country == ifelse(!!country == "all", "", !!country), transport == "",
version == "")
- if (country_ != "all" && events == "on") {
+ if (country != "all" && events == "on") {
u <- u %>%
mutate(downturns = clients < u$lower, upturns = clients > upper) %>%
select(date, clients, downturns, upturns, lower, upper)
- } else if (country_ != "all" && events != "off") {
+ } else if (country != "all" && events != "off") {
u <- u %>%
mutate(downturns = clients < u$lower, upturns = clients > upper) %>%
select(date, clients, downturns, upturns)
@@ -945,11 +945,11 @@ write_userstats_relay_country <- function(start, end,
country_, events,
write.csv(path, quote = FALSE, row.names = FALSE)
}
-write_userstats_bridge_country <- function(start, end, country_, path) {
+write_userstats_bridge_country <- function(start, end, country, path) {
load(paste(rdata_dir, "clients-bridge.RData", sep = ""))
data %>%
filter(date >= as.Date(start), date <= as.Date(end),
- country == ifelse(country_ == "all", "", country_), transport == "",
+ country == ifelse(!!country == "all", "", !!country), transport == "",
version == "") %>%
select(date, clients) %>%
rename(users = clients) %>%
@@ -982,11 +982,11 @@ write_userstats_bridge_transport <- function(start, end,
transports, path) {
write.csv(path, quote = FALSE, row.names = FALSE)
}
-write_userstats_bridge_version <- function(start, end, version_, path) {
+write_userstats_bridge_version <- function(start, end, version, path) {
load(paste(rdata_dir, "clients-bridge.RData", sep = ""))
data %>%
filter(date >= as.Date(start), date <= as.Date(end),
- country == "", transport == "", version == version_) %>%
+ country == "", transport == "", version ==
[tor-commits] [metrics-web/release] Fix flaw in noise-removing code of hidserv module.
commit 52b1051fb14f4091d3e1e02b0497bad5b0d66ca6
Author: Karsten Loesing
Date: Fri May 18 16:29:42 2018 +0200
Fix flaw in noise-removing code of hidserv module.
In the technical report that the hidserv module is based on we write:
"When processing hidden-service statistics, we need to handle the fact
that they have been obfuscated by relays. As first step, we're
attempting to remove the additive Laplace-distributed noise by
rounding up or down to the nearest multiple of bin_size. The idea is
that it's most likely that noise was added to the closest right side
of a bin than to the right side of another bin. In step two, we're
subtracting half of bin_size, because the relay added between 0 and
bin_size â 1 to the originally observed value."
However, our code has a flaw: we use integer truncation rather which
always rounds toward zero, whereas we really want to use the floor
function which rounds towards negative infinity.
The fix is to use Math.floorDiv() for the division rather than common
integer division and truncation.
Fixes #26022.
---
src/main/java/org/torproject/metrics/stats/hidserv/Parser.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
b/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
index 2423526..9c95db5 100644
--- a/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
+++ b/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
@@ -243,7 +243,7 @@ public class Parser {
* right side of a bin and subtracting half of the bin size. */
private long removeNoise(long reportedNumber, long binSize) {
long roundedToNearestRightSideOfTheBin =
-((reportedNumber + binSize / 2) / binSize) * binSize;
+Math.floorDiv((reportedNumber + binSize / 2), binSize) * binSize;
long subtractedHalfOfBinSize =
roundedToNearestRightSideOfTheBin - binSize / 2;
return subtractedHalfOfBinSize;
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Be clearer about Onionoo's exit_addresses field.
commit 18280f6b00e0d7726d42781fabda725c5b802059 Author: Karsten Loesing Date: Tue May 29 15:08:45 2018 +0200 Be clearer about Onionoo's exit_addresses field. Fixes #26031. --- src/main/resources/web/jsps/onionoo.jsp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index b225732..b58b6d9 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -1057,9 +1057,8 @@ IPv6 hex characters are all lower-case. optional # -Array of IPv4 or IPv6 addresses that the +Array of IPv4 addresses that the relay used to exit to the Internet in the past 24 hours. -IPv6 hex characters are all lower-case. Omitted if array is empty. Changed on April 17, 2018 to include all exit addresses, regardless of whether they are used as onion-routing addresses or not. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Prepare for 1.1.0 release.
commit 1284c8adf8158703f6abf2dccb6da72dc62267c7 Author: Karsten Loesing Date: Tue May 29 15:21:45 2018 +0200 Prepare for 1.1.0 release. --- CERT | 20 ++-- CHANGELOG.md | 5 - build.xml| 2 +- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/CERT b/CERT index ca92db0..61fa7c6 100644 --- a/CERT +++ b/CERT @@ -1,8 +1,8 @@ -BEGIN CERTIFICATE- -MIIDaTCCAlGgAwIBAgIEPrK+4DANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV +MIIDaTCCAlGgAwIBAgIENNGkczANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV UzELMAkGA1UECBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBU -b3IgUHJvamVjdCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwHhcNMTcx -MTE3MTA0MDQ5WhcNMTgwMjE1MTA0MDQ5WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE +b3IgUHJvamVjdCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwHhcNMTgw +NTIzMTUxNDU2WhcNMTgwODIxMTUxNDU2WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE CBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBUb3IgUHJvamVj dCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQChXn+IUp+o6G+k4ffxk3TkxZb3iXfiG7byNsG63olU @@ -11,11 +11,11 @@ Qw+VAhKTcEIv4yiR0BWapQyR07pgmKirYVjN6s6ef8NJzUptpxLlaYJ3ZfQfc4aE MXzScgaccwDFIWQ661lzLGCfeSxxa3Xy4wWsGwzNzLITYrrABcbg7yogLo2btNvD oEwGL3/baQdhl0dra6biVCZr9ydn3Hg57S55pUU0rBY25id78zUO8xrfNHw54wwX lOblGt75OOkahP/ZZSBxxoiknJ6y5VQV8y+noA4vigXFAgMBAAGjITAfMB0GA1Ud -DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAIqRy -T4uMqw1WwZ8CtJx5kSs/or+5025bGK6V2dY/jYapZcbyG7sq3KVFAqLik8Yn7a0D -dfbmtR7r2jbkWhYppa7MGSQ6SMG6nxas05l8yf9ShDnNLtJoay8JyGPeucaFyEwV -VxwN9/I4M4chrrvBzpMB/1U3oJxlKR+nssh4w55DoUijk3vHn01+20EPU9uNJrsr -mXhcrJLU/4R952ZMLxtiBfNP7vaxLpQctNqZxEfa9/rm0pLlfd2t62BWf8kkGqEy -Ke4WdNpQwk3WKjFlKWcvsF6ztJJlwA+qHk4IB9PynVRNggWimBrxWigbeBL8qpML -A5xMtX/MHDXreWGaKA== +DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAJtEQ +B2AVpVtjTGU7uujUtEjSn1ICv7QLW/37TmHn+m1SWvXOVlSG81eb5JwX449Pn3w5 +K1Bx0JJmJ6Kec2kHtoR1r5/DUlTMtRqKQ5ZHQbvYMx+Ifq8TRALHjcw8p5Vw+gep +0zYbSQycklzFaFqLB8Cus0ICb+UY54HTddpezQ3IXS/vc4Vy07ocm8rUCz1s8L/r +ehTOxSym7G7+kcRzNplFOLL5iO8o6uHPyLR1TIAIXa0XZ41ogNl2q+6CvXG+UI/j +qkzSXD6FJK1Px2nxNFIe/w9NL+chSytIGnV3CImyiORuV+1OxMglyQspSGEbl1XP +OfiTEYHnp12BYMeRyw== -END CERTIFICATE- diff --git a/CHANGELOG.md b/CHANGELOG.md index adb518d..a4fee6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,7 @@ -# Changes in version +# Changes in version 1.1.0 - 2018-05-29 + + * Medium changes + - Replace Gson with Jackson. * Minor changes - Avoid sending an error after a (partial) response. diff --git a/build.xml b/build.xml index 1675053..8baa764 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ - + https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Append _p to all plot_* and write_* parameters.
commit 7366398fdaf3caf6e02664d070e7c35dea29cc4b
Author: Karsten Loesing
Date: Thu May 17 11:37:36 2018 +0200
Append _p to all plot_* and write_* parameters.
This change makes our R code a little more readable in cases where
parameter names match data column names.
---
src/main/R/rserver/graphs.R| 685 +++--
.../torproject/metrics/web/RObjectGenerator.java | 6 +-
2 files changed, 351 insertions(+), 340 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index a9b7fc7..9a2b939 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -348,11 +348,11 @@ robust_call <- function(wrappee, filename) {
})
}
-prepare_networksize <- function(start, end) {
+prepare_networksize <- function(start_p, end_p) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
-filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(if (!is.null(start_p)) date >= as.Date(start_p) else TRUE) %>%
+filter(if (!is.null(end_p)) date <= as.Date(end_p) else TRUE) %>%
filter(flag == "") %>%
filter(country == "") %>%
filter(version == "") %>%
@@ -361,8 +361,8 @@ prepare_networksize <- function(start, end) {
select(date, relays, bridges)
}
-plot_networksize <- function(start, end, path) {
- prepare_networksize(start, end) %>%
+plot_networksize <- function(start_p, end_p, path_p) {
+ prepare_networksize(start_p, end_p) %>%
gather(variable, value, -date) %>%
complete(date = full_seq(date, period = 1),
variable = c("relays", "bridges")) %>%
@@ -375,19 +375,19 @@ plot_networksize <- function(start, end, path) {
labels = c("Relays", "Bridges")) +
ggtitle("Number of relays") +
labs(caption = copyright_notice)
- ggsave(filename = path, width = 8, height = 5, dpi = 150)
+ ggsave(filename = path_p, width = 8, height = 5, dpi = 150)
}
-write_networksize <- function(start = NULL, end = NULL, path) {
- prepare_networksize(start, end) %>%
-write.csv(path, quote = FALSE, row.names = FALSE, na = "")
+write_networksize <- function(start_p = NULL, end_p = NULL, path_p) {
+ prepare_networksize(start_p, end_p) %>%
+write.csv(path_p, quote = FALSE, row.names = FALSE, na = "")
}
-prepare_versions <- function(start, end) {
+prepare_versions <- function(start_p, end_p) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
-filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(if (!is.null(start_p)) date >= as.Date(start_p) else TRUE) %>%
+filter(if (!is.null(end_p)) date <= as.Date(end_p) else TRUE) %>%
filter(flag == "") %>%
filter(country == "") %>%
filter(version != "") %>%
@@ -396,8 +396,8 @@ prepare_versions <- function(start, end) {
select(date, version, relays)
}
-plot_versions <- function(start, end, path) {
- s <- prepare_versions(start, end)
+plot_versions <- function(start_p, end_p, path_p) {
+ s <- prepare_versions(start_p, end_p)
known_versions <- c("Other", "0.1.0", "0.1.1", "0.1.2", "0.2.0",
"0.2.1", "0.2.2", "0.2.3", "0.2.4", "0.2.5", "0.2.6", "0.2.7",
"0.2.8", "0.2.9", "0.3.0", "0.3.1", "0.3.2", "0.3.3", "0.3.4")
@@ -418,20 +418,20 @@ plot_versions <- function(start, end, path) {
breaks = visible_versions) +
ggtitle("Relay versions") +
labs(caption = copyright_notice)
- ggsave(filename = path, width = 8, height = 5, dpi = 150)
+ ggsave(filename = path_p, width = 8, height = 5, dpi = 150)
}
-write_versions <- function(start = NULL, end = NULL, path) {
- prepare_versions(start, end) %>%
+write_versions <- function(start_p = NULL, end_p = NULL, path_p) {
+ prepare_versions(start_p, end_p) %>%
spread(key = "version", value = "relays", fill = 0) %>%
-write.csv(path, quote = FALSE, row.names = FALSE, na = "")
+write.csv(path_p, quote = FALSE, row.names = FALSE, na = "")
}
-prepare_platforms <- function(start, end) {
+prepare_platforms <- function(start_p, end_p) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(if (!is.null(start)) date >= as.Date(start) else TRUE) %>%
-filter(if (!is.null(end)) date <= as.Date(end) else TRUE) %>%
+filter(if (!is.null(start_p)) date >= as.Date(start_p) else TRUE) %>%
+filter(if (!is.null(end_p)) date <= as.Date(end_p) else TRUE) %>%
filter(flag == "") %>%
filter(country == "") %>%
filter(version == "") %>%
@@ -442,8 +442,8 @@ prepare_platforms <- function(start, end) {
as.character(platform)))
}
-plot_platforms <- function(start, end, path) {
- prepare_platforms(start, end) %>%
+plot_platforms <- function(start_p, end_p, path_p) {
+
[tor-commits] [metrics-web/release] Update to latest rs (Relay Search).
commit 168f585198cc40fecbf17e5d7a7085c212a191d3 Author: Karsten Loesing Date: Wed May 16 09:06:22 2018 +0200 Update to latest rs (Relay Search). --- src/main/resources/web/rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/rs b/src/main/resources/web/rs index b6e6c32..887512f 16 --- a/src/main/resources/web/rs +++ b/src/main/resources/web/rs @@ -1 +1 @@ -Subproject commit b6e6c3235806c9d1521dbc2c08cecee626466928 +Subproject commit 887512f3a111981e1604e9cd6c46b2d32300db14 ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Put back updated OOI link.
commit ca6353e188e277a4461354e7e3aad6c63cd4212b
Author: Karsten Loesing
Date: Sat May 26 21:35:07 2018 +0200
Put back updated OOI link.
Resolves #25926.
---
src/main/resources/web/json/metrics.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/resources/web/json/metrics.json
b/src/main/resources/web/json/metrics.json
index b867f50..8dbdc66 100644
--- a/src/main/resources/web/json/metrics.json
+++ b/src/main/resources/web/json/metrics.json
@@ -279,7 +279,7 @@
"id": "oxford-anonymous-internet",
"title": "The anonymous Internet",
"type": "Link",
-"description": "The Oxford Internet Institute made a cartogram
visualization of Tor users as compared to the overall Internet population.
They used the average number of Tor users
per country from August 2012 to August 2013 and put it in relation to total
Internet users per country. "
+"description": "The Oxford Internet Institute made a cartogram
visualization of Tor users as compared to the overall Internet population.
They used the average number of Tor users
per country from August 2012 to August 2013 and put it in relation to total
Internet users per country. More details and conclusions can be found on the
http://geography.oii.ox.ac.uk/the-anonymous-internet/\;
target=\"_blank\">Information Geographies website at the Oxford Internet
Institute.http://geography.oii.ox.ac.uk/the-anonymous-internet/\;>"
},
{
"id": "torperf",
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Fix TB downloads by locale and Relays by flag graphs.
commit 4cb0fd677271a1faa3a7f99e5e7868d5cf50a7a8
Author: Karsten Loesing
Date: Wed May 9 15:13:55 2018 +0200
Fix TB downloads by locale and Relays by flag graphs.
---
src/main/R/rserver/graphs.R | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index b6eb221..79d5600 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -580,8 +580,8 @@ plot_relayflags <- function(start, end, flags, path) {
write_relayflags <- function(start, end, flags, path) {
prepare_relayflags(start, end, flags) %>%
-mutate(variable = tolower(variable)) %>%
-spread(variable, value) %>%
+mutate(flag = tolower(flag)) %>%
+spread(flag, relays) %>%
write.csv(path, quote = FALSE, row.names = FALSE)
}
@@ -1269,7 +1269,7 @@ write_webstats_tb_platform <- function(start, end, path) {
plot_webstats_tb_locale <- function(start, end, path) {
d <- read.csv(paste(stats_dir, "webstats.csv", sep = ""),
-colClasses = c("log_date" = "Date"))
+colClasses = c("log_date" = "Date", "locale" = "character"))
d <- d[d$log_date >= start & d$log_date <= end & d$request_type == "tbid", ]
e <- d
e <- aggregate(list(count = e$count), by = list(locale = e$locale), FUN =
sum)
@@ -1300,7 +1300,7 @@ plot_webstats_tb_locale <- function(start, end, path) {
# breaks and labels. Left as future work.
write_webstats_tb_locale <- function(start, end, path) {
d <- read.csv(paste(stats_dir, "webstats.csv", sep = ""),
-colClasses = c("log_date" = "Date"))
+colClasses = c("log_date" = "Date", "locale" = "character"))
d <- d[d$log_date >= start & d$log_date <= end & d$request_type == "tbid", ]
e <- d
e <- aggregate(list(count = e$count), by = list(locale = e$locale), FUN =
sum)
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Simplify censorship detector.
commit 27fe3f3b504ec59af6e4ea1bab9f991ef84be908 Author: Karsten Loesing Date: Tue May 15 20:25:39 2018 +0200 Simplify censorship detector. As part of the censorship detector, we're currently fitting a normal distribution to a set of approximately 50 values. After talking to George Danezis, the original author of our censorship detector, it's safe to approximate this step by computing the mean and standard deviation of these values and using those to parameterize the normal distribution. A test run with today's numbers produced the exact same result. The main benefit of this simplification is that it will be easier for others to reproduce our numbers, even without a statistics library as powerful as SciPy. --- src/main/python/clients/detector.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/main/python/clients/detector.py b/src/main/python/clients/detector.py index 3d17bf0..9944710 100644 --- a/src/main/python/clients/detector.py +++ b/src/main/python/clients/detector.py @@ -43,6 +43,7 @@ import matplotlib # Dep: numpy import numpy +from numpy import mean, std # Dep: scipy import scipy.stats @@ -199,7 +200,8 @@ def make_tendencies_minmax(l, INTERVAL = 1): minx += [None] maxx += [None] continue - mu, signma = norm.fit(vals) + mu = mean(vals) + signma = std(vals) dists += [(mu, signma)] maxx += [norm.ppf(0., mu, signma)] minx += [norm.ppf(1 - 0., mu, signma)] ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 260 of doc/MetricsTimeline.
commit 3ad2bab15c726fa4f9b1af65b59c0cd389b709e5
Author: Karsten Loesing
Date: Tue May 15 14:49:20 2018 +0200
Update news.json to version 260 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 4
1 file changed, 4 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 8d2fbd7..d333e2a 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4640,6 +4640,10 @@
"description": "Iranian ISPs block Telegram.",
"links": [
{
+"label": "ticket",
+"target": "https://bugs.torproject.org/26087;
+ },
+ {
"label": "article",
"target":
"https://arstechnica.com/tech-policy/2018/05/iran-blocks-telegram-pushes-replacement-with-death-to-america-emoji/;
},
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update to latest metrics-lib.
commit 1072602b66074e8b86f6a5880005eede67cdbdb0 Author: Karsten Loesing Date: Sat May 26 21:43:34 2018 +0200 Update to latest metrics-lib. --- src/submods/metrics-lib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index 77bbd74..feabf88 16 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit 77bbd742b63fd9708694d7d8623d82a9654b770e +Subproject commit feabf88fff34465134da9ea61592f7f9c30415b7 ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Simplify dirread and dirwrite in bandwidth.csv.
commit 6b53cd0e4619fa18fae8630de91c23e7ba7bbdaa Author: Karsten Loesing Date: Thu May 17 15:11:40 2018 +0200 Simplify dirread and dirwrite in bandwidth.csv. With this change we're making it clear that these two columns contain statistics from directory mirrors only, and we're taking out the extrapolation step. Implements #26002. --- src/main/resources/web/json/metrics.json | 2 +- src/main/resources/web/jsps/stats.jsp| 9 - src/main/sql/legacy/tordir.sql | 6 ++ 3 files changed, 7 insertions(+), 10 deletions(-) diff --git a/src/main/resources/web/json/metrics.json b/src/main/resources/web/json/metrics.json index 57b7c9c..b867f50 100644 --- a/src/main/resources/web/json/metrics.json +++ b/src/main/resources/web/json/metrics.json @@ -103,7 +103,7 @@ "id": "dirbytes", "title": "Bandwidth spent on answering directory requests", "type": "Graph", -"description": "This graph shows the portion of consumed bandwidth that directory authorities and mirrors have spent on answering directory requests. Not all directories report these statistics, so the graph shows an estimation of total consumed bandwidth as it would be observed if all directories reported these statistics.", +"description": "This graph shows the portion of consumed bandwidth that directory mirrors have spent on answering directory requests.", "function": "dirbytes", "parameters": [ "start", diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index 2674ba0..ea5faf8 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -125,15 +125,14 @@ This metric includes any kind of traffic. bwwrite: Similar to bwread, but for traffic written by relays. -dirread: Bandwidth in bytes per second that relays have read when -serving directory data. Not all relays report how many bytes they read when -serving directory data which is why this value is an estimate from the available -data. This metric is not available for subsets of relays with certain relay +dirread: Bandwidth in bytes per second that directory mirrors have +read when serving directory data. +This metric is not available for subsets of relays with certain relay flags, so that this column will contain the empty string if either isexit or isguard is non-empty. dirwrite: Similar to dirread, but for traffic written by -relays when serving directory data. +directory mirrors when serving directory data. diff --git a/src/main/sql/legacy/tordir.sql b/src/main/sql/legacy/tordir.sql index 5d254f6..bd91ab6 100644 --- a/src/main/sql/legacy/tordir.sql +++ b/src/main/sql/legacy/tordir.sql @@ -807,11 +807,9 @@ UNION ALL CASE WHEN total_bwhist.written IS NOT NULL THEN total_bwhist.written / 86400 END AS bwwrite, CASE WHEN u.date IS NOT NULL - THEN FLOOR(CAST(u.dr AS NUMERIC) * CAST(u.brp AS NUMERIC) / - CAST(u.brd AS NUMERIC) / CAST(86400 AS NUMERIC)) END AS dirread, + THEN FLOOR(CAST(u.dr AS NUMERIC) / CAST(86400 AS NUMERIC)) END AS dirread, CASE WHEN u.date IS NOT NULL - THEN FLOOR(CAST(u.dw AS NUMERIC) * CAST(u.bwp AS NUMERIC) / - CAST(u.bwd AS NUMERIC) / CAST(86400 AS NUMERIC)) END AS dirwrite + THEN FLOOR(CAST(u.dw AS NUMERIC) / CAST(86400 AS NUMERIC)) END AS dirwrite FROM total_bandwidth FULL OUTER JOIN total_bwhist ON total_bandwidth.date = total_bwhist.date FULL OUTER JOIN (SELECT * FROM user_stats WHERE country = 'zy' ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Say that the Tor Messenger graph will soon disappear.
commit 0060ed5155f015d481f719c59aadb22114cd4dbf Author: Karsten Loesing Date: Fri May 11 17:37:33 2018 +0200 Say that the Tor Messenger graph will soon disappear. --- src/main/resources/web/json/metrics.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/json/metrics.json b/src/main/resources/web/json/metrics.json index 7f04cb6..57b7c9c 100644 --- a/src/main/resources/web/json/metrics.json +++ b/src/main/resources/web/json/metrics.json @@ -408,7 +408,7 @@ "id": "webstats-tm", "title": "Tor Messenger downloads and updates", "type": "Graph", -"description": "This graph shows absolute numbers of requests to Tor's web servers related to Tor Messenger. Initial downloads are requests made by the user to download a Tor Messenger executable from the Tor website. Update pings are requests made by Tor Messenger to check whether a newer version is available.", +"description": "This graph shows absolute numbers of requests to Tor's web servers related to Tor Messenger. Initial downloads are requests made by the user to download a Tor Messenger executable from the Tor website. Update pings are requests made by Tor Messenger to check whether a newer version is available.Note that https://blog.torproject.org/sunsetting-tor-messenger\;>Tor Messenger has been sunset on April 2, 2018. This graph is soon going to disappear, too.", "function": "webstats_tm", "parameters": [ "start", ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Remove inconsistency between bandwidth history graphs.
commit 0eacbbecefecde7346b64629f3ade1403785213b
Author: Karsten Loesing
Date: Wed May 16 21:29:34 2018 +0200
Remove inconsistency between bandwidth history graphs.
The number of bytes in all bandwidth histories differed from the sum
of those numbers by Exit/Guard flag. The difference came from relays
that reported bandwidth histories but that the directory authorities
did not list as running.
With this commit we stop producing bandwidth numbers that are not
broken down by Exit/Guard flag, and we sum up those columns for the
bandwidth graph.
Implements #26015.
---
src/main/R/rserver/graphs.R| 9 ++--
src/main/sql/legacy/tordir.sql | 112 +++--
2 files changed, 13 insertions(+), 108 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 79d5600..2ac2756 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -451,10 +451,11 @@ write_platforms <- function(start, end, path) {
prepare_bandwidth <- function(start, end) {
read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), isexit == "",
- isguard == "") %>%
-mutate(advbw = advbw * 8 / 1e9) %>%
-mutate(bwhist = (bwread + bwwrite) * 8 / 2e9) %>%
+filter(date >= as.Date(start), date <= as.Date(end), isexit != "",
+ isguard != "") %>%
+group_by(date) %>%
+summarize(advbw = sum(advbw) * 8 / 1e9,
+ bwhist = sum(bwread + bwwrite) * 8 / 2e9) %>%
select(date, advbw, bwhist)
}
diff --git a/src/main/sql/legacy/tordir.sql b/src/main/sql/legacy/tordir.sql
index bd91ab6..16e7166 100644
--- a/src/main/sql/legacy/tordir.sql
+++ b/src/main/sql/legacy/tordir.sql
@@ -151,29 +151,6 @@ CREATE TABLE relay_versions (
CONSTRAINT relay_versions_pkey PRIMARY KEY(date, version)
);
--- TABLE total_bandwidth
--- Contains information for the whole network's total bandwidth which is
--- used in the bandwidth graphs.
-CREATE TABLE total_bandwidth (
-date DATE NOT NULL,
-bwavg BIGINT NOT NULL,
-bwburst BIGINT NOT NULL,
-bwobserved BIGINT NOT NULL,
-bwadvertised BIGINT NOT NULL,
-CONSTRAINT total_bandwidth_pkey PRIMARY KEY(date)
-);
-
--- TABLE total_bwhist
--- Contains the total number of read/written and the number of dir bytes
--- read/written by all relays in the network on a given day. The dir bytes
--- are an estimate based on the subset of relays that count dir bytes.
-CREATE TABLE total_bwhist (
-date DATE NOT NULL,
-read BIGINT,
-written BIGINT,
-CONSTRAINT total_bwhist_pkey PRIMARY KEY(date)
-);
-
-- TABLE bandwidth_flags
CREATE TABLE bandwidth_flags (
date DATE NOT NULL,
@@ -473,64 +450,6 @@ CREATE OR REPLACE FUNCTION refresh_relay_versions()
RETURNS INTEGER AS $$
END;
$$ LANGUAGE plpgsql;
--- FUNCTION refresh_total_bandwidth()
--- This keeps the table total_bandwidth up-to-date when necessary.
-CREATE OR REPLACE FUNCTION refresh_total_bandwidth() RETURNS INTEGER AS $$
-DECLARE
-min_date TIMESTAMP WITHOUT TIME ZONE;
-max_date TIMESTAMP WITHOUT TIME ZONE;
-BEGIN
-
-min_date := (SELECT MIN(date) FROM updates);
-max_date := (SELECT MAX(date) + 1 FROM updates);
-
-DELETE FROM total_bandwidth
-WHERE date IN (SELECT date FROM updates);
-
-EXECUTE '
-INSERT INTO total_bandwidth
-(bwavg, bwburst, bwobserved, bwadvertised, date)
-SELECT (SUM(bandwidthavg)
-/ relay_statuses_per_day.count)::BIGINT AS bwavg,
-(SUM(bandwidthburst)
-/ relay_statuses_per_day.count)::BIGINT AS bwburst,
-(SUM(bandwidthobserved)
-/ relay_statuses_per_day.count)::BIGINT AS bwobserved,
-(SUM(LEAST(bandwidthavg, bandwidthobserved))
-/ relay_statuses_per_day.count)::BIGINT AS bwadvertised,
-DATE(validafter)
-FROM descriptor RIGHT JOIN statusentry
-ON descriptor.descriptor = statusentry.descriptor
-JOIN relay_statuses_per_day
-ON DATE(validafter) = relay_statuses_per_day.date
-WHERE isrunning = TRUE
- AND validafter >= ''' || min_date || '''
- AND validafter < ''' || max_date || '''
- AND DATE(validafter) IN (SELECT date FROM updates)
- AND relay_statuses_per_day.date >= ''' || min_date || '''
- AND relay_statuses_per_day.date < ''' || max_date || '''
- AND DATE(relay_statuses_per_day.date) IN
- (SELECT date FROM updates)
-GROUP BY DATE(validafter), relay_statuses_per_day.count';
-
-RETURN 1;
-END;
-$$ LANGUAGE plpgsql;
-
-CREATE OR REPLACE FUNCTION refresh_total_bwhist() RETURNS INTEGER AS $$
- BEGIN
- DELETE FROM total_bwhist WHERE date IN (SELECT date FROM updates);
- INSERT INTO total_bwhist (date, read, written)
- SELECT date, SUM(read_sum) AS read, SUM(written_sum) AS written
- FROM bwhist
- WHERE date >= (SELECT
[tor-commits] [metrics-web/release] Update news.json to version 262 of doc/MetricsTimeline.
commit 8318bfcd1eb9bf5ee357c46fbd580c62d69707b7
Author: Karsten Loesing
Date: Wed May 16 09:06:06 2018 +0200
Update news.json to version 262 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 26 ++
1 file changed, 26 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index d333e2a..554f094 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4658,6 +4658,14 @@
]
},
{
+"start": "2018-05-08",
+"protocols": [
+ "meek"
+],
+"shortDescription": "The meek-amazon bridge imposes a rate limit of 1
MB/s.",
+"description": "The https://metrics.torproject.org/rs.html#details/F4AD82B2032EDEF6C02C5A529C42CFAFE516564D\;>meek-amazon
bridge imposes a rate limit of 1 MB/s."
+ },
+ {
"start": "2018-05-09",
"protocols": [
"snowflake"
@@ -4676,6 +4684,24 @@
]
},
{
+"start": "2018-05-14",
+"protocols": [
+ "meek"
+],
+"shortDescription": "Disabled the d2cly7j4zqgua7.cloudfront.net CDN
endpoint for meek-amazon.",
+"description": "Disabled the d2cly7j4zqgua7.cloudfront.net CDN endpoint
for meek-amazon.",
+"links": [
+ {
+"label": "Tor Browser ticket",
+"target": "https://bugs.torproject.org/26098;
+ },
+ {
+"label": "Orbot ticket",
+"target": "https://bugs.torproject.org/26111;
+ }
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Don't terminate the hidserv module when parsing descriptors.
commit 4333d0666dbb8a252bd85d1cc79ebd18dc84d4bd
Author: Karsten Loesing
Date: Fri May 18 16:26:12 2018 +0200
Don't terminate the hidserv module when parsing descriptors.
This issue came up when reprocessing descriptor archives for #26022:
we'd terminate the hidserv module when running into an issue with
parsing descriptors, where a possible issue is a consensus without a
"bandwidth-weights" line.
This seems wrong. It's okay to print out a warning in this case. But
in general, we don't want to abort the entire module in this case.
Related to #26022.
---
.../org/torproject/metrics/stats/hidserv/Main.java | 6 +-
.../torproject/metrics/stats/hidserv/Parser.java | 23 +-
2 files changed, 10 insertions(+), 19 deletions(-)
diff --git a/src/main/java/org/torproject/metrics/stats/hidserv/Main.java
b/src/main/java/org/torproject/metrics/stats/hidserv/Main.java
index a23c17f..eafeeb1 100644
--- a/src/main/java/org/torproject/metrics/stats/hidserv/Main.java
+++ b/src/main/java/org/torproject/metrics/stats/hidserv/Main.java
@@ -38,11 +38,7 @@ public class Main {
/* Parse new descriptors and store their contents using the document
* stores. */
System.out.println("Parsing descriptors...");
-if (!parser.parseDescriptors()) {
- System.err.println("Could not store parsed descriptors. "
- + "Terminating.");
- return;
-}
+parser.parseDescriptors();
/* Write the parse history to avoid parsing descriptor files again
* next time. It's okay to do this now and not at the end of the
diff --git a/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
b/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
index f2abc78..2423526 100644
--- a/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
+++ b/src/main/java/org/torproject/metrics/stats/hidserv/Parser.java
@@ -162,16 +162,14 @@ public class Parser {
/** Instructs the descriptor reader to parse descriptor files, and
* handles the resulting parsed descriptors if they are either
* extra-info descriptors or consensuses. */
- public boolean parseDescriptors() {
+ public void parseDescriptors() {
for (Descriptor descriptor : descriptorReader.readDescriptors(
this.inDirectories)) {
if (descriptor instanceof ExtraInfoDescriptor) {
this.parseExtraInfoDescriptor((ExtraInfoDescriptor) descriptor);
} else if (descriptor instanceof RelayNetworkStatusConsensus) {
-if (!this.parseRelayNetworkStatusConsensus(
-(RelayNetworkStatusConsensus) descriptor)) {
- return false;
-}
+this.parseRelayNetworkStatusConsensus(
+(RelayNetworkStatusConsensus) descriptor);
}
}
@@ -180,7 +178,7 @@ public class Parser {
* descriptors. In contrast, sets of computed network fractions are
* stored immediately after processing the consensus they are based
* on. */
-return this.reportedHidServStatsStore.store(
+this.reportedHidServStatsStore.store(
this.reportedHidServStatsFile, this.reportedHidServStats);
}
@@ -252,7 +250,7 @@ public class Parser {
}
/** Parses the given consensus. */
- public boolean parseRelayNetworkStatusConsensus(
+ public void parseRelayNetworkStatusConsensus(
RelayNetworkStatusConsensus consensus) {
/* Make sure that the consensus contains Wxx weights. */
@@ -262,7 +260,7 @@ public class Parser {
System.err.printf("Consensus with valid-after time %s doesn't "
+ "contain any Wxx weights. Skipping.%n",
DateTimeHelper.format(consensus.getValidAfterMillis()));
- return false;
+ return;
}
/* More precisely, make sure that it contains Wmx weights, and then
@@ -274,7 +272,7 @@ public class Parser {
System.err.printf("Consensus with valid-after time %s doesn't "
+ "contain expected Wmx weights. Skipping.%n",
DateTimeHelper.format(consensus.getValidAfterMillis()));
- return false;
+ return;
}
double wmg = ((double) bandwidthWeights.get("Wmg")) / 1.0;
double wmm = ((double) bandwidthWeights.get("Wmm")) / 1.0;
@@ -421,11 +419,8 @@ public class Parser {
DateTimeHelper.ISO_DATE_FORMAT);
File documentFile = new File(this.computedNetworkFractionsDirectory,
date);
-if (!this.computedNetworkFractionsStore.store(documentFile,
-computedNetworkFractions)) {
- return false;
-}
-return true;
+this.computedNetworkFractionsStore.store(documentFile,
+computedNetworkFractions);
}
}
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Fix torperf graph for single source.
commit de1799779b3eb1cf0f94ff3c2ff5fb5e9df6daab
Author: Karsten Loesing
Date: Mon May 7 10:57:24 2018 +0200
Fix torperf graph for single source.
---
src/main/R/rserver/graphs.R | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 332204b..b6eb221 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -589,7 +589,7 @@ plot_torperf <- function(start, end, source, server,
filesize, path) {
filesize_val <- ifelse(filesize == "50kb", 50 * 1024,
ifelse(filesize == "1mb", 1024 * 1024, 5 * 1024 * 1024))
t <- read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""),
-colClasses = c("date" = "Date"))
+colClasses = c("date" = "Date", "source" = "character"))
known_sources <- c("all", unique(t[t$source != "", "source"]))
colours <- data.frame(source = known_sources,
colour = brewer.pal(length(known_sources), "Paired"),
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 259 of doc/MetricsTimeline.
commit 2537885cfc34a8fa25d4d0e7d1c41db735d12fb7
Author: Karsten Loesing
Date: Sat May 12 09:30:52 2018 +0200
Update news.json to version 259 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 41 +++
1 file changed, 41 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 77d94aa..8d2fbd7 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4631,6 +4631,47 @@
]
},
{
+"start": "2018-04-30",
+"ongoing": true,
+"places": [
+ "ir"
+],
+"shortDescription": "Iranian ISPs block Telegram.",
+"description": "Iranian ISPs block Telegram.",
+"links": [
+ {
+"label": "article",
+"target":
"https://arstechnica.com/tech-policy/2018/05/iran-blocks-telegram-pushes-replacement-with-death-to-america-emoji/;
+ },
+ {
+"label": "relay graph",
+"target":
"https://metrics.torproject.org/userstats-relay-country.html?start=2018-03-01=2018-06-01=ir=off;
+ },
+ {
+"label": "bridge graph",
+"target":
"https://metrics.torproject.org/userstats-bridge-country.html?start=2018-03-11=2018-06-01=ir;
+ }
+]
+ },
+ {
+"start": "2018-05-09",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Release of Tor Browser 8.0a7, which changes the
default Snowflake rendezvous to use an Azure domain front, instead of the
Google one that had been non-functional since 2018-04-13.",
+"description": "Release of Tor Browser 8.0a7, which changes the default
Snowflake rendezvous to use an Azure domain front, instead of the Google one
that had been non-functional since 2018-04-13.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-80a7-released;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/26010;
+ }
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 255 of doc/MetricsTimeline.
commit d425de8f8a22e256398977e812fa69550506eb57
Author: Karsten Loesing
Date: Sun Apr 15 22:41:42 2018 +0200
Update news.json to version 255 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 89 +--
1 file changed, 86 insertions(+), 3 deletions(-)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index c503aa1..8385b6e 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -429,7 +429,7 @@
},
{
"start": "2012-02-08",
-"ongoing": true,
+"end": "2012-02-11",
"places": [
"ir"
],
@@ -442,6 +442,14 @@
{
"label": "blog post",
"target":
"https://blog.torproject.org/iran-partially-blocks-encrypted-network-traffic;
+ },
+ {
+"label": "relay graph",
+"target":
"https://metrics.torproject.org/userstats-relay-country.html?start=2012-01-01=2012-04-01=ir;
+ },
+ {
+"label": "bridge graph",
+"target":
"https://metrics.torproject.org/userstats-bridge-country.html?start=2012-01-01=2012-04-01=ir;
}
]
},
@@ -4489,6 +4497,10 @@
{
"label": "comment",
"target": "https://bugs.torproject.org/21312#comment:63;
+ },
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/25688#comment:1;
}
]
},
@@ -4497,8 +4509,14 @@
"protocols": [
"snowflake"
],
-"shortDescription": "Provisonally increased memory limits of the fallback
snowflake proxies to 400 MB, 800 MB, or unlimited.",
-"description": "Provisonally increased memory limits of the fallback
snowflake proxies to 400 MB, 800 MB, or unlimited."
+"shortDescription": "Provisionally increased memory limits of the fallback
snowflake proxies to 400 MB, 800 MB, or unlimited.",
+"description": "Provisionally increased memory limits of the fallback
snowflake proxies to 400 MB, 800 MB, or unlimited.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/25688#comment:1;
+ }
+]
},
{
"start": "2018-03-23",
@@ -4511,6 +4529,10 @@
{
"label": "comment",
"target": "https://bugs.torproject.org/21312#comment:64;
+ },
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/25688#comment:1;
}
]
},
@@ -4533,6 +4555,48 @@
]
},
{
+"start": "2018-03-27",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Restarted the fallback snowflake proxies with no
limit on memory.",
+"description": "Restarted the fallback snowflake proxies with no limit on
memory.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/25688#comment:1;
+ }
+]
+ },
+ {
+"start": "2018-04-02",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Reinstated periodic restarts of the fallback
snowflake proxies.",
+"description": "Reinstated periodic restarts of the fallback snowflake
proxies.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/25688#comment:1;
+ }
+]
+ },
+ {
+"start": "2018-04-13",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Snowflake client registrations",
+"description": "Snowflake client registrations (based on domain fronting)
stop working, because of a Google infrastructure change that stops domain
fronting from working. The time of the change is between 2018-04-13 14:00:00
and 2018-04-13 18:00:00, based on when the bandwidth graph of the Snowflake
bridge https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F\;>5481936581E23D2D178105D44DB6915AB06BFB7F
went to zero.",
+"links": [
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/25804;
+ }
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
@@ -5452,6 +5516,25 @@
"unknown": true
},
{
+"start": "2018-02-01",
+"end": "2018-02-05",
+"places": [
+ "ir"
+],
+"protocols": [
+ "meek"
+],
+"shortDescription": "Drop of meek users in Iran; other transports not
affected.",
+"description": "Drop of meek users in Iran; other transports not
affected.",
+"links": [
+ {
+"label": "graph",
+"target":
"https://metrics.torproject.org/userstats-bridge-combined.html?start=2018-01-14=2018-03-15=ir;
+ }
+],
+"unknown": true
+ },
+ {
"start": "2018-03-05",
"end": "2018-03-08",
"places": [
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Comment out OOI link.
commit 1ac237e113434ca5beb4d0dc88a496a3f0a1c8bf
Author: Karsten Loesing
Date: Thu Apr 26 14:39:51 2018 +0200
Comment out OOI link.
This is a temporary measure to address #25926.
---
src/main/resources/web/json/metrics.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/resources/web/json/metrics.json
b/src/main/resources/web/json/metrics.json
index 0739b88..7f04cb6 100644
--- a/src/main/resources/web/json/metrics.json
+++ b/src/main/resources/web/json/metrics.json
@@ -279,7 +279,7 @@
"id": "oxford-anonymous-internet",
"title": "The anonymous Internet",
"type": "Link",
-"description": "The Oxford Internet Institute made a cartogram
visualization of Tor users as compared to the overall Internet population.
They used the average number of Tor users
per country from August 2012 to August 2013 and put it in relation to total
Internet users per country. More details and conclusions can be found on the
http://geography.oii.ox.ac.uk/?page=tor\;
target=\"_blank\">Information Geographies website at the Oxford Internet
Institute.http://geography.oii.ox.ac.uk/?page=tor\;>"
+"description": "The Oxford Internet Institute made a cartogram
visualization of Tor users as compared to the overall Internet population.
They used the average number of Tor users
per country from August 2012 to August 2013 and put it in relation to total
Internet users per country. "
},
{
"id": "torperf",
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Provide no-data pictures, pdfs, and csv files instead of server error.
commit 533225c4250490857b0ed3e31232914da725cc13
Author: iwakeh
Date: Sat Mar 24 09:31:08 2018 +
Provide no-data pictures, pdfs, and csv files instead of server error.
Implements task-25264.
---
build.xml | 1 +
src/main/R/rserver/graphs.R | 20
.../torproject/metrics/web/RObjectGenerator.java| 7 ---
src/main/resources/web/images/no-data-available.pdf | Bin 0 -> 33824 bytes
src/main/resources/web/images/no-data-available.png | Bin 0 -> 26209 bytes
src/main/resources/web/images/no-data-available.xcf | Bin 0 -> 104450 bytes
6 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/build.xml b/build.xml
index 3b4222e..17160ee 100644
--- a/build.xml
+++ b/build.xml
@@ -504,6 +504,7 @@
+
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 57d2be2..332204b 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -328,6 +328,26 @@ stats_dir <-
"/srv/metrics.torproject.org/metrics/shared/stats/"
rdata_dir <- "/srv/metrics.torproject.org/metrics/shared/RData/"
+# Helper function that copies the appropriate no data object to filename.
+copy_no_data <- function(filename) {
+ len <- nchar(filename)
+ extension <- substr(filename, len - 3, len)
+ if (".csv" == extension) {
+write("# No data available for the given parameters.", file=filename)
+ } else {
+file.copy(paste(rdata_dir, "no-data-available", extension, sep = ""),
+ filename)
+ }
+}
+
+# Helper function wrapping calls into error handling.
+robust_call <- function(wrappee, filename) {
+ tryCatch(eval(wrappee), error = function(e) copy_no_data(filename),
+ finally = if (!file.exists(filename) || file.size(filename) == 0) {
+ copy_no_data(filename)
+ })
+}
+
prepare_networksize <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
diff --git a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
index 04bda0c..e3a2419 100644
--- a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
+++ b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
@@ -117,19 +117,20 @@ public class RObjectGenerator implements
ServletContextListener {
== null) {
return null;
}
-String function = this.availableGraphs.get(requestedGraph)
-.getFunction();
Map checkedParameters = GraphParameterChecker
.getInstance().checkParameters(requestedGraph, parameterMap);
if (checkedParameters == null) {
return null;
}
StringBuilder queryBuilder = new StringBuilder();
+queryBuilder.append("robust_call(as.call(list(");
if ("csv".equalsIgnoreCase(fileType)) {
queryBuilder.append("write_");
} else {
queryBuilder.append("plot_");
}
+String function = this.availableGraphs.get(requestedGraph)
+.getFunction();
queryBuilder.append(function).append("(");
StringBuilder imageFilenameBuilder =
new StringBuilder(requestedGraph);
@@ -154,7 +155,7 @@ public class RObjectGenerator implements
ServletContextListener {
}
imageFilenameBuilder.append(".").append(fileType);
String imageFilename = imageFilenameBuilder.toString();
-queryBuilder.append("path = '%s')");
+queryBuilder.append("path = '%1$s'))), '%1$s')");
String query = queryBuilder.toString();
File imageFile = new File(this.cachedGraphsDirectory + "/"
+ imageFilename);
diff --git a/src/main/resources/web/images/no-data-available.pdf
b/src/main/resources/web/images/no-data-available.pdf
new file mode 100644
index 000..8c9ca25
Binary files /dev/null and
b/src/main/resources/web/images/no-data-available.pdf differ
diff --git a/src/main/resources/web/images/no-data-available.png
b/src/main/resources/web/images/no-data-available.png
new file mode 100644
index 000..36310ad
Binary files /dev/null and
b/src/main/resources/web/images/no-data-available.png differ
diff --git a/src/main/resources/web/images/no-data-available.xcf
b/src/main/resources/web/images/no-data-available.xcf
new file mode 100644
index 000..42a7170
Binary files /dev/null and
b/src/main/resources/web/images/no-data-available.xcf differ
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Remove the PostgreSQL jar from our war file.
commit 5c8eec9d0ea3b3b39dfe99400622d8ad769c5559 Author: Karsten Loesing Date: Wed May 16 14:41:56 2018 +0200 Remove the PostgreSQL jar from our war file. Another part of #25699. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index bdb908b..81fa363 100644 --- a/build.xml +++ b/build.xml @@ -47,7 +47,6 @@ - @@ -57,6 +56,7 @@ + ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Reflect that webstats.tp.o is gone.
commit af48f3aeaf632089aeb50d0440cd1455e8aed0a3 Author: Karsten Loesing Date: Mon May 7 10:10:50 2018 +0200 Reflect that webstats.tp.o is gone. --- src/main/resources/web/json/categories.json | 2 +- src/main/resources/web/jsps/sources.jsp | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/src/main/resources/web/json/categories.json b/src/main/resources/web/json/categories.json index 06efe8f..b323c79 100644 --- a/src/main/resources/web/json/categories.json +++ b/src/main/resources/web/json/categories.json @@ -81,7 +81,7 @@ "icon": "fa-download", "header": "Applications", "summary": "How many Tor applications, like Tor Browser, have been downloaded or updated.", -"description": "The following application statistics are based on the analysis of web server requests. All data comes from https://webstats.torproject.org/\; target=\"_blank\">webstats.torproject.org which collects logs from torproject.org web servers and provides them as a stripped-down version of Apache's \"combined\" log format without IP addresses, log times, HTTP parameters, referers, and user agent strings.", +"description": "The following application statistics are based on the analysis of web server requests. All data comes from Tor web server logs which are provided in a stripped-down version of Apache's \"combined\" log format without IP addresses, log times, HTTP parameters, referers, and user agent strings.", "metrics": [ "webstats-tb", "webstats-tb-platform", diff --git a/src/main/resources/web/jsps/sources.jsp b/src/main/resources/web/jsps/sources.jsp index c956208..f8f9e07 100644 --- a/src/main/resources/web/jsps/sources.jsp +++ b/src/main/resources/web/jsps/sources.jsp @@ -37,7 +37,6 @@ https://github.com/robgjansen/onionperf; target="_blank">OnionPerf measures the performance of onion services. https://ooni.torproject.org/; target="_blank">OONI detects censorship, surveillance, and traffic manipulation on the internet. https://gitweb.torproject.org/user/phw/sybilhunter.git/; target="_blank">Sybilhunter attempts to detect Sybil attacks on the Tor network. -https://webstats.torproject.org/; target="_blank">Webstats collects logs from torproject.org web servers and provides them as a stripped-down version of Apache's "combined" log format without IP addresses, log times, HTTP parameters, referers, and user agent strings. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Use proper call list syntax.
commit 41907268f1b86337961b7035984ee084a3bd6bab
Author: iwakeh
Date: Tue Apr 3 07:32:38 2018 +
Use proper call list syntax.
---
src/main/java/org/torproject/metrics/web/RObjectGenerator.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
index e3a2419..aea6db7 100644
--- a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
+++ b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
@@ -131,7 +131,7 @@ public class RObjectGenerator implements
ServletContextListener {
}
String function = this.availableGraphs.get(requestedGraph)
.getFunction();
-queryBuilder.append(function).append("(");
+queryBuilder.append(function).append(", ");
StringBuilder imageFilenameBuilder =
new StringBuilder(requestedGraph);
for (Map.Entry parameter
@@ -155,7 +155,7 @@ public class RObjectGenerator implements
ServletContextListener {
}
imageFilenameBuilder.append(".").append(fileType);
String imageFilename = imageFilenameBuilder.toString();
-queryBuilder.append("path = '%1$s'))), '%1$s')");
+queryBuilder.append("path = '%1$s')), '%1$s')");
String query = queryBuilder.toString();
File imageFile = new File(this.cachedGraphsDirectory + "/"
+ imageFilename);
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Throw out Apache Commons Lang 2 in favor of 3.
commit f4c98122da6b21f93e00b149724e33f689cfc580 Author: Karsten Loesing Date: Wed May 16 12:42:26 2018 +0200 Throw out Apache Commons Lang 2 in favor of 3. Part of #25699. --- build.xml | 1 - src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/build.xml b/build.xml index 9b29452..bdb908b 100644 --- a/build.xml +++ b/build.xml @@ -44,7 +44,6 @@ - diff --git a/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java b/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java index 9959b93..cfbdd6c 100644 --- a/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java +++ b/src/main/java/org/torproject/metrics/stats/ipv6servers/Parser.java @@ -9,7 +9,7 @@ import org.torproject.descriptor.RelayNetworkStatusConsensus; import org.torproject.descriptor.RelayServerDescriptor; import org.torproject.descriptor.ServerDescriptor; -import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang3.StringUtils; import java.time.Instant; import java.time.ZoneId; ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Fix typo on Statistics page.
commit 215f3639c245a4ed200158526bf03b3d07f25677 Author: Karsten Loesing Date: Mon Apr 23 12:41:29 2018 +0200 Fix typo on Statistics page. --- src/main/resources/web/jsps/stats.jsp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/stats.jsp b/src/main/resources/web/jsps/stats.jsp index c224a06..2674ba0 100644 --- a/src/main/resources/web/jsps/stats.jsp +++ b/src/main/resources/web/jsps/stats.jsp @@ -452,7 +452,7 @@ uni-/bidirectional connection usage were reported. direction: Direction of reported fraction, which can be "read", "write", or "both" for connections classified as -"mostly reading", "mostly writing", or "both reading as writing". Connections +"mostly reading", "mostly writing", or "both reading and writing". Connections below the threshold have been removed from this statistics file entirely. quantile: Quantile of the reported fraction when considering all ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 257 of doc/MetricsTimeline.
commit d41b8eb4165cfac4c9e2b415f58603152977ddb0
Author: Karsten Loesing
Date: Mon May 7 10:03:44 2018 +0200
Update news.json to version 257 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 19 +++
1 file changed, 19 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 32b0630..77d94aa 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4569,6 +4569,25 @@
]
},
{
+"start": "2018-03-28",
+"ongoing": true,
+"places": [
+ "td"
+],
+"shortDescription": "Chad blocks social messaging apps.",
+"description": "Chad blocks social messaging apps.",
+"links": [
+ {
+"label": "article",
+"target":
"https://qz.com/1247234/chad-has-blocked-social-messaging-apps-bbc-amid-political-and-economic-anxiety/;
+ },
+ {
+"label": "graph",
+"target":
"https://metrics.torproject.org/userstats-relay-country.html?start=2018-02-01=2018-05-15=td=off;
+ }
+]
+ },
+ {
"start": "2018-04-02",
"protocols": [
"snowflake"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Convert shell script into Ant task.
commit 3be34c3be42a9fda3258c152efa5f56a2693519d Author: iwakeh Date: Wed Jan 31 19:29:57 2018 + Convert shell script into Ant task. --- build.xml | 49 ++ src/main/resources/spec/convert.sh | 6 - 2 files changed, 49 insertions(+), 6 deletions(-) diff --git a/build.xml b/build.xml index 57eab68..0e05d12 100644 --- a/build.xml +++ b/build.xml @@ -24,6 +24,8 @@ + + + + @@ -249,6 +255,49 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/main/resources/spec/convert.sh b/src/main/resources/spec/convert.sh deleted file mode 100755 index 95287e2..000 --- a/src/main/resources/spec/convert.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash -for specfile in "bridge-descriptors" "web-server-logs"; do - saxon-xslt $specfile.xml rfc2629.xslt xml2rfc-topblock=no | \ - tidy -q | awk -f convert.awk > ../web/jsps/$specfile.jsp -done - ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Link directly to CollecTor subdirectories.
commit 0af3488050be3a7877d34814e39eb42465afe7ec Author: Karsten Loesing Date: Wed Apr 4 16:15:40 2018 +0200 Link directly to CollecTor subdirectories. --- src/main/resources/web/jsps/collector.jsp | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/main/resources/web/jsps/collector.jsp b/src/main/resources/web/jsps/collector.jsp index 13865ba..3e98785 100644 --- a/src/main/resources/web/jsps/collector.jsp +++ b/src/main/resources/web/jsps/collector.jsp @@ -382,7 +382,7 @@ The sanitizing steps are specified in detail on a separate Bridge Network Statuses @type bridge-network-status 1.2 recent - archive + archive # @@ -407,7 +407,7 @@ authority which produced the document, to the header. Bridge Server descriptors @type bridge-server-descriptor 1.2 recent - archive + archive # @@ -441,7 +441,7 @@ ports. Bridge Extra-info Descriptors @type bridge-extra-info 1.3 recent - archive + archive # ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Make war-task depend on jar-task.
commit b1e05125fa3442e17cf78994e0818ba540b38571 Author: iwakeh Date: Fri Apr 6 09:02:03 2018 + Make war-task depend on jar-task. This makes the git-revision available. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index 17160ee..9b29452 100644 --- a/build.xml +++ b/build.xml @@ -137,7 +137,7 @@ + depends="submoduleupdate,jar"> ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update Onionoo specification for version 6.0.
commit 1ec5da2243334c27b95cf7011ae428c78c1010e0 Author: Karsten Loesing Date: Tue Apr 17 15:04:58 2018 +0200 Update Onionoo specification for version 6.0. --- src/main/resources/web/jsps/onionoo.jsp | 11 +-- 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index 540ca23..b225732 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -304,9 +304,8 @@ required fields on March 14, 2018. Added "version_status" field to details documents on April 6, 2018. # 6.0 -(scheduled, but not deployed yet!): Include all exit addresses in -"exit_addresses", regardless of whether they're used as OR addresses or not, to -be deployed after April 14, 2018. +Included all exit addresses in "exit_addresses", regardless of whether they are +used as onion-routing addresses or not on April 17, 2018. # @@ -1053,7 +1052,7 @@ IPv6 hex characters are all lower-case. -exit_addresses +exit_addresses updated array of strings optional # @@ -1061,9 +1060,9 @@ IPv6 hex characters are all lower-case. Array of IPv4 or IPv6 addresses that the relay used to exit to the Internet in the past 24 hours. IPv6 hex characters are all lower-case. -Only those addresses are listed that are different from onion-routing -addresses. Omitted if array is empty. +Changed on April 17, 2018 to include all exit addresses, regardless of whether +they are used as onion-routing addresses or not. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update to latest metrics-lib.
commit d85e4368a4d91846a1541aa1a72242bb001a5e3c Author: Karsten Loesing Date: Wed Mar 28 16:43:16 2018 +0200 Update to latest metrics-lib. --- src/submods/metrics-lib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index 2e235ef..f826234 16 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit 2e235ef8f38aac679817fa8e8b9171aadb3d5b9d +Subproject commit f826234e8064014487d871838e8074e83a142e42 ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update to latest metrics-lib.
commit 6e681a231ef35247a3df920e87eedaeef5f155b1 Author: Karsten Loesing Date: Mon Apr 23 12:26:05 2018 +0200 Update to latest metrics-lib. --- src/submods/metrics-lib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index f826234..77bbd74 16 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit f826234e8064014487d871838e8074e83a142e42 +Subproject commit 77bbd742b63fd9708694d7d8623d82a9654b770e ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update to latest rs (Relay Search).
commit c139211db16559df38ad54287243fcac5f60a676 Author: Karsten Loesing Date: Thu Mar 22 14:58:59 2018 +0100 Update to latest rs (Relay Search). --- src/main/resources/web/rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/rs b/src/main/resources/web/rs index 295f48a..b6e6c32 16 --- a/src/main/resources/web/rs +++ b/src/main/resources/web/rs @@ -1 +1 @@ -Subproject commit 295f48afbf7912f993379173456c7913daf22a16 +Subproject commit b6e6c3235806c9d1521dbc2c08cecee626466928 ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Make generate-specs target optional.
commit ac10cec5a86dbbfd821bb6572636fe4cb23cbdde Author: Karsten Loesing Date: Wed Mar 28 16:41:19 2018 +0200 Make generate-specs target optional. Without this change, `ant war` would not succeed, because it finds the spec JSP files in two places with the same name, and that is not allowed. --- build.xml | 6 +- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/build.xml b/build.xml index 0e05d12..3b4222e 100644 --- a/build.xml +++ b/build.xml @@ -151,7 +151,6 @@ - - @@ -287,7 +283,7 @@ ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 247 of doc/MetricsTimeline.
commit b5f3e402159a7f5ed9d955aa83324343123f0cb1
Author: Karsten Loesing
Date: Wed Mar 28 17:17:43 2018 +0200
Update news.json to version 247 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 18 ++
1 file changed, 18 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 2f72791..8081a2f 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4515,6 +4515,24 @@
]
},
{
+"start": "2018-03-26",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Release of Tor Browser 8.0a5",
+"description": "Release of Tor Browser 8.0a5. Improves snowflake client
performance.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-80a5-released;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/21312;
+ }
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Make userstats pgTAP test work again.
commit fee3d6a0d38fbb9709dc2b242b5f206879d46429
Author: Karsten Loesing
Date: Tue Feb 20 21:07:40 2018 +0100
Make userstats pgTAP test work again.
All calls of merge() exept for the first have been broken. The reason
is that we're creating a temporary table as part of that function,
which will only be dropped at the end of the session. But given that
all pgTAP tests run in a single session, that temporary table was not
dropped. The fix is to drop it explicitly after calling merge().
This change doesn't affect production, because we're only calling the
merge() function once per session in production.
It's not entirely clear why these tests worked in the beginning. My
current guess is that there was a PostgreSQL or pgTAP change regarding
temporary tables or running tests in a single session. And maybe that
change happened years ago and we just did not notice until now.
Fixes #24713.
---
src/test/sql/userstats/test-userstats.sql | 76 +--
1 file changed, 51 insertions(+), 25 deletions(-)
diff --git a/src/test/sql/userstats/test-userstats.sql
b/src/test/sql/userstats/test-userstats.sql
index 405bf51..5be866d 100644
--- a/src/test/sql/userstats/test-userstats.sql
+++ b/src/test/sql/userstats/test-userstats.sql
@@ -192,6 +192,7 @@ PREPARE new_merged(timestamp without time zone,
EXECUTE new_imported('2013-04-11 14:00:00', '2013-04-11 15:00:00');
SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 15:00:00'::timestamp without time zone)$$,
'Should insert new entry into empty table as is');
@@ -200,7 +201,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 14:00:00');
EXECUTE new_imported('2013-04-11 16:00:00', '2013-04-11 17:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 14:00:00'::timestamp without time zone),
('2013-04-11 17:00:00'::timestamp without time zone)$$,
@@ -210,7 +212,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 15:00:00');
EXECUTE new_imported('2013-04-11 15:00:00', '2013-04-11 17:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 17:00:00'::timestamp without time zone)$$,
'Should merge two contiguous entries');
@@ -219,7 +222,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 16:00:00');
EXECUTE new_imported('2013-04-11 14:00:00', '2013-04-11 17:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 16:00:00'::timestamp without time zone)$$,
'Should skip entry that starts before and ends after the start of ' ||
@@ -229,7 +233,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 15:00:00');
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 16:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 15:00:00'::timestamp without time zone)$$,
'Should skip entry that starts at and ends after the start of ' ||
@@ -239,7 +244,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 16:00:00');
EXECUTE new_imported('2013-04-11 14:00:00', '2013-04-11 15:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 16:00:00'::timestamp without time zone)$$,
'Should skip entry that starts after another new entry starts and ' ||
@@ -249,7 +255,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 16:00:00');
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 16:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 16:00:00'::timestamp without time zone)$$,
'Should skip entry that has same start and end as another new entry');
@@ -258,7 +265,8 @@ DELETE FROM merged;
EXECUTE new_imported('2013-04-11 13:00:00', '2013-04-11 16:00:00');
EXECUTE new_imported('2013-04-11 14:00:00', '2013-04-11 16:00:00');
--- SELECT merge(); -- TODO fix: ERROR: relation "merged_part" already exists
+SELECT merge();
+DROP TABLE merged_part;
SELECT bag_eq('SELECT stats_end FROM merged',
$$VALUES ('2013-04-11 16:00:00'::timestamp without time zone)$$,
[tor-commits] [metrics-web/release] Use 'StringBuilder#append' consistently.
commit 03fc04edbd31490dcb1c31499c6ddce12a214328
Author: iwakeh
Date: Sat Mar 24 09:31:07 2018 +
Use 'StringBuilder#append' consistently.
---
.../java/org/torproject/metrics/web/RObjectGenerator.java | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
index 5d9b29c..04bda0c 100644
--- a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
+++ b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
@@ -188,18 +188,18 @@ public class RObjectGenerator implements
ServletContextListener {
String parameterName = parameter.getKey();
String[] parameterValues = parameter.getValue();
for (String param : parameterValues) {
-tableFilenameBuilder.append("-" + param);
+tableFilenameBuilder.append("-").append(param);
}
if (parameterValues.length < 2) {
-queryBuilder.append(parameterName + " = '"
-+ parameterValues[0] + "', ");
+queryBuilder.append(parameterName).append(" = '")
+.append(parameterValues[0]).append("', ");
} else {
-queryBuilder.append(parameterName + " = c(");
+queryBuilder.append(parameterName).append(" = c(");
for (int i = 0; i < parameterValues.length - 1; i++) {
- queryBuilder.append("'" + parameterValues[i] + "', ");
+ queryBuilder.append("'").append(parameterValues[i]).append("', ");
}
-queryBuilder.append("'" + parameterValues[
-parameterValues.length - 1] + "'), ");
+queryBuilder.append("'").append(parameterValues[
+parameterValues.length - 1]).append("'), ");
}
}
tableFilenameBuilder.append(".tbl");
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Document Onionoo's new protocol version 5.2.
commit c2e0babcf84bbf0a84bbb890d2e39a39da4582d2 Author: Karsten Loesing Date: Fri Apr 6 11:02:02 2018 +0200 Document Onionoo's new protocol version 5.2. --- src/main/resources/web/jsps/onionoo.jsp | 49 + 1 file changed, 49 insertions(+) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index 28b16c6..540ca23 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -300,6 +300,9 @@ Always added a relay's own fingerprint to its "effective_family" and made nickname fields "n" in summary documents and "nickname" in details documents required fields on March 14, 2018. # +5.2 +Added "version_status" field to details documents on April 6, 2018. +# 6.0 (scheduled, but not deployed yet!): Include all exit addresses in "exit_addresses", regardless of whether they're used as OR addresses or not, to @@ -1490,6 +1493,29 @@ the relay did not report which version it runs. + +version_status new +string +optional +# + +Status of the Tor software version of this relay based on the versions +recommended by the directory authorities. +Possible version statuses are: +"recommended" if a version is listed as recommended; +"experimental" if a version is newer than every recommended version; +"obsolete" if a version is older than every recommended version; +"new in series" if a version has other recommended versions with the same first +three components, and the version is newer than all such recommended versions, +but it is not newer than every recommended version; +"unrecommended" if none of the above conditions hold. +Omitted if either the directory authorities did not recommend versions, or the +relay did not report which version it runs. +Added on April 6, 2018. + + + + effective_family updated array of strings @@ -1829,6 +1855,29 @@ the bridge did not report which version it runs. + +version_status new +string +optional +# + +Status of the Tor software version of this bridge based on the versions +recommended by the directory authorities. +Possible version statuses are: +"recommended" if a version is listed as recommended; +"experimental" if a version is newer than every recommended version; +"obsolete" if a version is older than every recommended version; +"new in series" if a version has other recommended versions with the same first +three components, and the version is newer than all such recommended versions, +but it is not newer than every recommended version; +"unrecommended" if none of the above conditions hold. +Omitted if either the directory authorities did not recommend versions, or the +bridge did not report which version it runs. +Added on April 6, 2018. + + + + transports array of strings ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Move connbidirect graph to Traffic category.
commit 3ef4b4b43a700814388e2230dd1ff8e015891a1f
Author: Karsten Loesing
Date: Mon Apr 23 12:06:51 2018 +0200
Move connbidirect graph to Traffic category.
Implements #25896.
---
src/main/resources/web/json/categories.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main/resources/web/json/categories.json
b/src/main/resources/web/json/categories.json
index 82cf1f4..06efe8f 100644
--- a/src/main/resources/web/json/categories.json
+++ b/src/main/resources/web/json/categories.json
@@ -49,6 +49,7 @@
"advbwdist-relay",
"bwhist-flags",
"dirbytes",
+ "connbidirect",
"uncharted-data-flow"
]
},
@@ -60,8 +61,7 @@
"description": "We use https://gitweb.torproject.org/torperf.git\;>Torperf and https://github.com/robgjansen/onionperf\;>OnionPerf to run
performance measurements. Both work by fetching files of different sizes over
Tor and measuring how long that takes.",
"metrics": [
"torperf",
- "torperf-failures",
- "connbidirect"
+ "torperf-failures"
]
},
{
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 248 of doc/MetricsTimeline.
commit 3e81659b4a87ad8d2a5071400f4101b014f6356d
Author: Karsten Loesing
Date: Tue Apr 3 09:32:37 2018 +0200
Update news.json to version 248 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 12
1 file changed, 12 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 8081a2f..c503aa1 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -5331,6 +5331,10 @@
{
"label": "reddit thread",
"target":
"https://www.reddit.com/r/TOR/comments/7b53kq/direct_users_from_germany_go_from_200k_to_500k_in/;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/24669;
}
],
"unknown": true
@@ -5399,6 +5403,10 @@
{
"label": "reddit thread",
"target":
"https://www.reddit.com/r/TOR/comments/7ldrox/tor_usage_in_germany_what_happend/;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/24669;
}
],
"unknown": true
@@ -5459,6 +5467,10 @@
{
"label": "graph",
"target":
"https://metrics.torproject.org/userstats-relay-country.html?start=2017-11-10=2018-03-10=de=off;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/24669;
}
],
"unknown": true
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Adapt webstats to read logs from CollecTor.
commit 7be397890d2ff66d7479b52aa245afdd3d487f9d
Author: Karsten Loesing
Date: Thu Mar 22 21:37:28 2018 +0100
Adapt webstats to read logs from CollecTor.
In this change we:
- update metrics-lib to 2.2.0,
- start downloading and processing logs from CollecTor rather than
from webstats.torproject.org,
- change log line counts from int to long,
- remove webstats tests which are now contained in metrics-lib,
- update the CollecTor page, and
- take out the beta notice from the Tor web server logs page.
Implements #25520.
---
build.xml | 2 +-
.../metrics/stats/collectdescs/Main.java | 3 +-
.../torproject/metrics/stats/webstats/Main.java| 223 ++---
src/main/resources/spec/web-server-logs.xml| 4 -
src/main/resources/web/jsps/collector.jsp | 38
src/main/resources/web/jsps/web-server-logs.jsp| 16 +-
src/main/sql/webstats/init-webstats.sql| 2 +-
.../metrics/stats/webstats/MainTest.java | 110 --
8 files changed, 106 insertions(+), 292 deletions(-)
diff --git a/build.xml b/build.xml
index e98757e..57eab68 100644
--- a/build.xml
+++ b/build.xml
@@ -9,7 +9,7 @@
-
+
diff --git a/src/main/java/org/torproject/metrics/stats/collectdescs/Main.java
b/src/main/java/org/torproject/metrics/stats/collectdescs/Main.java
index 04dc86d..4c64425 100644
--- a/src/main/java/org/torproject/metrics/stats/collectdescs/Main.java
+++ b/src/main/java/org/torproject/metrics/stats/collectdescs/Main.java
@@ -24,7 +24,8 @@ public class Main {
"/recent/relay-descriptors/consensuses/",
"/recent/relay-descriptors/extra-infos/",
"/recent/relay-descriptors/server-descriptors/",
-"/recent/torperf/"
+"/recent/torperf/",
+"/recent/webstats/"
}, 0L, new File("../../shared/in"), true);
}
}
diff --git a/src/main/java/org/torproject/metrics/stats/webstats/Main.java
b/src/main/java/org/torproject/metrics/stats/webstats/Main.java
index f70963f..5d4 100644
--- a/src/main/java/org/torproject/metrics/stats/webstats/Main.java
+++ b/src/main/java/org/torproject/metrics/stats/webstats/Main.java
@@ -3,14 +3,19 @@
package org.torproject.metrics.stats.webstats;
-import org.apache.commons.compress.compressors.xz.XZCompressorInputStream;
+import static java.util.stream.Collectors.counting;
+import static java.util.stream.Collectors.groupingByConcurrent;
+
+import org.torproject.descriptor.Descriptor;
+import org.torproject.descriptor.DescriptorParseException;
+import org.torproject.descriptor.DescriptorSourceFactory;
+import org.torproject.descriptor.WebServerAccessLog;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.BufferedReader;
+import java.io.File;
import java.io.IOException;
-import java.io.InputStreamReader;
-import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
@@ -23,19 +28,17 @@ import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.text.DateFormat;
-import java.text.ParseException;
import java.text.SimpleDateFormat;
+import java.time.LocalDate;
+import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Calendar;
-import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.SortedSet;
import java.util.TimeZone;
import java.util.TreeSet;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
/** Main class of the webstats module that downloads log files from the server,
* imports them into a database, and exports aggregate statistics to a CSV
@@ -45,26 +48,6 @@ public class Main {
/** Logger for this class. */
private static Logger log = LoggerFactory.getLogger(Main.class);
- /** Pattern for links contained in directory listings. */
- static final Pattern URL_STRING_PATTERN =
- Pattern.compile(".*.*");
-
- static final Pattern LOG_FILE_URL_PATTERN =
- Pattern.compile("^.*/([^/]+)/([^/]+)-access.log-(\\d{8}).xz$");
-
- private static DateFormat logDateFormat;
-
- static {
-logDateFormat = new SimpleDateFormat("MMdd");
-logDateFormat.setLenient(false);
-logDateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
- }
-
- static final Pattern LOG_LINE_PATTERN = Pattern.compile(
- "^0.0.0.[01] - - \\[\\d{2}/\\w{3}/\\d{4}:00:00:00 \\+\\] "
- + "\"(GET|HEAD) ([^ ]{1,2048}) HTTP[^ ]+\" (\\d+) (-|\\d+) \"-\" \"-\" "
- + "-$");
-
private static final String LOG_DATE = "log_date";
private static final String REQUEST_TYPE = "request_type";
@@ -88,12 +71,10 @@ public class Main {
log.info("Starting webstats module.");
String dbUrlString = "jdbc:postgresql:webstats";
Connection connection = connectToDatabase(dbUrlString);
-
[tor-commits] [metrics-web/release] Update news.json to version 243 of doc/MetricsTimeline.
commit d16e66ba863706704a993f6821e58bc7041f820b
Author: Karsten Loesing
Date: Thu Mar 22 14:58:35 2018 +0100
Update news.json to version 243 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 97 +++
1 file changed, 97 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 00c4cf1..8e103d7 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4230,6 +4230,20 @@
"description": "Tor 0.3.3.2-alpha reaches deb.torproject.org repositories"
},
{
+"start": "2018-02-24",
+"places": [
+ "cn"
+],
+"shortDescription": "Blocking in China intensifies following the
announcement of proposed changes to remove presidential term limits.",
+"description": "Blocking in China intensifies following the announcement
of proposed changes to remove presidential term limits.",
+"links": [
+ {
+"label": "article",
+"target":
"https://chinadigitaltimes.net/2018/02/sensitive-words-emperor-xi-jinping-ascend-throne/;
+ }
+]
+ },
+ {
"start": "2018-02-27",
"end": "2018-03-02",
"shortDescription": "Less than 3 bandwidth authorities are available.",
@@ -4257,6 +4271,89 @@
]
},
{
+"start": "2018-03-07",
+"end": "2018-03-15",
+"places": [
+ "lk"
+],
+"shortDescription": "Sri Lanka blocks Facebook, Instagram, WhatsApp, and
Viber.",
+"description": "Sri Lanka blocks Facebook, Instagram, WhatsApp, and
Viber.",
+"links": [
+ {
+"label": "relay graph",
+"target":
"https://metrics.torproject.org/userstats-relay-country.html?start=2018-01-01=2018-04-01=lk;
+ },
+ {
+"label": "bridge graph",
+"target":
"https://metrics.torproject.org/userstats-bridge-country.html?start=2018-01-01=2018-04-01=lk;
+ },
+ {
+"label": "article",
+"target":
"https://www.reuters.com/article/us-sri-lanka-clashes-socialmedia/sri-lanka-lifts-ban-on-facebook-imposed-after-spasm-of-communal-violence-idUSKCN1GR31R;
+ }
+]
+ },
+ {
+"start": "2018-03-14",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Deployed an attempt to fix frequent crashes of the
fallback snowflake proxies.",
+"description": "Deployed an attempt to fix frequent crashes of the
fallback snowflake proxies.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/21312#comment:39;
+ },
+ {
+"label": "commit",
+"target":
"https://gitweb.torproject.org/pluggable-transports/snowflake.git/commit/?id=f2abf5b60c289103d8661b8e54b3d3aee43a4aaf;
+ }
+]
+ },
+ {
+"start": "2018-03-16",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Deployed another attempt to fix frequent crashes of
the fallback snowflake proxies.",
+"description": "Deployed another attempt to fix frequent crashes of the
fallback snowflake proxies.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/21312#comment:57;
+ }
+]
+ },
+ {
+"start": "2018-03-22",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Disabled periodic restarts of the fallback snowflake
proxies; gave them a memory limit of 200 MB.",
+"description": "Disabled periodic restarts of the fallback snowflake
proxies; gave them a memory limit of 200 MB.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/21312#comment:63;
+ }
+]
+ },
+ {
+"start": "2018-03-22",
+"protocols": [
+ "snowflake"
+],
+"shortDescription": "Increased memory limit of the fallback snowflake
proxies from 200 MB to 400 MB.",
+"description": "Increased memory limit of the fallback snowflake proxies
from 200 MB to 400 MB.",
+"links": [
+ {
+"label": "comment",
+"target": "https://bugs.torproject.org/21312#comment:63;
+ }
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 246 of doc/MetricsTimeline.
commit e11c2962bf241f52023866373d597add77fd5882
Author: Karsten Loesing
Date: Mon Mar 26 11:48:36 2018 +0200
Update news.json to version 246 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 167 +-
1 file changed, 164 insertions(+), 3 deletions(-)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 8e103d7..2f72791 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -2826,6 +2826,21 @@
]
},
{
+"start": "2016-11-16",
+"shortDescription": "Tor Browser 6.5a4 is released",
+"description": "Tor Browser 6.5a4 is released. It changes the
app.update.url.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-65a4-released;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/19481;
+ }
+]
+ },
+ {
"start": "2016-11-18",
"protocols": [
"obfs4"
@@ -2937,6 +2952,20 @@
]
},
{
+"start": "2016-12-13",
+"protocols": [
+ "obfs4"
+],
+"shortDescription": "Tor Browser 6.5a6 is released",
+"description": "Tor Browser 6.5a6 is released. It fixes the
app.update.url. It adds (uncomments) the default obfs4 bridge
NX01:443. It activates timing obfuscation for certain obfs4 bridges. It changes
the port of other obfs4 bridges.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-65a6-released;
+ }
+]
+ },
+ {
"start": "2016-12-15",
"ongoing": true,
"places": [
@@ -3011,6 +3040,44 @@
]
},
{
+"start": "2017-01-24",
+"shortDescription": "Tor Browser 6.5 is released",
+"description": "Tor Browser 6.5 is released. It changes the
app.update.url, which causes an apparent drop in update pings.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-65-released;
+ },
+ {
+"label": "graph",
+"target":
"https://metrics.torproject.org/webstats-tb.html?start=2017-01-01=2017-06-01;
+ },
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/19481;
+ },
+ {
+"label": "comment about app.update.url",
+"target": "https://bugs.torproject.org/22346#comment:1;
+ }
+]
+ },
+ {
+"start": "2017-01-25",
+"end": "2017-04-05",
+"protocols": [
+ "webstats"
+],
+"shortDescription": "Tor Browser update pings roughly halve",
+"description": "Tor Browser update pings roughly halve. The beginning
corresponds with the release of Tor Browser 6.5.2, which changed the
app.update.url. The end doesn't correspond to any release.",
+"links": [
+ {
+"label": "ticket",
+"target": "https://bugs.torproject.org/22346;
+ }
+]
+ },
+ {
"start": "2017-02-10",
"ongoing": true,
"protocols": [
@@ -3182,6 +3249,41 @@
]
},
{
+"start": "2017-04-19",
+"protocols": [
+ "meek",
+ "obfs4"
+],
+"shortDescription": "Tor Browser 6.5.2 is released",
+"description": "Tor Browser 6.5.2 is released. It changes the
app.update.url. It changes the meek-amazon backend. It adds the
new default obfs4 bridges cymrubridge31:80 and cymrubridge33:80.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-652-released;
+ },
+ {
+"label": "comment about app.update.url",
+"target": "https://bugs.torproject.org/22346#comment:1;
+ }
+]
+ },
+ {
+"start": "2017-04-20",
+"protocols": [
+ "meek",
+ "obfs4",
+ "scramblesuit"
+],
+"shortDescription": "Tor Browser 7.0a3 is released",
+"description": "Tor Browser 7.0a3 is released. It changes the
app.update.url. It removes the last remaining scramblesuit bridge.
It changes the meek-amazon backend. It adds the new default obfs4 bridges
cymrubridge31:80 and cymrubridge33:80.",
+"links": [
+ {
+"label": "blog post",
+"target": "https://blog.torproject.org/tor-browser-70a3-released;
+ }
+]
+ },
+ {
"start": "2017-04-30",
"protocols": [
"meek"
@@ -4090,14 +4192,18 @@
},
{
"start": "2017-12-28",
-"ongoing": true,
+"end": "2018-01-13",
"places": [
"ir"
],
-"shortDescription": "Protests in Iran, blocking of various services
including Tor.",
-"description": "Protests in Iran, blocking of various services including
Tor.",
+"shortDescription": "Protests in Iran, blocking of various services
including Tor",
+"description": "Protests in Iran, blocking of various services including
Tor. Instagram was unblocked 2018-01-06. Telegram was unblocked 2018-01-13.",
"links": [
{
+"label":
[tor-commits] [metrics-web/release] Update news.json to version 256 of doc/MetricsTimeline.
commit b0506154323fa2c73152afb147976e5e7611a853
Author: Karsten Loesing
Date: Mon Apr 23 12:27:19 2018 +0200
Update news.json to version 256 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 15 +++
1 file changed, 15 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 8385b6e..32b0630 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -4597,6 +4597,21 @@
]
},
{
+"start": "2018-04-16",
+"ongoing": true,
+"places": [
+ "ru"
+],
+"shortDescription": "Russia tries to block Telegram; disrupts many
services on Amazon and Google cloud services.",
+"description": "Russia tries to block Telegram; disrupts many services on
Amazon and Google cloud services.",
+"links": [
+ {
+"label": "article",
+"target":
"https://www.bleepingcomputer.com/news/government/russia-bans-18-million-amazon-and-google-ips-in-attempt-to-block-telegram/;
+ }
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Imports UpdateNews.java
commit 664195d24da6408707aecda856fb7f6f9ff3c494
Author: Iain R. Learmonth
Date: Tue Mar 20 14:11:49 2018 +
Imports UpdateNews.java
This commit imports UpdateNews.java verbatim, as attached to #23854 by
Karsten.
https://trac.torproject.org/projects/tor/attachment/ticket/23854/UpdateNews.java
---
.../org/torproject/metrics/web/UpdateNews.java | 131 +
1 file changed, 131 insertions(+)
diff --git a/src/main/java/org/torproject/metrics/web/UpdateNews.java
b/src/main/java/org/torproject/metrics/web/UpdateNews.java
new file mode 100644
index 000..527988f
--- /dev/null
+++ b/src/main/java/org/torproject/metrics/web/UpdateNews.java
@@ -0,0 +1,131 @@
+package org.torproject.metrics.web;
+
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+
+import java.io.BufferedReader;
+import java.io.FileWriter;
+import java.io.InputStreamReader;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+
+public class UpdateNews {
+ private static class News {
+String start;
+String end;
+Boolean ongoing;
+List places;
+List protocols;
+String description;
+List links;
+Boolean unknown;
+ }
+
+ /** Update news. */
+ public static void main(String[] args) throws Exception {
+URL textFile = new URL(
+"https://trac.torproject.org/projects/tor/wiki/doc/;
++ "MetricsTimeline?format=txt");
+Gson gson = new GsonBuilder().disableHtmlEscaping().create();
+List news = new ArrayList<>();
+try (BufferedReader br = new BufferedReader(new InputStreamReader(
+textFile.openStream( {
+ String line;
+ Boolean unknown = null;
+ while ((line = br.readLine()) != null) {
+if (line.startsWith("== Unknown")) {
+ unknown = true;
+ continue;
+}
+if (!line.startsWith("||") || line.startsWith("||=start")) {
+ continue;
+}
+line = line.trim()
+.replaceAll("Ã", "")
+.replaceAll("§", "")
+.replaceAll("â", "")
+.replaceAll("â", "")
+.replaceAll("â", "");
+String[] parts = line.split("\\|\\|");
+News entry = new News();
+entry.start = parts[1].replaceAll("~", "").trim();
+if (entry.start.contains(" ")) {
+ entry.start = entry.start.substring(0, entry.start.indexOf(" "));
+} else if (entry.start.isEmpty()) {
+ entry.start = null;
+}
+entry.end = parts[2].replaceAll("~", "").trim();
+if ("ongoing".equalsIgnoreCase(entry.end)) {
+ entry.end = null;
+ entry.ongoing = true;
+} else if (entry.end.contains(" ")) {
+ entry.end = entry.end.substring(0, entry.end.indexOf(" "));
+} else if (entry.end.isEmpty()) {
+ entry.end = null;
+}
+for (String place : parts[3].trim().split(" +")) {
+ if (!place.isEmpty()) {
+if (null == entry.places) {
+ entry.places = new ArrayList<>();
+}
+entry.places.add(place);
+ }
+}
+for (String protocol : parts[4].trim().split(" +")) {
+ if (!protocol.isEmpty()) {
+if (null == entry.protocols) {
+ entry.protocols = new ArrayList<>();
+}
+entry.protocols.add(protocol);
+ }
+}
+String desc = parts[5].trim();
+while (desc.contains("[")) {
+ int open = desc.indexOf("[");
+ int space = desc.indexOf(" ", open);
+ int close = desc.indexOf("]", open);
+ if (open < 0 || space < 0 || close < 0) {
+System.err.println("Cannot convert link.");
+System.exit(1);
+ }
+ desc = desc.substring(0, open) + ""
+ + desc.substring(space + 1, close) + ""
+ + desc.substring(close + 1);
+}
+while (desc.contains("`")) {
+ int open = desc.indexOf("`");
+ int close = desc.indexOf("`", open + 1);
+ if (open < 0 || close < 0) {
+System.err.println("Cannot convert code fragment.");
+System.exit(1);
+ }
+ desc = desc.substring(0, open) + ""
+ + desc.substring(open + 1, close) + ""
+ + desc.substring(close + 1);
+}
+entry.description = desc;
+if (parts.length >= 7) {
+ for (String link : parts[6].split("[\\[\\]]")) {
+link = link.trim();
+if (link.isEmpty()) {
+ continue;
+}
+if (null == entry.links) {
+ entry.links = new ArrayList<>();
+}
+entry.links.add("" + link.substring(link.indexOf(" ") + 1) + "");
+ }
+}
+entry.unknown = unknown;
+news.add(entry);
+ }
+}
+try (FileWriter fw = new FileWriter(
+
[tor-commits] [metrics-web/release] Cut off lower number of expected clients at 0.
commit 5d039ad44c10cd82dd9ba8fb1efd6a7caa9b6090
Author: Karsten Loesing
Date: Mon Mar 12 17:33:47 2018 +0100
Cut off lower number of expected clients at 0.
Fixes #25397.
---
src/main/python/clients/detector.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/main/python/clients/detector.py
b/src/main/python/clients/detector.py
index 611f25b..6cf1c7d 100644
--- a/src/main/python/clients/detector.py
+++ b/src/main/python/clients/detector.py
@@ -363,6 +363,8 @@ def write_all(tss, minc, maxc, RANGES_FILE, INTERVAL=7):
if not minv < maxv:
print minv, maxv, series[i-INTERVAL], minc[i], maxc[i]
assert minv < maxv
+if minv < 0.0:
+ minv = 0.0
ranges_file.write("%s,%s,%s,%s\n" % (tss.all_dates[i], c, minv, maxv))
ranges_file.close()
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Adds a link to the ATOM feed to the News page
commit 82e089646a7af66ff3ed1937fac034ea25398356 Author: Iain R. Learmonth Date: Tue Mar 20 18:53:21 2018 + Adds a link to the ATOM feed to the News page --- src/main/resources/web/jsps/news.jsp | 5 + 1 file changed, 5 insertions(+) diff --git a/src/main/resources/web/jsps/news.jsp b/src/main/resources/web/jsps/news.jsp index c0972a0..abde1e0 100644 --- a/src/main/resources/web/jsps/news.jsp +++ b/src/main/resources/web/jsps/news.jsp @@ -13,6 +13,11 @@ + + + + + News # You're a journalist or more generally a person who wants to know what's going on in the Tor network? We're collecting unusual events in the Tor network together with any insights we have into what we think has happened. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update to latest metrics-lib.
commit 08b646bb16142f627a0ce804fce1722bdd272096 Author: Karsten Loesing Date: Tue Mar 20 10:31:55 2018 +0100 Update to latest metrics-lib. --- src/submods/metrics-lib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index 7648780..2e235ef 16 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit 7648780844f809f3068d3e625973fe43c78ccbb3 +Subproject commit 2e235ef8f38aac679817fa8e8b9171aadb3d5b9d ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Fix specified full path of sanitized webstats.
commit acc26196bcac4dc5ce908b1667fc22c677dd97f1 Author: Karsten Loesing Date: Mon Mar 19 14:50:01 2018 +0100 Fix specified full path of sanitized webstats. Also fix path in convert script. Fixes #25525. --- src/main/resources/spec/convert.sh | 2 +- src/main/resources/spec/web-server-logs.xml | 2 +- src/main/resources/web/jsps/web-server-logs.jsp | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/main/resources/spec/convert.sh b/src/main/resources/spec/convert.sh index 0287416..95287e2 100755 --- a/src/main/resources/spec/convert.sh +++ b/src/main/resources/spec/convert.sh @@ -1,6 +1,6 @@ #!/bin/bash for specfile in "bridge-descriptors" "web-server-logs"; do saxon-xslt $specfile.xml rfc2629.xslt xml2rfc-topblock=no | \ - tidy -q | awk -f convert.awk > ../web/WEB-INF/$specfile.jsp + tidy -q | awk -f convert.awk > ../web/jsps/$specfile.jsp done diff --git a/src/main/resources/spec/web-server-logs.xml b/src/main/resources/spec/web-server-logs.xml index 13cfad7..c180f8c 100644 --- a/src/main/resources/spec/web-server-logs.xml +++ b/src/main/resources/spec/web-server-logs.xml @@ -190,7 +190,7 @@ mod_log_config module. Sanitized log files may additionally be sorted into directories by virtual host and date as in: - virtual-host//MM/virtual-host_physical-host_access.log_MMDD[.xz] + virtual-host//MM/DD/virtual-host_physical-host_access.log_MMDD[.xz] The virtual hostnames, like 'metrics.torproject.org' or 'dist.torproject.org', are more familiar to the public and were therefore diff --git a/src/main/resources/web/jsps/web-server-logs.jsp b/src/main/resources/web/jsps/web-server-logs.jsp index 5e9cc79..8832b2a 100644 --- a/src/main/resources/web/jsps/web-server-logs.jsp +++ b/src/main/resources/web/jsps/web-server-logs.jsp @@ -277,7 +277,7 @@ of the filename. by virtual host and date as in: -virtual-host//MM/virtual-host_physical-host_access.log_MMDD[.xz] +virtual-host//MM/DD/virtual-host_physical-host_access.log_MMDD[.xz] The virtual hostnames, like 'metrics.torproject.org' or 'dist.torproject.org', are more familiar to the public and were ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 232 of doc/MetricsTimeline.
commit c155ed5ed97983fb719a94ba267a3ff58f9fab19
Author: Karsten Loesing
Date: Thu Mar 8 09:19:55 2018 +0100
Update news.json to version 232 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 8
1 file changed, 8 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 7036b7a..2cd27c6 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -2827,6 +2827,14 @@
]
},
{
+"start": "2018-03-03",
+"description": "Release of tor 0.3.2.10, 0.3.1.10, and 0.2.9.15, which
offer better resistance against DoS attacks.",
+"links": [
+ "https://blog.torproject.org/new-stable-tor-releases-security-fixes-and-dos-prevention-03210-03110-02915\;>blog
post",
+ "https://bugs.torproject.org/24902\;>ticket"
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Remove unused code from the censorship detector.
commit 4a30e7a9fbdded5318506302543b5d5e07571670
Author: Karsten Loesing
Date: Mon Mar 12 17:49:40 2018 +0100
Remove unused code from the censorship detector.
---
src/main/python/clients/detector.py | 199 +---
1 file changed, 2 insertions(+), 197 deletions(-)
diff --git a/src/main/python/clients/detector.py
b/src/main/python/clients/detector.py
index 6cf1c7d..3d17bf0 100644
--- a/src/main/python/clients/detector.py
+++ b/src/main/python/clients/detector.py
@@ -60,8 +60,6 @@ import country_info
# write utf8 to file
import codecs
-days = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
-
def get_country_name_from_cc(country_code):
if (country_code.lower() in country_info.countries):
return country_info.countries[country_code.lower()]
@@ -208,145 +206,6 @@ def make_tendencies_minmax(l, INTERVAL = 1):
## print minx[-1], maxx[-1]
return minx, maxx
-# Makes pretty plots
-def raw_plot(series, minc, maxc, labels, xtitle):
-assert len(xtitle) == 3
-fname, stitle, slegend = xtitle
-
-font = {'family' : 'Bitstream Vera Sans',
-'weight' : 'normal',
-'size' : 8}
-matplotlib.rc('font', **font)
-
-ylim( (-max(series)*0.1, max(series)*1.1) )
-plot(labels, series, linewidth=1.0, label="Users")
-
-wherefill = []
-for mm,mx in zip(minc, maxc):
- wherefill += [not (mm == None and mx == None)]
- assert mm < mx or (mm == None and mx == None)
-
-fill_between(labels, minc, maxc, where=wherefill, color="gray",
label="Prediction")
-
-vdown = []
-vup = []
-for i,v in enumerate(series):
- if minc[i] != None and v < minc[i]:
-vdown += [v]
-vup += [None]
- elif maxc[i] != None and v > maxc[i]:
-vdown += [None]
-vup += [v]
- else:
-vup += [None]
-vdown += [None]
-
-plot(labels, vdown, 'o', ms=10, lw=2, alpha=0.5, mfc='orange',
label="Downturns")
-plot(labels, vup, 'o', ms=10, lw=2, alpha=0.5, mfc='green',
label="Upturns")
-
-legend(loc=2)
-
-xlabel('Time (days)')
-ylabel('Users')
-title(stitle)
-grid(True)
-F = gcf()
-
-F.set_size_inches(10,5)
-F.savefig(fname, format="png", dpi = (150))
-close()
-
-def absolute_plot(series, minc, maxc, labels,INTERVAL, xtitle):
- in_minc = []
- in_maxc = []
- for i, v in enumerate(series):
-if i > 0 and i - INTERVAL >= 0 and series[i] != None and
series[i-INTERVAL] != None and series[i-INTERVAL] != 0 and minc[i]!= None and
maxc[i]!= None:
- in_minc += [minc[i] * poisson.ppf(1-0., series[i-INTERVAL])]
- in_maxc += [maxc[i] * poisson.ppf(0., series[i-INTERVAL])]
- if not in_minc[-1] < in_maxc[-1]:
-print in_minc[-1], in_maxc[-1], series[i-INTERVAL], minc[i], maxc[i]
- assert in_minc[-1] < in_maxc[-1]
-else:
- in_minc += [None]
- in_maxc += [None]
- raw_plot(series, in_minc, in_maxc, labels, xtitle)
-
-"""Return the number of downscores and upscores of a time series
-'series', given tendencies 'minc' and 'maxc' for the time interval
-'INTERVAL'.
-
-If 'scoring_interval' is specifed we only consider upscore/downscore
-that happened in the latest 'scoring_interval' days.
-"""
-def censor_score(series, minc, maxc, INTERVAL, scoring_interval=None):
- upscore = 0
- downscore = 0
-
- if scoring_interval is None:
-scoring_interval = len(series)
- assert(len(series) >= scoring_interval)
-
- for i, v in enumerate(series):
-if i > 0 and i - INTERVAL >= 0 and series[i] != None and
series[i-INTERVAL] != None and series[i-INTERVAL] != 0 and minc[i]!= None and
maxc[i]!= None:
- in_minc = minc[i] * poisson.ppf(1-0., series[i-INTERVAL])
- in_maxc = maxc[i] * poisson.ppf(0., series[i-INTERVAL])
- if (i >= (len(series) - scoring_interval)):
-downscore += 1 if minc[i] != None and v < in_minc else 0
-upscore += 1 if maxc[i] != None and v > in_maxc else 0
-
- return downscore, upscore
-
-def plot_target(tss, TARGET, xtitle, minx, maxx, DAYS=365, INTERV = 7):
- ctarget = tss.get_country_series(TARGET)
- c = n_day_rel(ctarget, INTERV)
- absolute_plot(ctarget[-DAYS:], minx[-DAYS:], maxx[-DAYS:],
tss.all_dates[-DAYS:],INTERV, xtitle = xtitle)
-
-def write_censorship_report_prologue(report_file, dates, notification_period):
- if (notification_period == 1):
-date_str = "%s" % (dates[-1]) # no need for date range if it's just one day
- else:
-date_str = "%s to %s" % (dates[-notification_period], dates[-1])
-
- prologue = "===\n"
- prologue += "Automatic Censorship Report for %s\n" % (date_str)
- prologue += "===\n\n"
- report_file.write(prologue)
-
-## Make a league table of censorship + nice graphs
-def plot_all(tss, minx, maxx, INTERV, DAYS=None, rdir="img"):
- rdir = os.path.realpath(rdir)
- if not os.path.exists(rdir) or not os.path.isdir(rdir):
-print "ERROR: %s does not exist or is not a directory." %
[tor-commits] [metrics-web/release] Adds an ATOM feed for the News (Fixes: #23854)
commit 556fa300319e5cce3e224df1b79684c473e6be77
Author: Iain R. Learmonth
Date: Tue Mar 20 18:37:17 2018 +
Adds an ATOM feed for the News (Fixes: #23854)
This extends NewsServlet to have an ATOM output format, as well as the
existing HTML output. A new JSP is used to serialize the output.
The feed has been tested to be compatible with both Firefox and
Thunderbird.
---
.../org/torproject/metrics/web/NewsServlet.java| 16 +
.../org/torproject/metrics/web/UpdateNews.java | 2 +-
src/main/resources/web.xml | 1 +
src/main/resources/web/jsps/news-atom.jsp | 76 ++
4 files changed, 94 insertions(+), 1 deletion(-)
diff --git a/src/main/java/org/torproject/metrics/web/NewsServlet.java
b/src/main/java/org/torproject/metrics/web/NewsServlet.java
index 3bfb1b7..a5775dd 100644
--- a/src/main/java/org/torproject/metrics/web/NewsServlet.java
+++ b/src/main/java/org/torproject/metrics/web/NewsServlet.java
@@ -53,7 +53,15 @@ public class NewsServlet extends AnyServlet {
@Override
public void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException, ServletException {
+if (request.getRequestURI().endsWith("news.atom")) {
+ doGetAtom(request, response);
+} else {
+ doGetHtml(request, response);
+}
+ }
+ private void doGetHtml(HttpServletRequest request,
+ HttpServletResponse response) throws IOException, ServletException {
/* Create categories based on current system time. */
Map cutOffDates = new LinkedHashMap<>();
Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC"),
Locale.US);
@@ -107,5 +115,13 @@ public class NewsServlet extends AnyServlet {
request.getRequestDispatcher("WEB-INF/news.jsp").forward(request,
response);
}
+
+ private void doGetAtom(HttpServletRequest request,
+ HttpServletResponse response) throws IOException, ServletException {
+request.setAttribute("news", this.sortedNews);
+request.setAttribute("updated", this.sortedNews.get(0).getStart());
+request.getRequestDispatcher("WEB-INF/news-atom.jsp").forward(request,
+response);
+ }
}
diff --git a/src/main/java/org/torproject/metrics/web/UpdateNews.java
b/src/main/java/org/torproject/metrics/web/UpdateNews.java
index 3705d4a..b8c95d6 100644
--- a/src/main/java/org/torproject/metrics/web/UpdateNews.java
+++ b/src/main/java/org/torproject/metrics/web/UpdateNews.java
@@ -116,7 +116,7 @@ public class UpdateNews {
continue;
}
entry.addLink(link.substring(link.indexOf(" ") + 1),
- link.substring(0, link.indexOf(" ")));
+link.substring(0, link.indexOf(" ")));
}
}
entry.unknown = unknown;
diff --git a/src/main/resources/web.xml b/src/main/resources/web.xml
index a5c3212..9c83591 100644
--- a/src/main/resources/web.xml
+++ b/src/main/resources/web.xml
@@ -238,6 +238,7 @@
NewsServlet
+/news.atom
/news.html
diff --git a/src/main/resources/web/jsps/news-atom.jsp
b/src/main/resources/web/jsps/news-atom.jsp
new file mode 100644
index 000..0c25b72
--- /dev/null
+++ b/src/main/resources/web/jsps/news-atom.jsp
@@ -0,0 +1,76 @@
+<%@ page contentType="text/xml" %>
+<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core; %>
+<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions; %>
+http://www.w3.org/2005/Atom;>
+ Tor Project
+ Metrics Timeline
+ T12:00:00Z
+ https://metrics.torproject.org/news.atom; rel="self" />
+ https://metrics.torproject.org/news.html; />
+ https://metrics.torproject.org/news.atom
+
+
+
+ T12:00:00Z
+ https://metrics.torproject.org/news.atom#${fn:length(entry.shortDescription)}
+
+http://www.w3.org/1999/xhtml;>
+
+Dates
+
+
+
+ N/A
+
+
+ to present
+
+
+
+
+
+ to
+
+
+
+
+
+ Places
+
+
+
+
+
+
+
+ Protocols
+
+
+
+
+
+
+
+ Unknown
+ Yes
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tor Metrics
+
+
+
+
___
tor-commits mailing list
tor-commits@lists.torproject.org
[tor-commits] [metrics-web/release] Update news.json to version 240 of doc/MetricsTimeline.
commit 21e145d947eb60765955b470cdd55707223404b4
Author: Karsten Loesing
Date: Thu Mar 15 10:51:40 2018 +0100
Update news.json to version 240 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 75 ++-
1 file changed, 74 insertions(+), 1 deletion(-)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 6beb053..49ba5a2 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -2621,6 +2621,20 @@
]
},
{
+"start": "2017-10-25",
+"protocols": [
+ "",
+ "relay"
+],
+"description": "Release of tor 0.2.5.15, 0.2.8.16, 0.2.9.13, 0.3.0.12, and
0.3.1.8, which add a new directory authority bastet, and change the IP address
of the directory authority Longclaw.",
+"links": [
+ "https://blog.torproject.org/new-stable-tor-releases-0318-03012-02913-02816-02515\;>blog
post",
+ "https://blog.torproject.org/introducing-bastet-our-new-directory-authority\;>Bastet
blog post",
+ "https://bugs.torproject.org/23910\;>bastet ticket",
+ "https://bugs.torproject.org/23592\;>Longclaw ticket"
+]
+ },
+ {
"start": "2017-11-01",
"places": [
"ru"
@@ -2661,6 +2675,17 @@
]
},
{
+"start": "2017-11-14",
+"protocols": [
+ "",
+ "relay"
+],
+"description": "Release of Tor Browser 7.0.10, containing tor 0.3.1.8,
which adds a new directory authority bastet, and changes the IP address of the
directory authority Longclaw.",
+"links": [
+ "https://blog.torproject.org/tor-browser-7010-released\;>blog
post"
+]
+ },
+ {
"start": "2017-11-15",
"end": "2017-11-15",
"protocols": [
@@ -2672,6 +2697,17 @@
]
},
{
+"start": "2017-11-15",
+"protocols": [
+ "",
+ "relay"
+],
+"description": "Release of Tor Browser 7.5a8, containing tor
0.3.2.4-alpha, which adds a new directory authority bastet, and changes the IP
address of the directory authority Longclaw.",
+"links": [
+ "https://blog.torproject.org/tor-browser-75a8-released\;>blog
post"
+]
+ },
+ {
"start": "2017-11-25",
"end": "2017-11-27",
"places": [
@@ -2685,6 +2721,19 @@
]
},
{
+"start": "2017-12-01",
+"protocols": [
+ "",
+ "ipv6",
+ "relay"
+],
+"description": "Release of tor 0.2.8.17, 0.2.9.14, 0.3.0.13, 0.3.1.9, and
0.3.2.6-alpha, which add an IPv6 address for the bridge authority bastet.",
+"links": [
+ "https://bugs.torproject.org/24394\;>ticket",
+ "https://blog.torproject.org/new-stable-tor-releases-security-fixes-0319-03013-02914-02817-02516\;>blog
post"
+]
+ },
+ {
"start": "2017-12-04",
"ongoing": true,
"description": "DDoS attack creates load on the network.",
@@ -2719,6 +2768,30 @@
]
},
{
+"start": "2017-12-09",
+"protocols": [
+ "",
+ "ipv6",
+ "relay"
+],
+"description": "Release of Tor Browser 7.0.11, containing tor 0.3.1.9,
which adds an IPv6 address for the bridge authority bastet.",
+"links": [
+ "https://blog.torproject.org/tor-browser-7011-released\;>blog
post"
+]
+ },
+ {
+"start": "2017-12-09",
+"protocols": [
+ "",
+ "ipv6",
+ "relay"
+],
+"description": "Release of Tor Browser 7.5a9, containing tor
0.3.2.6-alpha, which adds an IPv6 address for the bridge authority bastet.",
+"links": [
+ "https://blog.torproject.org/tor-browser-75a9-released\;>blog
post"
+]
+ },
+ {
"start": "2017-12-12",
"end": "2018-01-18",
"protocols": [
@@ -2739,7 +2812,7 @@
},
{
"start": "2017-12-21",
-"description": "Release of tor 0.3.2.8-rc, intended to fix the KIST bug
that enabled a DoS on relays by running them out of memory..",
+"description": "Release of tor 0.3.2.8-rc, intended to fix the KIST bug
that enabled a DoS on relays by running them out of memory.",
"links": [
"https://lists.torproject.org/pipermail/tor-talk/2017-December/043844.html\;>announcement",
"https://bugs.torproject.org/24665\;>ticket"
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Add 0.3.4 to known versions.
commit 77a2dfb994cdbbd6fe4330b61e195057a3ca307b
Author: Karsten Loesing
Date: Tue Mar 13 09:41:50 2018 +0100
Add 0.3.4 to known versions.
---
src/main/R/rserver/graphs.R | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index a31790d..57d2be2 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -370,7 +370,7 @@ plot_versions <- function(start, end, path) {
s <- prepare_versions(start, end)
known_versions <- c("Other", "0.1.0", "0.1.1", "0.1.2", "0.2.0",
"0.2.1", "0.2.2", "0.2.3", "0.2.4", "0.2.5", "0.2.6", "0.2.7",
-"0.2.8", "0.2.9", "0.3.0", "0.3.1", "0.3.2", "0.3.3")
+"0.2.8", "0.2.9", "0.3.0", "0.3.1", "0.3.2", "0.3.3", "0.3.4")
getPalette <- colorRampPalette(brewer.pal(12, "Paired"))
colours <- data.frame(breaks = known_versions,
values = rep(brewer.pal(min(12, length(known_versions)), "Paired"),
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update Onionoo specification to version 5.1.
commit a34a1c8804863877489d59cee27a935bea3e2196 Author: Karsten Loesing Date: Tue Mar 13 17:40:52 2018 +0100 Update Onionoo specification to version 5.1. Also schedule version 6.0 for release in one month. --- src/main/resources/web/jsps/onionoo.jsp | 45 - 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index 0f29507..28b16c6 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -295,6 +295,16 @@ Extended the "version" parameter to bridges, added a Removed the $ from fingerprints in fields "effective_family", "alleged_family", and "indirect_family" on December 20, 2017. # +5.1 +Always added a relay's own fingerprint to its "effective_family" and made +nickname fields "n" in summary documents and "nickname" in details documents +required fields on March 14, 2018. +# +6.0 +(scheduled, but not deployed yet!): Include all exit addresses in +"exit_addresses", regardless of whether they're used as OR addresses or not, to +be deployed after April 14, 2018. +# @@ -873,13 +883,13 @@ Relay summary objects contain the following key-value pairs: -n +n updated string -optional +required # Relay nickname consisting of 119 alphanumerical characters. -Omitted if the relay nickname is "Unnamed". +Turned into required field on March 14, 2018. @@ -936,13 +946,13 @@ Bridge summary objects contain the following key-value pairs: -n +n updated string -optional +required # Bridge nickname consisting of 119 alphanumerical characters. -Omitted if the bridge nickname is "Unnamed". +Turned into required field on March 14, 2018. @@ -1001,13 +1011,13 @@ Relay details objects contain the following key-value pairs: -nickname +nickname updated string -optional +required # Relay nickname consisting of 119 alphanumerical characters. -Omitted if the relay nickname is "Unnamed". +Turned into required field on March 14, 2018. @@ -1481,7 +1491,7 @@ the relay did not report which version it runs. -effective_family +effective_family updated array of strings optional # @@ -1490,8 +1500,10 @@ Array of fingerprints of relays that are in an effective, mutual family relationship with this relay. These relays are part of this relay's family and they consider this relay to be part of their family. -Omitted if empty or if descriptor containing this information cannot be -found. +Always contains the relay's own fingerprint. +Omitted if the descriptor containing this information cannot be found. +Updated to always include the relay's own fingerprint on +March 14, 2018. @@ -1646,14 +1658,13 @@ Bridge details objects contain the following key-value pairs: -nickname +nickname updated string -optional +required # -Bridge nickname consisting of 119 -alphanumerical characters. -Omitted if the bridge nickname is "Unnamed". +Bridge nickname consisting of 119 alphanumerical characters. +Turned into required field on March 14, 2018. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Cut off too recent dates from several CSV files.
commit dc50c9ec1e58698d7a97553203ccd6c28c043f67
Author: Karsten Loesing
Date: Tue Mar 6 17:27:15 2018 +0100
Cut off too recent dates from several CSV files.
With this patch we're cutting off the last (1) day from:
- servers.csv (2 -> 1)
- ipv6servers.csv (2 -> 1)
Before this patch we already cut off the last (1) day from:
- advbwdist.csv (1 -> 1)
With this patch we're cutting off the last three (3) days from:
- bandwidth.csv (4 -> 3)
With this patch we're cutting off the last two (2) days from:
- clients.csv (0 -> 2)
- userstats-combined.csv (0 -> 2)
- connbidirect2.csv (0 -> 2)
- torperf-1.1.csv (0 -> 2)
With the previous patch we started cutting off the last two (2) days
from:
- hidserv.csv (0 -> 2)
In a future patch we might start cutting off days from:
- webstats.csv (0 -> ?)
Fixes more of #25196.
---
.../metrics/stats/connbidirect/Main.java | 5 +++
src/main/sql/clients/init-userstats.sql| 6
src/main/sql/ipv6servers/init-ipv6servers.sql | 2 +-
src/main/sql/legacy/tordir.sql | 36 +++---
src/main/sql/onionperf/init-onionperf.sql | 2 ++
5 files changed, 32 insertions(+), 19 deletions(-)
diff --git a/src/main/java/org/torproject/metrics/stats/connbidirect/Main.java
b/src/main/java/org/torproject/metrics/stats/connbidirect/Main.java
index 1911472..4474966 100644
--- a/src/main/java/org/torproject/metrics/stats/connbidirect/Main.java
+++ b/src/main/java/org/torproject/metrics/stats/connbidirect/Main.java
@@ -429,6 +429,8 @@ public class Main {
SortedSet rawStats) {
DateFormat dateFormat = new SimpleDateFormat("-MM-dd");
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
+String yesterday = dateFormat.format(System.currentTimeMillis()
+- ONE_DAY_IN_MILLIS);
SortedMap> fractionsByDateAndDirection
= new TreeMap<>();
final String[] directions = new String[] { "read", "write", "both" };
@@ -436,6 +438,9 @@ public class Main {
if (rawStat.fingerprint != null) {
String date = dateFormat.format(rawStat.dateDays
* ONE_DAY_IN_MILLIS);
+if (date.compareTo(yesterday) >= 0) {
+ continue;
+}
short[] fractions = new short[] { rawStat.fractionRead,
rawStat.fractionWrite, rawStat.fractionBoth };
for (int i = 0; i < directions.length; i++) {
diff --git a/src/main/sql/clients/init-userstats.sql
b/src/main/sql/clients/init-userstats.sql
index 733a516..38521f2 100644
--- a/src/main/sql/clients/init-userstats.sql
+++ b/src/main/sql/clients/init-userstats.sql
@@ -681,6 +681,9 @@ CREATE OR REPLACE VIEW estimated AS SELECT
-- request statistics.
WHERE a.frac BETWEEN 0.1 AND 1.0
+ -- Skip estimates that are as recent as yesterday or newer.
+ AND a.date < current_date - 1
+
-- Order results.
ORDER BY date DESC, node, version, transport, country;
@@ -749,6 +752,9 @@ CREATE OR REPLACE VIEW combined AS SELECT
-- request statistics.
WHERE a.frac BETWEEN 0.1 AND 1.0
+ -- Skip estimates that are as recent as yesterday or newer.
+ AND a.date < current_date - 1
+
-- Order results.
ORDER BY date DESC;
diff --git a/src/main/sql/ipv6servers/init-ipv6servers.sql
b/src/main/sql/ipv6servers/init-ipv6servers.sql
index a5d56f3..8ed9372 100644
--- a/src/main/sql/ipv6servers/init-ipv6servers.sql
+++ b/src/main/sql/ipv6servers/init-ipv6servers.sql
@@ -99,7 +99,7 @@ WITH included_statuses AS (
GROUP BY DATE(valid_after), server
HAVING COUNT(status_id) >= 12
AND DATE(valid_after)
-< (SELECT MAX(DATE(valid_after)) FROM included_statuses) - 1
+< (SELECT MAX(DATE(valid_after)) FROM included_statuses)
)
SELECT DATE(valid_after) AS valid_after_date, server,
CASE WHEN server = 'relay' THEN guard_relay ELSE NULL END AS guard_relay,
diff --git a/src/main/sql/legacy/tordir.sql b/src/main/sql/legacy/tordir.sql
index e0ea651..5d254f6 100644
--- a/src/main/sql/legacy/tordir.sql
+++ b/src/main/sql/legacy/tordir.sql
@@ -720,7 +720,7 @@ CREATE VIEW stats_servers AS
(SELECT date, NULL AS flag, NULL AS country, NULL AS version,
NULL AS platform, TRUE AS ec2bridge, NULL AS relays,
avg_running_ec2 AS bridges FROM bridge_network_size
- WHERE date < current_date - 1)
+ WHERE date < current_date)
UNION ALL
(SELECT COALESCE(network_size.date, bridge_network_size.date) AS date,
NULL AS flag, NULL AS country, NULL AS version, NULL AS platform,
@@ -729,57 +729,57 @@ UNION ALL
FULL OUTER JOIN bridge_network_size
ON network_size.date = bridge_network_size.date
WHERE COALESCE(network_size.date, bridge_network_size.date) <
- current_date - 1)
+ current_date)
UNION ALL
(SELECT date, 'Exit' AS flag, NULL AS country, NULL AS version,
NULL AS platform, NULL AS ec2bridge, avg_exit AS relays,
- NULL AS bridges FROM network_size WHERE date <
[tor-commits] [metrics-web/release] Adds generate-news-json Ant task
commit f70408df045bd1343d300647a64bbe9dd71cdde7
Author: Iain R. Learmonth
Date: Tue Mar 20 14:23:01 2018 +
Adds generate-news-json Ant task
This commit updates UpdateNews to use the existing News class and to
accept the output filename as an argument. It also enables pretty
printing the output to remove the additional manual step of piping the
output through jq. (See #23854 comment:9)
The new Ant task requires that the JAR be built prior to running. It
will update news.json in place.
---
build.xml | 14 +++
src/main/java/org/torproject/metrics/web/News.java | 45 ++
.../org/torproject/metrics/web/UpdateNews.java | 19 +++--
3 files changed, 49 insertions(+), 29 deletions(-)
diff --git a/build.xml b/build.xml
index 59f3720..e98757e 100644
--- a/build.xml
+++ b/build.xml
@@ -235,6 +235,20 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/src/main/java/org/torproject/metrics/web/News.java
b/src/main/java/org/torproject/metrics/web/News.java
index f21d4e3..ee1b09f 100644
--- a/src/main/java/org/torproject/metrics/web/News.java
+++ b/src/main/java/org/torproject/metrics/web/News.java
@@ -9,21 +9,21 @@ import java.util.TreeMap;
public class News {
- private String start;
+ String start;
- private String end;
+ String end;
- private boolean ongoing;
+ Boolean ongoing;
- private List places;
+ List places;
- private String[] protocols;
+ List protocols;
- private String description;
+ String description;
- private String[] links;
+ List links;
- private boolean unknown;
+ Boolean unknown;
String getStart() {
return this.start;
@@ -33,8 +33,16 @@ public class News {
return this.end;
}
+ /**
+ * Returns whether or not the event is ongoing. If no value was set, it is
+ * assumed that the event is not ongoing.
+ */
boolean getOngoing() {
-return this.ongoing;
+if (this.ongoing != null) {
+ return this.ongoing;
+} else {
+ return false;
+}
}
List getPlaces() {
@@ -42,7 +50,7 @@ public class News {
}
String[] getProtocols() {
-return this.protocols;
+return (String[]) this.protocols.toArray();
}
String getDescription() {
@@ -50,11 +58,19 @@ public class News {
}
String[] getLinks() {
-return this.links;
+return (String[]) this.links.toArray();
}
+ /**
+ * Returns whether or not the reason for an event is known. If no value was
+ * set, it is assumed that the reason is known.
+ */
boolean isUnknown() {
-return this.unknown;
+if (this.unknown != null) {
+ return this.unknown;
+} else {
+ return false;
+}
}
static SortedMap countries;
@@ -72,7 +88,7 @@ public class News {
if (null == this.start) {
/* Invalid event without start date. */
sb.append("N/A");
-} else if (this.ongoing) {
+} else if (this.getOngoing()) {
/* Ongoing event. */
sb.append(this.start).append(" to present");
} else if (null == this.end || this.start.equals(this.end)) {
@@ -117,7 +133,7 @@ public class News {
}
}
}
-if (this.unknown) {
+if (this.isUnknown()) {
sb.append(" Unknown");
}
sb.append("");
@@ -144,4 +160,3 @@ public class News {
return sb.toString();
}
}
-
diff --git a/src/main/java/org/torproject/metrics/web/UpdateNews.java
b/src/main/java/org/torproject/metrics/web/UpdateNews.java
index 527988f..3e23233 100644
--- a/src/main/java/org/torproject/metrics/web/UpdateNews.java
+++ b/src/main/java/org/torproject/metrics/web/UpdateNews.java
@@ -11,23 +11,15 @@ import java.util.ArrayList;
import java.util.List;
public class UpdateNews {
- private static class News {
-String start;
-String end;
-Boolean ongoing;
-List places;
-List protocols;
-String description;
-List links;
-Boolean unknown;
- }
-
/** Update news. */
public static void main(String[] args) throws Exception {
URL textFile = new URL(
"https://trac.torproject.org/projects/tor/wiki/doc/;
+ "MetricsTimeline?format=txt");
-Gson gson = new GsonBuilder().disableHtmlEscaping().create();
+Gson gson = new GsonBuilder()
+.disableHtmlEscaping()
+.setPrettyPrinting()
+.create();
List news = new ArrayList<>();
try (BufferedReader br = new BufferedReader(new InputStreamReader(
textFile.openStream( {
@@ -123,8 +115,7 @@ public class UpdateNews {
news.add(entry);
}
}
-try (FileWriter fw = new FileWriter(
-"src/main/resources/web/json/news-raw.json")) {
+try (FileWriter fw = new FileWriter(args[0])) {
fw.write(gson.toJson(news));
}
}
___
tor-commits mailing list
tor-commits@lists.torproject.org
[tor-commits] [metrics-web/release] Update news.json to version 237 of doc/MetricsTimeline.
commit 1cba0348e912c6f8e9b4f6620384e4edd0247787
Author: Karsten Loesing
Date: Mon Mar 12 16:50:25 2018 +0100
Update news.json to version 237 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 56 ++-
1 file changed, 55 insertions(+), 1 deletion(-)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 2cd27c6..6beb053 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -2492,6 +2492,7 @@
"description": "Outage of the bridge authority https://metrics.torproject.org/rs.html#details/1D8F3A91C37C5D1C4C19B1AD1D0CFBE8BF72D8E1\;>Bifroest.",
"links": [
"https://lists.torproject.org/pipermail/metrics-team/2017-August/000441.html\;>mailing
list post",
+ "https://lists.torproject.org/pipermail/metrics-team/2018-March/000713.html\;>mailing
list post",
"https://metrics.torproject.org/networksize.html?start=2017-07-01=2017-08-27\;>graph"
]
},
@@ -2541,6 +2542,18 @@
},
{
"start": "2017-09-15",
+"end": "2017-09-16",
+"protocols": [
+ "bridge"
+],
+"description": "The bridge authority was down due to an issue with its
offline master key.",
+"links": [
+ "https://metrics.torproject.org/networksize.html?start=2017-07-01=2018-01-01\;>graph",
+ "https://lists.torproject.org/pipermail/metrics-team/2018-March/000713.html\;>mailing
list post"
+]
+ },
+ {
+"start": "2017-09-15",
"end": "2017-10-12",
"description": "Outage of the op-hk OnionPerf instance. Reported timeouts
on the tor network increased, download speed improved.",
"links": [
@@ -2694,6 +2707,18 @@
]
},
{
+"start": "2017-12-08",
+"end": "2017-12-11",
+"protocols": [
+ "bridge"
+],
+"description": "The bridge authority was down for unknown reasons.",
+"links": [
+ "https://metrics.torproject.org/networksize.html?start=2017-07-01=2018-01-01\;>graph",
+ "https://lists.torproject.org/pipermail/metrics-team/2018-March/000713.html\;>mailing
list post"
+]
+ },
+ {
"start": "2017-12-12",
"end": "2018-01-18",
"protocols": [
@@ -2820,7 +2845,7 @@
},
{
"start": "2018-02-27",
-"ongoing": true,
+"end": "2018-03-02",
"description": "Less than 3 bandwidth authorities are available.",
"links": [
"https://lists.torproject.org/pipermail/tor-relays/2018-February/014676.html\;>tor-relays"
@@ -3473,5 +3498,34 @@
"https://metrics.torproject.org/userstats-relay-country.html?start=2017-10-01=2018-03-01=fr=off\;>graph"
],
"unknown": true
+ },
+ {
+"start": "2017-12-16",
+"end": "2017-12-17",
+"protocols": [
+ "bridge"
+],
+"description": "Drop in the number of measured bridges.",
+"links": [
+ "https://metrics.torproject.org/networksize.html?start=2017-07-01=2018-01-01\;>graph",
+ "https://lists.torproject.org/pipermail/metrics-team/2018-March/000713.html\;>mailing
list post"
+],
+"unknown": true
+ },
+ {
+"start": "2018-03-05",
+"end": "2018-03-08",
+"places": [
+ "de"
+],
+"protocols": [
+ "",
+ "relay"
+],
+"description": "Steady drop in the number of relay users in Germany, from
1.5M to 650k. (Drop coincides with the release of tor 0.3.2.10, 0.3.1.10, and
0.2.9.15, which offer better resistance against DoS attacks; as well as
improvements in speed for downloads from onions https://metrics.torproject.org/torperf.html?start=2017-12-10=2018-03-10=all=onion=5mb\;>graph;
see this https://metrics.torproject.org/connbidirect.html\;>graph as well.
More datapoints: DDoS of the network happened https://lists.torproject.org/pipermail/tor-project/2017-December/001604.html\;>around
the time that saw an increase in the number of relay users in Germany, see
entry above or https://metrics.torproject.org/userstats-relay-country.html?start=2017-12-11=2017-12-22=de=off\;>graph)",
+"links": [
+ "https://metrics.torproject.org/userstats-relay-country.html?start=2017-11-10=2018-03-10=de=off\;>graph"
+],
+"unknown": true
}
]
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Make all graph data available as CSV.
commit b75358eace4e8e0a75c60fe0ea1814aa479ecb9f
Author: Karsten Loesing
Date: Wed Feb 28 20:55:10 2018 +0100
Make all graph data available as CSV.
Previously, we provided links to CSV files that graphs are based on.
But in some cases it would require some data wrangling to obtain the
data in the graph, which is less usable than it could be. Now we're
generating CSV files based on the graph and selected parameters. This
will enable users to quickly obtain the data in a graph and further
process it using tools of their choice.
Implements #25382.
---
src/main/R/rserver/graphs.R| 562 +
.../torproject/metrics/web/GraphImageServlet.java | 3 +-
.../org/torproject/metrics/web/GraphServlet.java | 1 -
.../org/torproject/metrics/web/LinkServlet.java| 1 -
.../java/org/torproject/metrics/web/Metric.java| 6 -
.../org/torproject/metrics/web/MetricServlet.java | 5 -
.../torproject/metrics/web/RObjectGenerator.java | 9 +-
.../org/torproject/metrics/web/TableServlet.java | 1 -
src/main/resources/web.xml | 35 +-
src/main/resources/web/json/metrics.json | 149 +-
src/main/resources/web/jsps/graph.jsp | 10 +-
src/main/resources/web/jsps/table.jsp | 9 -
12 files changed, 533 insertions(+), 258 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index ab1a60d..fd5201d 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -328,7 +328,7 @@ stats_dir =
"/srv/metrics.torproject.org/metrics/shared/stats/"
rdata_dir = "/srv/metrics.torproject.org/metrics/shared/RData/"
-plot_networksize <- function(start, end, path) {
+prepare_networksize <- function(start, end) {
s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
@@ -336,6 +336,11 @@ plot_networksize <- function(start, end, path) {
s$ec2bridge == '', ]
s <- data.frame(date = as.Date(s$date, "%Y-%m-%d"), relays = s$relays,
bridges = s$bridges)
+ s
+}
+
+plot_networksize <- function(start, end, path) {
+ s <- prepare_networksize(start, end)
dates <- seq(from = as.Date(start, "%Y-%m-%d"),
to = as.Date(end, "%Y-%m-%d"), by="1 day")
missing <- setdiff(dates, as.Date(s$date, origin = "1970-01-01"))
@@ -356,7 +361,12 @@ plot_networksize <- function(start, end, path) {
ggsave(filename = path, width = 8, height = 5, dpi = 150)
}
-plot_versions <- function(start, end, path) {
+write_networksize <- function(start, end, path) {
+ prepare_networksize(start, end) %>%
+write.csv(path, quote = FALSE, row.names = FALSE)
+}
+
+prepare_versions <- function(start, end) {
s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
@@ -364,6 +374,11 @@ plot_versions <- function(start, end, path) {
s$ec2bridge == '', ]
s <- data.frame(date = as.Date(s$date, "%Y-%m-%d"), version = s$version,
relays = s$relays)
+ s
+}
+
+plot_versions <- function(start, end, path) {
+ s <- prepare_versions(start, end)
known_versions <- c("Other", "0.1.0", "0.1.1", "0.1.2", "0.2.0",
"0.2.1", "0.2.2", "0.2.3", "0.2.4", "0.2.5", "0.2.6", "0.2.7",
"0.2.8", "0.2.9", "0.3.0", "0.3.1", "0.3.2", "0.3.3")
@@ -388,14 +403,26 @@ plot_versions <- function(start, end, path) {
ggsave(filename = path, width = 8, height = 5, dpi = 150)
}
-plot_platforms <- function(start, end, path) {
+write_versions <- function(start, end, path) {
+ prepare_versions(start, end) %>%
+spread(key = "version", value = "relays", fill = 0) %>%
+write.csv(path, quote = FALSE, row.names = FALSE)
+}
+
+prepare_platforms <- function(start, end) {
s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
s$country == '' & s$version == '' & s$platform != '' &
s$ec2bridge == '', ]
platforms <- data.frame(date = as.Date(s$date, "%Y-%m-%d"),
- variable = s$platform, value = s$relays)
+ variable = ifelse(s$platform == "Darwin", "macOS", s$platform),
+ value = s$relays)
+ platforms
+}
+
+plot_platforms <- function(start, end, path) {
+ platforms <- prepare_platforms(start, end)
ggplot(platforms, aes(x = as.Date(date, "%Y-%m-%d"), y = value,
colour = variable)) +
geom_line() +
@@ -403,25 +430,35 @@ plot_platforms <- function(start, end, path) {
labels = custom_labels, minor_breaks = custom_minor_breaks) +
scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) +
scale_colour_manual(name = "Platform",
- breaks = c("Linux", "Darwin", "BSD", "Windows", "Other"),
- labels = c("Linux", "macOS", "BSD", "Windows",
[tor-commits] [metrics-web/release] Add comment section to CSV file headers.
commit b8a0bfbbe8a6f784156c69e3881b5542eec43717
Author: Karsten Loesing
Date: Mon Mar 5 16:00:13 2018 +0100
Add comment section to CSV file headers.
Implements #25387, though without legend.
Also fix MIME type from image/csv to text/csv.
---
.../org/torproject/metrics/web/GraphImageServlet.java | 19 ---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/src/main/java/org/torproject/metrics/web/GraphImageServlet.java
b/src/main/java/org/torproject/metrics/web/GraphImageServlet.java
index f24d698..1c5a8ca 100644
--- a/src/main/java/org/torproject/metrics/web/GraphImageServlet.java
+++ b/src/main/java/org/torproject/metrics/web/GraphImageServlet.java
@@ -64,13 +64,26 @@ public class GraphImageServlet extends HttpServlet {
}
/* Write graph bytes to response. */
-response.setContentType("image/" + fileType);
-response.setHeader("Content-Length",
-String.valueOf(graph.getBytes().length));
+byte[] headerBytes = ("#\n"
++ "# The Tor Project\n"
++ "#\n"
++ "# URL: https://metrics.torproject.org;
+ + request.getRequestURI()
+ + (null == request.getQueryString() ? ""
+ : "?" + request.getQueryString()) + "\n"
++ "#\n").getBytes();
+response.setContentType(
+("csv".equals(fileType) ? "text/" : "image/") + fileType);
+response.setHeader("Content-Length", String.valueOf(
+("csv".equals(fileType) ? headerBytes.length : 0)
++ graph.getBytes().length));
response.setHeader("Content-Disposition",
"inline; filename=\"" + graph.getFileName() + "\"");
BufferedOutputStream output = new BufferedOutputStream(
response.getOutputStream(), 1024);
+if ("csv".equals(fileType)) {
+ output.write(headerBytes);
+}
output.write(graph.getBytes(), 0, graph.getBytes().length);
output.flush();
output.close();
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Adds RelaySearchServlet and accompanying JSP (Fixes: #25258)
commit 9b42e1712595546144fe56e3e0afdc58388e492c
Author: Iain R. Learmonth
Date: Thu Feb 15 13:00:57 2018 +
Adds RelaySearchServlet and accompanying JSP (Fixes: #25258)
---
.../torproject/metrics/web/RelaySearchServlet.java | 33
src/main/resources/web/jsps/rs.jsp | 44 ++
2 files changed, 77 insertions(+)
diff --git a/src/main/java/org/torproject/metrics/web/RelaySearchServlet.java
b/src/main/java/org/torproject/metrics/web/RelaySearchServlet.java
new file mode 100644
index 000..9a235ce
--- /dev/null
+++ b/src/main/java/org/torproject/metrics/web/RelaySearchServlet.java
@@ -0,0 +1,33 @@
+/* Copyright 2011--2018 The Tor Project
+ * See LICENSE for licensing information */
+
+package org.torproject.metrics.web;
+
+import java.io.IOException;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+public class RelaySearchServlet extends AnyServlet {
+
+ private static final long serialVersionUID = 1L;
+
+ @Override
+ public void init() throws ServletException {
+super.init();
+ }
+
+ @Override
+ public void doGet(HttpServletRequest request,
+ HttpServletResponse response) throws IOException, ServletException {
+
+/* Forward the request to the JSP that does all the hard work. */
+String[] additionalStylesheets = {"/rs/css/atlas.css"};
+request.setAttribute("additionalStylesheets", additionalStylesheets);
+request.setAttribute("categories", this.categories);
+request.getRequestDispatcher("WEB-INF/rs.jsp").forward(request,
+response);
+ }
+}
+
diff --git a/src/main/resources/web/jsps/rs.jsp
b/src/main/resources/web/jsps/rs.jsp
new file mode 100644
index 000..22a8377
--- /dev/null
+++ b/src/main/resources/web/jsps/rs.jsp
@@ -0,0 +1,44 @@
+<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core; %>
+<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions; %>
+
+
+
+
+
+
+
+Home
+Services
+Relay Search
+
+
+
+
+
+
+
+
+
+
+
+ Relay Search
+
+Rendering results...
+
+
+
+
+ JavaScript required
+ Please enable JavaScript to use this service. If you are using Tor
Browser on High Security mode, it is possible to enable JavaScript to run only
on this page. Click the NoScript icon on your
address bar and select "Temporarily allow all on this page". Relay Search only
uses JavaScript resources that are hosted by the Tor Metrics team.
+
+
+
+
+
+ var require = {
+urlArgs: "v29"
+ };
+
+
+
+
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Refactor even more of the graphing code.
commit 5c8e727b61f516ddb30d2400d6f3c97462443801
Author: Karsten Loesing
Date: Thu Mar 1 12:27:07 2018 +0100
Refactor even more of the graphing code.
---
src/main/R/rserver/graphs.R | 428 +---
1 file changed, 167 insertions(+), 261 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 8847967..93ff919 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -329,28 +329,20 @@ stats_dir =
"/srv/metrics.torproject.org/metrics/shared/stats/"
rdata_dir = "/srv/metrics.torproject.org/metrics/shared/RData/"
prepare_networksize <- function(start, end) {
- s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
-stringsAsFactors = FALSE)
- s <- s[s$date >= start & s$date <= end & s$flag == '' &
- s$country == '' & s$version == '' & s$platform == '' &
- s$ec2bridge == '', ]
- s <- data.frame(date = as.Date(s$date, "%Y-%m-%d"), relays = s$relays,
- bridges = s$bridges)
- s
+ read.csv(paste(stats_dir, "servers.csv", sep = ""),
+colClasses = c("date" = "Date")) %>%
+filter(date >= as.Date(start), date <= as.Date(end), flag == '',
+ country == '', version == '', platform == '', ec2bridge == '') %>%
+select(date, relays, bridges)
}
plot_networksize <- function(start, end, path) {
- s <- prepare_networksize(start, end)
- dates <- seq(from = as.Date(start, "%Y-%m-%d"),
- to = as.Date(end, "%Y-%m-%d"), by="1 day")
- missing <- setdiff(dates, as.Date(s$date, origin = "1970-01-01"))
- if (length(missing) > 0)
-s <- rbind(s,
-data.frame(date = as.Date(missing, origin = "1970-01-01"),
-relays = NA, bridges = NA))
- networksize <- gather(s, variable, value, -date)
- ggplot(networksize, aes(x = as.Date(date, "%Y-%m-%d"), y = value,
-colour = variable)) + geom_line() +
+ prepare_networksize(start, end) %>%
+gather(variable, value, -date) %>%
+complete(date = full_seq(date, period = 1),
+ variable = c("relays", "bridges")) %>%
+ggplot(aes(x = date, y = value, colour = variable)) +
+geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) +
@@ -367,14 +359,11 @@ write_networksize <- function(start, end, path) {
}
prepare_versions <- function(start, end) {
- s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
-stringsAsFactors = FALSE)
- s <- s[s$date >= start & s$date <= end & s$flag == '' &
- s$country == '' & s$version != '' & s$platform == '' &
- s$ec2bridge == '', ]
- s <- data.frame(date = as.Date(s$date, "%Y-%m-%d"), version = s$version,
- relays = s$relays)
- s
+ read.csv(paste(stats_dir, "servers.csv", sep = ""),
+colClasses = c("date" = "Date")) %>%
+filter(date >= as.Date(start), date <= as.Date(end), flag == '',
+ country == '', version != '', platform == '', ec2bridge == '') %>%
+select(date, version, relays)
}
plot_versions <- function(start, end, path) {
@@ -389,8 +378,7 @@ plot_versions <- function(start, end, path) {
stringsAsFactors = FALSE)
versions <- s[s$version %in% known_versions, ]
visible_versions <- sort(unique(versions$version))
- ggplot(versions, aes(x = as.Date(date, "%Y-%m-%d"), y = relays,
- colour = version)) +
+ ggplot(versions, aes(x = date, y = relays, colour = version)) +
geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
@@ -410,21 +398,18 @@ write_versions <- function(start, end, path) {
}
prepare_platforms <- function(start, end) {
- s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
-stringsAsFactors = FALSE)
- s <- s[s$date >= start & s$date <= end & s$flag == '' &
- s$country == '' & s$version == '' & s$platform != '' &
- s$ec2bridge == '', ]
- platforms <- data.frame(date = as.Date(s$date, "%Y-%m-%d"),
- variable = ifelse(s$platform == "Darwin", "macOS", s$platform),
- value = s$relays)
- platforms
+ read.csv(paste(stats_dir, "servers.csv", sep = ""),
+colClasses = c("date" = "Date")) %>%
+filter(date >= as.Date(start), date <= as.Date(end), flag == '',
+ country == '', version == '', platform != '', ec2bridge == '') %>%
+select(date, platform, relays) %>%
+mutate(platform = ifelse(platform == "Darwin", "macOS",
+ as.character(platform)))
}
plot_platforms <- function(start, end, path) {
- platforms <- prepare_platforms(start, end)
- ggplot(platforms, aes(x = as.Date(date, "%Y-%m-%d"), y = value,
- colour = variable)) +
+ prepare_platforms(start, end) %>%
+ggplot(aes(x = date, y = relays, colour = platform)) +
geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
[tor-commits] [metrics-web/release] Cut off too recent dates from hidserv.csv.
commit f810b3f8259750a68977f38fe2e71e6c48794816
Author: Karsten Loesing
Date: Mon Feb 12 15:17:16 2018 +0100
Cut off too recent dates from hidserv.csv.
In commit 8831ece we stopped cutting off data from CSV files when
graphing the data.
With this commit we're cuting off the last two days from hidserv.csv.
Fixes #25196.
---
src/main/java/org/torproject/metrics/stats/hidserv/Aggregator.java | 7 +++
1 file changed, 7 insertions(+)
diff --git a/src/main/java/org/torproject/metrics/stats/hidserv/Aggregator.java
b/src/main/java/org/torproject/metrics/stats/hidserv/Aggregator.java
index a57fde5..5cafb1b 100644
--- a/src/main/java/org/torproject/metrics/stats/hidserv/Aggregator.java
+++ b/src/main/java/org/torproject/metrics/stats/hidserv/Aggregator.java
@@ -64,6 +64,10 @@ public class Aggregator {
return;
}
+/* Exclude any statistics being as recent as yesterday or newer. */
+String yesterday = DateTimeHelper.format(System.currentTimeMillis()
+- DateTimeHelper.ONE_DAY, DateTimeHelper.ISO_DATE_FORMAT);
+
/* Re-arrange extrapolated network totals by statistics interval end
* date, and include the computed network total as weight for the
* extrapolated value. More precisely, map keys are ISO-formatted
@@ -76,6 +80,9 @@ public class Aggregator {
String date = DateTimeHelper.format(
extrapolated.getStatsDateMillis(),
DateTimeHelper.ISO_DATE_FORMAT);
+ if (date.compareTo(yesterday) >= 0) {
+continue;
+ }
if (extrapolated.getFractionRendRelayedCells() > 0.0) {
if (!extrapolatedCells.containsKey(date)) {
extrapolatedCells.put(date, new ArrayList<>());
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Adds a servlet mapping for RelaySearchServlet (See: #25258)
commit e1035d8af074797c37bc5acc276a59a1c795ee8d Author: Iain R. Learmonth Date: Thu Feb 15 13:01:32 2018 + Adds a servlet mapping for RelaySearchServlet (See: #25258) --- src/main/resources/web.xml | 11 +++ 1 file changed, 11 insertions(+) diff --git a/src/main/resources/web.xml b/src/main/resources/web.xml index e6e6a69..99df1db 100644 --- a/src/main/resources/web.xml +++ b/src/main/resources/web.xml @@ -324,6 +324,17 @@ /research.html + +RelaySearch + + org.torproject.metrics.web.RelaySearchServlet + + + +RelaySearch +/rs.html + + index.html ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 231 of doc/MetricsTimeline.
commit ed1a21d7892729d3d02219d438dba7e18dab0500
Author: Karsten Loesing
Date: Wed Mar 7 20:42:18 2018 +0100
Update news.json to version 231 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 24
1 file changed, 24 insertions(+)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 4c6d2cf..7036b7a 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -2819,6 +2819,14 @@
"description": "Tor 0.3.3.2-alpha reaches deb.torproject.org repositories"
},
{
+"start": "2018-02-27",
+"ongoing": true,
+"description": "Less than 3 bandwidth authorities are available.",
+"links": [
+ "https://lists.torproject.org/pipermail/tor-relays/2018-February/014676.html\;>tor-relays"
+]
+ },
+ {
"start": "2016-02-24",
"places": [
"tm"
@@ -3441,5 +3449,21 @@
"https://www.reddit.com/r/TOR/comments/7ldrox/tor_usage_in_germany_what_happend/\;>reddit
thread"
],
"unknown": true
+ },
+ {
+"start": "2017-12-12",
+"end": "2018-01-10",
+"places": [
+ "fr"
+],
+"protocols": [
+ "",
+ "relay"
+],
+"description": "Temporary tripling of relay users in France. Bridges not
affected.",
+"links": [
+ "https://metrics.torproject.org/userstats-relay-country.html?start=2017-10-01=2018-03-01=fr=off\;>graph"
+],
+"unknown": true
}
]
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Remove quote from header.
commit ad2a4c2c190156ced859dc277e8591881d672e29
Author: Karsten Loesing
Date: Mon Mar 5 14:17:58 2018 +0100
Remove quote from header.
Fixes #25411.
---
src/main/resources/web/css/style.css | 3 +++
src/main/resources/web/jsps/top.jsp | 5 -
2 files changed, 3 insertions(+), 5 deletions(-)
diff --git a/src/main/resources/web/css/style.css
b/src/main/resources/web/css/style.css
index 97e91b8..108f95c 100644
--- a/src/main/resources/web/css/style.css
+++ b/src/main/resources/web/css/style.css
@@ -125,6 +125,9 @@ body {
}
}
+.navbar-secondary {
+min-height:0px;
+}
.navbar-secondary .section-header {
padding:3px 14px;
text-transform:uppercase;
diff --git a/src/main/resources/web/jsps/top.jsp
b/src/main/resources/web/jsps/top.jsp
index 146c45a..440b2ee 100644
--- a/src/main/resources/web/jsps/top.jsp
+++ b/src/main/resources/web/jsps/top.jsp
@@ -106,11 +106,6 @@ document.write('
-
-
- Tor metrics are the ammunition that lets Tor and other
security advocates argue for a more private and secure Internet from a position
of data, rather than just dogma or perspective. Bruce
Schneier (June 1, 2016)
-
-
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update URL for TorDNSEL.
commit 4c75d196127f6d316ae7d7aa76a4dea975a7ce13 Author: Karsten Loesing Date: Thu Mar 1 14:46:22 2018 +0100 Update URL for TorDNSEL. Fixes #25287. --- src/main/resources/web/jsps/collector.jsp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/main/resources/web/jsps/collector.jsp b/src/main/resources/web/jsps/collector.jsp index 92f6c7c..33ae7dd 100644 --- a/src/main/resources/web/jsps/collector.jsp +++ b/src/main/resources/web/jsps/collector.jsp @@ -566,7 +566,7 @@ archived. The exit list service -https://www.torproject.org/tordnsel/dist/; target="_blank">TorDNSEL +https://gitweb.torproject.org/tordnsel.git/tree/README; target="_blank">TorDNSEL publishes exit lists containing the IP addresses of relays that it found when exiting through them. @@ -579,7 +579,7 @@ when exiting through them. -Tor Check makes the list of known exits and corresponding exit IP +TorDNSEL makes the list of known exits and corresponding exit IP addresses available in a specific format. The document below shows an entry of the exit list written on December 28, 2010 at 15:21:44 UTC. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update to latest metrics-lib.
commit ff1fbdd2f208049df061c8449ba86ceda961be91 Author: Karsten Loesing Date: Thu Mar 1 14:43:32 2018 +0100 Update to latest metrics-lib. --- src/submods/metrics-lib | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index 9f2db9a..7648780 16 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit 9f2db9a194d5e290b456765f83cc6eeaf745e619 +Subproject commit 7648780844f809f3068d3e625973fe43c78ccbb3 ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Bumps copyright year to 2018 in common JSP footer
commit 187cb1e86808eda2555eee7ac25fe953452e766b Author: Iain R. Learmonth Date: Fri Mar 2 13:08:09 2018 + Bumps copyright year to 2018 in common JSP footer --- src/main/resources/web/jsps/bottom.jsp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/bottom.jsp b/src/main/resources/web/jsps/bottom.jsp index 6719adc..85db407 100644 --- a/src/main/resources/web/jsps/bottom.jsp +++ b/src/main/resources/web/jsps/bottom.jsp @@ -3,7 +3,7 @@ -20092017 https://www.torproject.org/; target="_blank">The Tor Project +20092018 https://www.torproject.org/; target="_blank">The Tor Project ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Adds an Ant target to generate PNGs using Graphviz
commit 252d4635df29db2a984f4bf75e0b0a8910e44370 Author: Iain R. Learmonth Date: Sat Mar 3 06:35:34 2018 + Adds an Ant target to generate PNGs using Graphviz --- build.xml | 15 +++ 1 file changed, 15 insertions(+) diff --git a/build.xml b/build.xml index 4b67ae6..59f3720 100644 --- a/build.xml +++ b/build.xml @@ -220,6 +220,21 @@ + + + + + + + + + + + + + ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Expand Philosophy text on about page (See: #23169)
commit 4a6b9deb39fe1011963066a78ee1512d20bad64f Author: Iain R. Learmonth Date: Thu Mar 1 14:21:06 2018 + Expand Philosophy text on about page (See: #23169) --- src/main/resources/web/jsps/about.jsp | 37 +++ 1 file changed, 37 insertions(+) diff --git a/src/main/resources/web/jsps/about.jsp b/src/main/resources/web/jsps/about.jsp index bc14b08..dbd1be9 100644 --- a/src/main/resources/web/jsps/about.jsp +++ b/src/main/resources/web/jsps/about.jsp @@ -24,6 +24,43 @@ through a rigorous review and discussion process before appearing here. We never publish statisticsor aggregate statisticsof sensitive data, such as unencrypted contents of traffic. + The goals of a privacy and anonymity network like Tor are not easily + combined with extensive data gathering, but at the same time data is needed for + monitoring, understanding, and improving the network. Data can be used to + detect possible censorship events or attacks against the network. Safety and + privacy concerns regarding data collection by Tor Metrics are guided by the https://research.torproject.org/safetyboard.html#guidelines;>Tor Research + Safety Board's guidelines. Safety and privacy assessment is usually done + openly by discussion during the proposal process for changes to the Tor source, + and/or supported by closer analysis in form of https://research.torproject.org/techreports.html;>Tor Technical + Reports. + For data we collect from the public Tor network, we will always follow + three main guidelines: + + Data minimalism + The first and most important guideline is that only the minimum + amount of statistical data should be gathered to solve a given problem. + The level of detail of measured data should be as small as possible. + Source aggregation + Possibly sensitive data should exist for as short a time + as possible. Data should be aggregated at its source, including + categorizing single events and memorizing category counts only, summing + up event counts over large time frames, and being imprecise regarding + exact event counts. + Transparency + All algorithms to gather statistical data need to be discussed + publicly before deploying them. All measured statistical data should be + made publicly available as a safeguard to not gather data that is too + sensitive. + + You can read more about safety considerations when collecting data in + the Tor network in "https://www.freehaven.net/anonbib/#wecsr10measuring-tor;>A Case + Study on Measuring Statistical Data in the Tor Anonymity Network" by + Karsten Loesing, Steven J. Murdoch, and Roger Dingledine. In the + Proceedings of the Workshop on Ethics in Computer Security Research + (WECSR 2010), Tenerife, Canary Islands, Spain, January 2010. Ecosystem # Tor relays and bridges collect aggregated statistics about their usage ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 228 of doc/MetricsTimeline.
commit 1fd568f6f687bcb0baf10e52b012d3cb846260f5
Author: Karsten Loesing
Date: Fri Feb 16 20:38:39 2018 +0100
Update news.json to version 228 of doc/MetricsTimeline.
This wiki page version only updates Relay Search links and doesn't add
any new content.
---
src/main/resources/web/json/news.json | 36 +--
1 file changed, 18 insertions(+), 18 deletions(-)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index c0d069e..4c6d2cf 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -1453,7 +1453,7 @@
"protocols": [
"meek"
],
-"description": "Established an unthrottled bridge https://atlas.torproject.org/#details/C20658946DD706A7A2181159A1A04CD838570D04\;>C20658946DD706A7A2181159A1A04CD838570D04
for people who set up their own meek CDN configuration."
+"description": "Established an unthrottled bridge https://metrics.torproject.org/rs.html#details/C20658946DD706A7A2181159A1A04CD838570D04\;>C20658946DD706A7A2181159A1A04CD838570D04
for people who set up their own meek CDN configuration."
},
{
"start": "2016-01-07",
@@ -1819,7 +1819,7 @@
"protocols": [
"obfs3"
],
-"description": "Default obfs3 bridges \"Unnamed\" and \"Unnamed\"
(fingerprint https://atlas.torproject.org/#details/AF9F66B7B04F8FF6F32D455F05135250A16543C9\;>AF9F66B7B04F8FF6F32D455F05135250A16543C9)
upgrade and begin reporting statistics to the new bridge authority Bifroest."
+"description": "Default obfs3 bridges \"Unnamed\" and \"Unnamed\"
(fingerprint https://metrics.torproject.org/rs.html#details/AF9F66B7B04F8FF6F32D455F05135250A16543C9\;>AF9F66B7B04F8FF6F32D455F05135250A16543C9)
upgrade and begin reporting statistics to the new bridge authority Bifroest."
},
{
"start": "2016-09-23",
@@ -1827,7 +1827,7 @@
"protocols": [
"obfs3"
],
-"description": "Outage of default obfs3 bridges \"Unnamed\" and
\"Unnamed\" (fingerprint https://atlas.torproject.org/#details/AF9F66B7B04F8FF6F32D455F05135250A16543C9\;>AF9F66B7B04F8FF6F32D455F05135250A16543C9).
(Start date not known for sure, though it must have been after 2016-09-23;
discussed in non-archived tor-team email.)"
+"description": "Outage of default obfs3 bridges \"Unnamed\" and
\"Unnamed\" (fingerprint https://metrics.torproject.org/rs.html#details/AF9F66B7B04F8FF6F32D455F05135250A16543C9\;>AF9F66B7B04F8FF6F32D455F05135250A16543C9).
(Start date not known for sure, though it must have been after 2016-09-23;
discussed in non-archived tor-team email.)"
},
{
"start": "2016-09-24",
@@ -2003,7 +2003,7 @@
"protocols": [
"obfs4"
],
-"description": "Default obfs4 bridge https://atlas.torproject.org/#details/D9C805C955CB124D188C0D44F271E9BE57DE2109\;>Lisbeth
turns on timing obfuscation (iat-mode=1).",
+"description": "Default obfs4 bridge https://metrics.torproject.org/rs.html#details/D9C805C955CB124D188C0D44F271E9BE57DE2109\;>Lisbeth
turns on timing obfuscation (iat-mode=1).",
"links": [
"https://bugs.torproject.org/20837\;>ticket"
]
@@ -2246,7 +2246,7 @@
"protocols": [
"snowflake"
],
-"description": "Set AssumeReachable 1 on the Snowflake bridge
https://atlas.torproject.org/#details/2B280B23E1107BB62ABFC40DDCC8824814F80A72\;>2B280B23E1107BB62ABFC40DDCC8824814F80A72
in an attempt to make it start publishing statistics.",
+"description": "Set AssumeReachable 1 on the Snowflake bridge
https://metrics.torproject.org/rs.html#details/2B280B23E1107BB62ABFC40DDCC8824814F80A72\;>2B280B23E1107BB62ABFC40DDCC8824814F80A72
in an attempt to make it start publishing statistics.",
"links": [
"https://lists.torproject.org/pipermail/tor-dev/2017-May/012243.html\;>mailing
list post"
]
@@ -2290,7 +2290,7 @@
"protocols": [
"obfs4"
],
-"description": "Set AssumeReachable 1 on the default obfs4
bridges https://atlas.torproject.org/#details/C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4\;>cymrubridge31
and https://atlas.torproject.org/#details/0BAC39417268B96B9F514E7F63FA6FBA1A788955\;>cymrubridge33
in an attempt to make them publish statistics.",
+"description": "Set AssumeReachable 1 on the default obfs4
bridges https://metrics.torproject.org/rs.html#details/C8CBDB2464FC9804A69531437BCF2BE31FDD2EE4\;>cymrubridge31
and https://metrics.torproject.org/rs.html#details/0BAC39417268B96B9F514E7F63FA6FBA1A788955\;>cymrubridge33
in an attempt to make them publish statistics.",
"links": [
"https://lists.torproject.org/pipermail/tor-dev/2017-May/012283.html\;>mailing
list post"
]
@@ -2301,7 +2301,7 @@
"ipv6",
"obfs4"
],
-"description": "Tor Browser 7.0 released. Adds an IPv6 address for default
obfs4 bridge Lisbeth. Adds new default obfs4 bridges https://atlas.torproject.org/#details/854173307E33686BBBAC36A3A093BEF320B719D4\;>frosty
[tor-commits] [metrics-web/release] Update Relay Search link on the Services page.
commit 78769f653bde8eb7aa5a372465c2d407dd17396f Author: Karsten Loesing Date: Fri Feb 16 09:36:15 2018 +0100 Update Relay Search link on the Services page. --- src/main/resources/web/jsps/services.jsp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/services.jsp b/src/main/resources/web/jsps/services.jsp index dcd296c..2283dbe 100644 --- a/src/main/resources/web/jsps/services.jsp +++ b/src/main/resources/web/jsps/services.jsp @@ -29,7 +29,7 @@ Network status # The following tools let you explore currently running relays and bridges. -https://atlas.torproject.org/;>Relay Search displays data about single relays and bridges in the Tor Network. +Relay Search displays data about single relays and bridges in the Tor Network. https://consensus-health.torproject.org/; target="_blank">Consensus Health displays information about the current directory consensus and votes. https://tormap.void.gr/; target="_blank">Tor Map displays an interactive map of Tor relays and provides KML files for relay locations. https://nusenu.github.io/OrNetStats/; target="_blank">OrNetStats displays statistics for monitoring diversity in the Tor network. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Include Relay Search assets in generated WAR (See: #25258)
commit f5a27ce3ae8fdbcbeebd978f1b7d1f55358327e2 Author: Iain R. Learmonth Date: Thu Feb 15 12:59:53 2018 + Include Relay Search assets in generated WAR (See: #25258) --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index 28a6a62..4b67ae6 100644 --- a/build.xml +++ b/build.xml @@ -179,7 +179,7 @@ includes="logback.xml" /> + includes="css/* images/* js/* fonts/** rs/**" /> ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Fix first batch of style issues found by lintr.
commit debef097e14c83523a9ae4e077bf1bb1dd945720
Author: Karsten Loesing
Date: Thu Mar 1 22:30:34 2018 +0100
Fix first batch of style issues found by lintr.
---
src/main/R/rserver/graphs.R | 218 ++--
1 file changed, 109 insertions(+), 109 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 93ff919..a31790d 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -308,7 +308,7 @@ custom_labels <- function(breaks) {
# Helper function to format numbers in non-scientific notation with spaces as
# thousands separator.
formatter <- function(x, ...) {
- format(x, ..., scientific = FALSE, big.mark = ' ')
+ format(x, ..., scientific = FALSE, big.mark = " ")
}
theme_update(
@@ -322,17 +322,17 @@ theme_update(
# Set the default line size of geom_line() to 1.
update_geom_defaults("line", list(size = 1))
-copyright_notice = "The Tor Project - https://metrics.torproject.org/;
+copyright_notice <- "The Tor Project - https://metrics.torproject.org/;
-stats_dir = "/srv/metrics.torproject.org/metrics/shared/stats/"
+stats_dir <- "/srv/metrics.torproject.org/metrics/shared/stats/"
-rdata_dir = "/srv/metrics.torproject.org/metrics/shared/RData/"
+rdata_dir <- "/srv/metrics.torproject.org/metrics/shared/RData/"
prepare_networksize <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), flag == '',
- country == '', version == '', platform == '', ec2bridge == '') %>%
+filter(date >= as.Date(start), date <= as.Date(end), flag == "",
+ country == "", version == "", platform == "", ec2bridge == "") %>%
select(date, relays, bridges)
}
@@ -361,8 +361,8 @@ write_networksize <- function(start, end, path) {
prepare_versions <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), flag == '',
- country == '', version != '', platform == '', ec2bridge == '') %>%
+filter(date >= as.Date(start), date <= as.Date(end), flag == "",
+ country == "", version != "", platform == "", ec2bridge == "") %>%
select(date, version, relays)
}
@@ -371,7 +371,7 @@ plot_versions <- function(start, end, path) {
known_versions <- c("Other", "0.1.0", "0.1.1", "0.1.2", "0.2.0",
"0.2.1", "0.2.2", "0.2.3", "0.2.4", "0.2.5", "0.2.6", "0.2.7",
"0.2.8", "0.2.9", "0.3.0", "0.3.1", "0.3.2", "0.3.3")
- getPalette = colorRampPalette(brewer.pal(12, "Paired"))
+ getPalette <- colorRampPalette(brewer.pal(12, "Paired"))
colours <- data.frame(breaks = known_versions,
values = rep(brewer.pal(min(12, length(known_versions)), "Paired"),
len = length(known_versions)),
@@ -400,8 +400,8 @@ write_versions <- function(start, end, path) {
prepare_platforms <- function(start, end) {
read.csv(paste(stats_dir, "servers.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), flag == '',
- country == '', version == '', platform != '', ec2bridge == '') %>%
+filter(date >= as.Date(start), date <= as.Date(end), flag == "",
+ country == "", version == "", platform != "", ec2bridge == "") %>%
select(date, platform, relays) %>%
mutate(platform = ifelse(platform == "Darwin", "macOS",
as.character(platform)))
@@ -431,8 +431,8 @@ write_platforms <- function(start, end, path) {
prepare_bandwidth <- function(start, end) {
read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), isexit == '',
- isguard == '') %>%
+filter(date >= as.Date(start), date <= as.Date(end), isexit == "",
+ isguard == "") %>%
mutate(advbw = advbw * 8 / 1e9) %>%
mutate(bwhist = (bwread + bwwrite) * 8 / 2e9) %>%
select(date, advbw, bwhist)
@@ -464,11 +464,11 @@ write_bandwidth <- function(start, end, path) {
prepare_bwhist_flags <- function(start, end) {
read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
colClasses = c("date" = "Date")) %>%
-filter(date >= as.Date(start), date <= as.Date(end), isexit != '',
- isguard != '') %>%
-mutate(variable = ifelse(isexit == 't',
-ifelse(isguard == 't', "guard_and_exit", "exit_only"),
-ifelse(isguard == 't', "guard_only", "middle_only")),
+filter(date >= as.Date(start), date <= as.Date(end), isexit != "",
+ isguard != "") %>%
+mutate(variable = ifelse(isexit == "t",
+ifelse(isguard == "t", "guard_and_exit", "exit_only"),
+ifelse(isguard == "t", "guard_only", "middle_only")),
value = (bwread + bwwrite) * 8 / 2e9) %>%
select(date, variable, value)
}
@@ -502,8 +502,8 @@ write_bwhist_flags <- function(start, end, path) {
[tor-commits] [metrics-web/release] Adds text about Metrics ecosystem to about page (See: #23169)
commit 28d5eada13f1f8b2bb509e280afba01da5770ad3
Author: Iain R. Learmonth
Date: Thu Mar 1 13:45:44 2018 +
Adds text about Metrics ecosystem to about page (See: #23169)
---
src/main/resources/graphviz/ecosystem.gv| 56 ++
src/main/resources/web/images/ecosystem.png | Bin 0 -> 73561 bytes
src/main/resources/web/jsps/about.jsp | 69
3 files changed, 115 insertions(+), 10 deletions(-)
diff --git a/src/main/resources/graphviz/ecosystem.gv
b/src/main/resources/graphviz/ecosystem.gv
new file mode 100644
index 000..c5cdd5c
--- /dev/null
+++ b/src/main/resources/graphviz/ecosystem.gv
@@ -0,0 +1,56 @@
+digraph ecosystem {
+node [fontname="Source Sans Pro"];
+subgraph cluster_network {
+label="Public Tor Network";
+fontname="Source Sans Pro Semi-Bold";
+node [color="#59316b",fillcolor="#e1d3e8",style="filled"];
+relays [label="Relays"];
+bridges [label="Bridges"];
+dirauths [label="Directory Authorities"];
+bridgeauth [label="Bridge Authority"];
+}
+subgraph cluster_metrics {
+label="Tor Metrics";
+fontname="Source Sans Pro Semi-Bold";
+labelloc=b;
+subgraph resdev {
+node [color="#6b5831",style="filled",fillcolor="#ede9e1"];
+collector [label="CollecTor"];
+onionoo [label="Onionoo"];
+subgraph lib {
+node
[shape=box,margin="0.11,0.11",height=0,width=0,fontsize=8];
+lib1 [label="metrics-lib"];
+lib2 [label="metrics-lib"];
+lib3 [label="metrics-lib"];
+}
+}
+subgraph user {
+node [color="#436b31",style="filled",fillcolor="#d4e5cc"];
+statistics [label="Tor Metrics Graphs"];
+exonerator [label="Exonerator"];
+rs [label="Relay Search"];
+bot [label="metrics-bot"];
+consensushealth [label="Consensus Health"];
+}
+}
+relays -> dirauths;
+bridges -> bridgeauth;
+dirauths -> collector;
+dirauths -> consensushealth;
+bridgeauth -> collector;
+collector -> lib1;
+lib1 -> statistics;
+collector -> lib2;
+lib2 -> onionoo;
+collector -> lib3;
+lib3 -> exonerator;
+onionoo -> rs;
+onionoo -> bot;
+legend [rank=sink,fontsize=10,shape=none,label=
+ <
+Legend
+Public Tor Network
Servers
+Tools for Researchers and
Developers
+Tools for End-Users
+ >];
+}
diff --git a/src/main/resources/web/images/ecosystem.png
b/src/main/resources/web/images/ecosystem.png
new file mode 100644
index 000..3b80ed1
Binary files /dev/null and b/src/main/resources/web/images/ecosystem.png differ
diff --git a/src/main/resources/web/jsps/about.jsp
b/src/main/resources/web/jsps/about.jsp
index 7a4d303..bc14b08 100644
--- a/src/main/resources/web/jsps/about.jsp
+++ b/src/main/resources/web/jsps/about.jsp
@@ -14,20 +14,71 @@
About #
- You're curious who has put together all this information, and you're
thinking about contributing? Here we tell you a bit about the Tor metrics team
and give you some pointers for joining our team.
-
+ Tor Metrics archives historical data about the Tor ecosystem, collects
+ data from the public Tor network and related services, and assists in
+ developing novel approaches to safe, privacy preserving data
+ collection.
-
Philosophy #
- We only use public, non-sensitive data for metrics. Each metric goes
through a rigorous review and discussion process before appearing here. We
never publish statisticsor aggregate statisticsof sensitive data,
such as unencrypted contents of traffic.
-
+ We only use public, non-sensitive data for metrics. Each metric goes
+ through a rigorous review and discussion process before appearing here.
+ We never publish statisticsor aggregate statisticsof
+ sensitive data, such as unencrypted contents of traffic.
+
+ Ecosystem #
+ Tor relays and bridges collect aggregated statistics about their usage
+ including bandwidth and connecting clients per country. Source
+ aggregation is used to protect the privacy of connecting
+ usersdiscarding IP addresses and only reporting country
+ information from a local database mapping IP address ranges to countries.
+ These statistics are sent periodically to the directory authorities.
+ CollecTor
+ downloads the latest server descriptors, extra info descriptors
+ containing the aggregated statistics, and consensus documents from the
+ directory authorities and archives them. This archive is public and the
+ metrics-lib
+ Java library can be used to parse the contents of the archive to perform
+ analysis of the data.
+ In order to provide easy access to visualizations of the historical
+ data
[tor-commits] [metrics-web/release] Remove unused code for plotting relays by country.
commit e797903fb93b70c675d2b1655c9d333a713f6bfb
Author: Karsten Loesing
Date: Fri Feb 9 17:57:26 2018 +0100
Remove unused code for plotting relays by country.
---
src/main/R/rserver/graphs.R | 35 ---
1 file changed, 35 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index e24a617..2a055c2 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -311,41 +311,6 @@ plot_networksize <- function(start, end, path) {
ggsave(filename = path, width = 8, height = 5, dpi = 150)
}
-plot_relaycountries <- function(start, end, country, path) {
- end <- min(end, as.character(Sys.Date() - 2))
- s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"servers.csv", sep = ""), stringsAsFactors = FALSE)
- s <- s[s$date >= start & s$date <= end & s$flag == '' &
- s$country == ifelse(country == "all", '', country) &
- s$version == '' & s$platform == '' & s$ec2bridge == '', ]
- s <- data.frame(date = as.Date(s$date, "%Y-%m-%d"), relays = s$relays)
- dates <- seq(from = as.Date(start, "%Y-%m-%d"),
- to = as.Date(end, "%Y-%m-%d"), by="1 day")
- missing <- setdiff(dates, s$date)
- if (length(missing) > 0)
-s <- rbind(s,
-data.frame(date = as.Date(missing, origin = "1970-01-01"),
-relays = NA))
- title <- ifelse(country == "all",
-"Number of relays in all countries\n",
-paste("Number of relays in ", countryname(country), "\n", sep = ""))
- formatter <- function(x, ...) { format(x, scientific = FALSE, ...) }
- date_breaks <- date_breaks(
-as.numeric(max(as.Date(s$date, "%Y-%m-%d")) -
-min(as.Date(s$date, "%Y-%m-%d"
- ggplot(s, aes(x = as.Date(date, "%Y-%m-%d"), y = relays)) +
-geom_line(size = 1) +
-scale_x_date(name = paste("\nThe Tor Project - ",
-"https://metrics.torproject.org/;, sep = ""),
-labels = date_format(date_breaks$format),
-date_breaks = date_breaks$major,
-date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name = "", limits = c(0, max(s$relays,
-na.rm = TRUE)), formatter = formatter) +
-ggtitle(title)
- ggsave(filename = path, width = 8, height = 5, dpi = 150)
-}
-
plot_versions <- function(start, end, path) {
end <- min(end, as.character(Sys.Date() - 2))
s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Don't cut off any data from CSV files.
commit 8831ecebd4ac186a798c2cb844289ed94318a6ed
Author: Karsten Loesing
Date: Fri Feb 9 21:24:01 2018 +0100
Don't cut off any data from CSV files.
When graphing data from CSV files it's not our job to make sure the
data is stable enough to be graphed. That would mean that whoever uses
our CSV files directly would have to make that sure by themselves. If
data is too recent to be graphed, it shouldn't be included in the CSV
files. As a side effect this makes the graphing process a little
easier.
---
src/main/R/rserver/graphs.R | 31 +++
1 file changed, 3 insertions(+), 28 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 440a691..768b776 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -294,7 +294,6 @@ update_geom_defaults("line", list(size = 1))
copyright_notice = "The Tor Project - https://metrics.torproject.org/;
plot_networksize <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 2))
s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"servers.csv", sep = ""), stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
@@ -327,7 +326,6 @@ plot_networksize <- function(start, end, path) {
}
plot_versions <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 2))
s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"servers.csv", sep = ""), stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
@@ -364,7 +362,6 @@ plot_versions <- function(start, end, path) {
}
plot_platforms <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 2))
s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"servers.csv", sep = ""), stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
@@ -392,7 +389,6 @@ plot_platforms <- function(start, end, path) {
}
plot_bandwidth <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 4))
b <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"bandwidth.csv", sep = ""), stringsAsFactors = FALSE)
b <- b[b$date >= start & b$date <= end & b$isexit == '' &
@@ -421,7 +417,6 @@ plot_bandwidth <- function(start, end, path) {
}
plot_bwhist_flags <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 4))
b <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"bandwidth.csv", sep = ""), stringsAsFactors = FALSE)
b <- b[b$date >= start & b$date <= end & b$isexit != '' &
@@ -465,7 +460,6 @@ plot_bwhist_flags <- function(start, end, path) {
}
plot_dirbytes <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 4))
b <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"bandwidth.csv", sep = ""), stringsAsFactors = FALSE)
b <- b[b$date >= start & b$date <= end & b$isexit == '' &
@@ -493,7 +487,6 @@ plot_dirbytes <- function(start, end, path) {
}
plot_relayflags <- function(start, end, flags, path) {
- end <- min(end, as.character(Sys.Date() - 2))
s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"servers.csv", sep = ""), stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$country == '' &
@@ -532,7 +525,6 @@ plot_relayflags <- function(start, end, flags, path) {
}
plot_torperf <- function(start, end, source, server, filesize, path) {
- end <- min(end, as.character(Sys.Date() - 2))
filesizeVal <- ifelse(filesize == '50kb', 50 * 1024,
ifelse(filesize == '1mb', 1024 * 1024, 5 * 1024 * 1024))
t <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
@@ -582,7 +574,6 @@ plot_torperf <- function(start, end, source, server,
filesize, path) {
}
plot_torperf_failures <- function(start, end, source, server, filesize, path) {
- end <- min(end, as.character(Sys.Date() - 2))
filesizeVal <- ifelse(filesize == '50kb', 50 * 1024,
ifelse(filesize == '1mb', 1024 * 1024, 5 * 1024 * 1024))
t <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
@@ -633,7 +624,6 @@ plot_torperf_failures <- function(start, end, source,
server, filesize, path) {
}
plot_connbidirect <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 2))
c <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
"connbidirect2.csv", sep = ""), stringsAsFactors = FALSE)
c <- c[c$date >= start & c$date <= end, ]
@@ -669,7 +659,6 @@ plot_connbidirect <- function(start, end, path) {
}
plot_bandwidth_flags <- function(start, end, path) {
- end <- min(end, as.character(Sys.Date() - 4))
b <-
[tor-commits] [metrics-web/release] Use variables for paths.
commit ee0218d8e3dfc5163ef74e443ecfbcfac293977f
Author: Karsten Loesing
Date: Fri Feb 9 22:00:59 2018 +0100
Use variables for paths.
---
src/main/R/rserver/graphs.R | 93 ++---
1 file changed, 46 insertions(+), 47 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 768b776..9a37d88 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -293,9 +293,13 @@ update_geom_defaults("line", list(size = 1))
copyright_notice = "The Tor Project - https://metrics.torproject.org/;
+stats_dir = "/srv/metrics.torproject.org/metrics/shared/stats/"
+
+rdata_dir = "/srv/metrics.torproject.org/metrics/shared/RData/"
+
plot_networksize <- function(start, end, path) {
- s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"servers.csv", sep = ""), stringsAsFactors = FALSE)
+ s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
+stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
s$country == '' & s$version == '' & s$platform == '' &
s$ec2bridge == '', ]
@@ -326,8 +330,8 @@ plot_networksize <- function(start, end, path) {
}
plot_versions <- function(start, end, path) {
- s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"servers.csv", sep = ""), stringsAsFactors = FALSE)
+ s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
+stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
s$country == '' & s$version != '' & s$platform == '' &
s$ec2bridge == '', ]
@@ -362,8 +366,8 @@ plot_versions <- function(start, end, path) {
}
plot_platforms <- function(start, end, path) {
- s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"servers.csv", sep = ""), stringsAsFactors = FALSE)
+ s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
+stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$flag == '' &
s$country == '' & s$version == '' & s$platform != '' &
s$ec2bridge == '', ]
@@ -389,8 +393,8 @@ plot_platforms <- function(start, end, path) {
}
plot_bandwidth <- function(start, end, path) {
- b <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"bandwidth.csv", sep = ""), stringsAsFactors = FALSE)
+ b <- read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
+stringsAsFactors = FALSE)
b <- b[b$date >= start & b$date <= end & b$isexit == '' &
b$isguard == '', ]
b <- data.frame(date = as.Date(b$date, "%Y-%m-%d"),
@@ -417,8 +421,8 @@ plot_bandwidth <- function(start, end, path) {
}
plot_bwhist_flags <- function(start, end, path) {
- b <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"bandwidth.csv", sep = ""), stringsAsFactors = FALSE)
+ b <- read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
+stringsAsFactors = FALSE)
b <- b[b$date >= start & b$date <= end & b$isexit != '' &
b$isguard != '', ]
bw <- data.frame(date = as.Date(b$date, "%Y-%m-%d"),
@@ -460,8 +464,8 @@ plot_bwhist_flags <- function(start, end, path) {
}
plot_dirbytes <- function(start, end, path) {
- b <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"bandwidth.csv", sep = ""), stringsAsFactors = FALSE)
+ b <- read.csv(paste(stats_dir, "bandwidth.csv", sep = ""),
+stringsAsFactors = FALSE)
b <- b[b$date >= start & b$date <= end & b$isexit == '' &
b$isguard == '', ]
b <- data.frame(date = as.Date(b$date, "%Y-%m-%d"),
@@ -487,8 +491,8 @@ plot_dirbytes <- function(start, end, path) {
}
plot_relayflags <- function(start, end, flags, path) {
- s <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"servers.csv", sep = ""), stringsAsFactors = FALSE)
+ s <- read.csv(paste(stats_dir, "servers.csv", sep = ""),
+stringsAsFactors = FALSE)
s <- s[s$date >= start & s$date <= end & s$country == '' &
s$version == '' & s$platform == '' & s$ec2bridge == '', ]
s <- data.frame(date = as.Date(s$date, "%Y-%m-%d"),
@@ -527,8 +531,8 @@ plot_relayflags <- function(start, end, flags, path) {
plot_torperf <- function(start, end, source, server, filesize, path) {
filesizeVal <- ifelse(filesize == '50kb', 50 * 1024,
ifelse(filesize == '1mb', 1024 * 1024, 5 * 1024 * 1024))
- t <- read.csv(paste("/srv/metrics.torproject.org/metrics/shared/stats/",
-"torperf-1.1.csv", sep = ""), stringsAsFactors = FALSE)
+ t <- read.csv(paste(stats_dir, "torperf-1.1.csv", sep = ""),
+stringsAsFactors = FALSE)
known_sources <- c("all", unique(t[t$source != "", "source"]))
colours <- data.frame(source = known_sources,
colour = brewer.pal(length(known_sources), "Paired"),
@@ -576,8 +580,8 @@ plot_torperf <- function(start,
[tor-commits] [metrics-web/release] Adds extended introduction and services to the home page (See: #23169)
commit 8f55bff0cc8fa70cce1ce8a87e0bb54a1bf739e4
Author: Iain R. Learmonth
Date: Thu Mar 1 13:44:20 2018 +
Adds extended introduction and services to the home page (See: #23169)
---
src/main/resources/web/css/style.css | 68 +
src/main/resources/web/jsps/index.jsp | 81 +++
2 files changed, 115 insertions(+), 34 deletions(-)
diff --git a/src/main/resources/web/css/style.css
b/src/main/resources/web/css/style.css
index b2485ff..97e91b8 100644
--- a/src/main/resources/web/css/style.css
+++ b/src/main/resources/web/css/style.css
@@ -14,7 +14,7 @@ body #wrapper {
/* Page header - override bootstrap defaults */
.page-header {
-padding: 0em 2.0em 2.0em 2.0em;
+padding: 0em 2.0em 2.0em 2.0em;
margin: 0;
border-bottom: 0;
color:#fff;
@@ -33,20 +33,20 @@ body #wrapper {
max-width:40%;
height:auto;
}
-.page-header img#metrics-logo {
-height:75px;
-width:auto !important;
-}
+.page-header img#metrics-logo {
+height:75px;
+width:auto !important;
+}
.page-header div {
text-align:right;
position:absolute;
right:2.5em;
-bottom: 0.9em;
-max-width: 400px;
+bottom: 0.9em;
+max-width: 400px;
}
.page-header div p {
-font-size: 14px;
-line-height: 16px;
+font-size: 14px;
+line-height: 16px;
}
@media (max-width: 767.9px) {
.page-header {
@@ -59,7 +59,7 @@ body #wrapper {
display:none;
}
}
-@media (max-width: 1199.9px) {
+@media (max-width: 1199.9px) {
.page-header div {
max-width:330px;
}
@@ -136,9 +136,9 @@ body {
.navbar-secondary .section-header:first-of-type {
margin-top:0px;
}
-.navbar-secondary .navbar-nav>li>a {
-margin-bottom: 0px !important;
-}
+.navbar-secondary .navbar-nav>li>a {
+margin-bottom: 0px !important;
+}
/* images */
@@ -336,10 +336,27 @@ body .topButton {
color: #5bc0de;
}
+/* home jumbotron */
+.jumbotron {
+padding-bottom: 10px;
+}
+.jumbotron h1 {
+margin-top: -20px;
+font-size: 50px;
+}
+@media only screen and (max-width : 991px) { .jumbotron h1 { font-size: 46px;
} }
+@media only screen and (max-width : 352px) {
+.jumbotron h1 {
+font-size: 38px;
+}
+.jumbotron p {
+display: none;
+}
+}
/* home dashboard page */
.dashboard {
-padding:3em 5em 0 5em;
+padding:0em 5em 0 5em;
}
.dashboard .col-sm-4 {
margin-bottom:2em;
@@ -353,10 +370,27 @@ body .topButton {
padding:2em 2em 1em 2em;
color:#999;
}
-.dashboard h2, .dashboard p {
-color:#999;
+.dashboard h2 {
+font-size: 42px;
+color: rgb(125, 70, 152);
+padding-top: 0;
+margin-top: 0;
+}
+.dashboard h3 {
+font-size: 26px;
+}
+@media (max-width: 992px) {
+.dashboard h3 {
+font-size: 20px;
+}
+}
+.dashboard h3, .dashboard p {
+color: #999;
+}
+.dashboard p.lead {
+color: #333;
}
-.dashboard a[href], .dashboard a[href]:hover, .dashboard a[href]:focus,
.dashboard a[href]:active, .dashboard a[href] h2 {
+.dashboard a[href], .dashboard a[href]:hover, .dashboard a[href]:focus,
.dashboard a[href]:active, .dashboard a[href] h3 {
color:#68b030;
}
.dashboard a[href]:hover {
diff --git a/src/main/resources/web/jsps/index.jsp
b/src/main/resources/web/jsps/index.jsp
index 56e2afa..5968727 100644
--- a/src/main/resources/web/jsps/index.jsp
+++ b/src/main/resources/web/jsps/index.jsp
@@ -5,32 +5,79 @@
-
-
-
-
-
-
-
-Welcome!
-
- What would you like to know about the https://www.torproject.org/; target="_blank">Tor network?
-
+
+Welcome to Tor Metrics!
+
+
+The https://www.torproject.org/; target="_blank">Tor
+network is one of the largest deployed anonymity networks, consisting
of thousands of volunteer-run
+relays and millions of users.
+Users, advocates, relay operators, and journalists can better
understand the Tor network
+through data and analysis made available by Tor Metrics.
+
+
+Analyzing a live anonymity system must be performed with great care
so that
+the users' privacy is not put at risk. Any metrics collected must
+not undermine the anonymity or security properties of the Tor
+ network. Read more
+
+
-
+
+ Analysis
+View visualizations of statistics collected from the
public Tor network and from Tor Project infrastructure.
-
-
href="${category[0]}.html"> ${category[1]} ${category[2]}
+ Users Where Tor users are from and
how they connect to Tor.
+
+
+
+ Servers How many relays and bridges are
online and what we know about them.
+
+
+
+ Traffic How much traffic the Tor network
can handle and how much traffic there is.
+
+
+
+ Performance How fast and reliable the Tor
network is.
+
[tor-commits] [metrics-web/release] Simplify setting the lower y axis limit to 0.
commit 73dd9973a5a8f65aec79764042e1ca587829d00f
Author: Karsten Loesing
Date: Fri Feb 9 20:38:43 2018 +0100
Simplify setting the lower y axis limit to 0.
---
src/main/R/rserver/graphs.R | 64 +
1 file changed, 24 insertions(+), 40 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 2a055c2..c7f7d90 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -303,8 +303,7 @@ plot_networksize <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name = "", limits = c(0, max(networksize$value,
-na.rm = TRUE))) +
+scale_y_continuous(name = "", limits = c(0, NA)) +
scale_colour_hue("", breaks = c("relays", "bridges"),
labels = c("Relays", "Bridges")) +
ggtitle("Number of relays\n")
@@ -341,8 +340,7 @@ plot_versions <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name = "",
- limits = c(0, max(versions$relays, na.rm = TRUE))) +
+scale_y_continuous(name = "", limits = c(0, NA)) +
scale_colour_manual(name = "Tor version",
values = colours[colours$breaks %in% visible_versions, 2],
breaks = visible_versions) +
@@ -370,8 +368,7 @@ plot_platforms <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name = "",
- limits = c(0, max(platforms$value, na.rm = TRUE))) +
+scale_y_continuous(name = "", limits = c(0, NA)) +
scale_colour_manual(name = "Platform",
breaks = c("Linux", "Darwin", "BSD", "Windows", "Other"),
labels = c("Linux", "macOS", "BSD", "Windows", "Other"),
@@ -401,8 +398,7 @@ plot_bandwidth <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name = "Bandwidth (Gbit/s)",
-limits = c(0, max(bandwidth$value, na.rm = TRUE) * 8 / 1e9)) +
+scale_y_continuous(name = "Bandwidth (Gbit/s)", limits = c(0, NA)) +
scale_colour_hue(name = "", h.start = 90,
breaks = c("bwadv", "bwhist"),
labels = c("Advertised bandwidth", "Bandwidth history")) +
@@ -448,8 +444,7 @@ plot_bwhist_flags <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name="Bandwidth (Gbit/s)",
-limits = c(0, max(bw$value, na.rm = TRUE) * 8 / 1e9)) +
+scale_y_continuous(name="Bandwidth (Gbit/s)", limits = c(0, NA)) +
scale_colour_manual(name = "",
values = c("#E69F00", "#56B4E9", "#009E73", "#0072B2")) +
ggtitle("Bandwidth history by relay flags") +
@@ -477,8 +472,7 @@ plot_dirbytes <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name="Bandwidth (Gbit/s)",
-limits = c(0, max(dir$value, na.rm = TRUE) * 8 / 1e9)) +
+scale_y_continuous(name="Bandwidth (Gbit/s)", limits = c(0, NA)) +
scale_colour_hue(name = "",
breaks = c("dirwrite", "dirread"),
labels = c("Written dir bytes", "Read dir bytes")) +
@@ -519,8 +513,7 @@ plot_relayflags <- function(start, end, flags, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor, limits = as.Date(c(start,
end))) +
-scale_y_continuous(name = "", limits = c(0, max(networksize$value,
-na.rm = TRUE))) +
+scale_y_continuous(name = "", limits = c(0, NA)) +
scale_colour_manual(name = "Relay flags", values = c("#E69F00",
"#56B4E9", "#009E73", "#EE6A50", "#00", "#0072B2"),
breaks = flags, labels = flags) +
@@ -567,8 +560,7 @@ plot_torperf <- function(start, end, source, server,
filesize, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-scale_y_continuous(name = "") +
-expand_limits(y = 0) +
+scale_y_continuous(name = "", limits = c(0, NA)) +
scale_fill_manual(name = paste("Measured times on",
ifelse(source == "all", "all sources", source), "per day"),
breaks = c("line", "ribbon"),
@@ -724,8 +716,7 @@ plot_bandwidth_flags <- function(start, end, path) {
labels = date_format(date_breaks$format),
date_breaks = date_breaks$major,
date_minor_breaks = date_breaks$minor) +
-
[tor-commits] [metrics-web/release] Tweak y scales some more.
commit 0021881b15972010675be6176c40160b19fa7a91
Author: Karsten Loesing
Date: Tue Feb 13 15:04:52 2018 +0100
Tweak y scales some more.
---
src/main/R/rserver/graphs.R | 32 +---
1 file changed, 17 insertions(+), 15 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index 04f1b10..c7ef2bd 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -348,7 +348,7 @@ plot_networksize <- function(start, end, path) {
colour = variable)) + geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", limits = c(0, NA)) +
+scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) +
scale_colour_hue("", breaks = c("relays", "bridges"),
labels = c("Relays", "Bridges")) +
ggtitle("Number of relays") +
@@ -379,7 +379,7 @@ plot_versions <- function(start, end, path) {
geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", limits = c(0, NA)) +
+scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) +
scale_colour_manual(name = "Tor version",
values = colours[colours$breaks %in% visible_versions, 2],
breaks = visible_versions) +
@@ -401,7 +401,7 @@ plot_platforms <- function(start, end, path) {
geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", limits = c(0, NA)) +
+scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) +
scale_colour_manual(name = "Platform",
breaks = c("Linux", "Darwin", "BSD", "Windows", "Other"),
labels = c("Linux", "macOS", "BSD", "Windows", "Other"),
@@ -525,7 +525,7 @@ plot_relayflags <- function(start, end, flags, path) {
colour = as.factor(variable))) + geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", limits = c(0, NA)) +
+scale_y_continuous(name = "", labels = formatter, limits = c(0, NA)) +
scale_colour_manual(name = "Relay flags", values = c("#E69F00",
"#56B4E9", "#009E73", "#EE6A50", "#00", "#0072B2"),
breaks = flags, labels = flags) +
@@ -566,13 +566,14 @@ plot_torperf <- function(start, end, source, server,
filesize, path) {
ymax = q3/1e3, fill = "ribbon")) +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", limits = c(0, NA)) +
+scale_y_continuous(name = "", labels = unit_format(unit = "s"),
+ limits = c(0, NA)) +
scale_fill_manual(name = paste("Measured times on",
ifelse(source == "all", "all sources", source), "per day"),
breaks = c("line", "ribbon"),
labels = c("Median", "1st to 3rd quartile"),
values = paste(colour, c("", "66"), sep = "")) +
-ggtitle(paste("Time in seconds to complete", filesizeStr,
+ggtitle(paste("Time to complete", filesizeStr,
"request to", server, "server")) +
labs(caption = copyright_notice) +
theme(legend.position = "top")
@@ -613,7 +614,7 @@ plot_torperf_failures <- function(start, end, source,
server, filesize, path) {
geom_point(size = 2) +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", labels = percent) +
+scale_y_continuous(name = "", labels = percent, limits = c(0, NA)) +
scale_colour_hue(name = paste("Problems encountered on",
ifelse(source == "all", "all sources", source)),
h.start = 45, breaks = c("timeouts", "failures"),
@@ -641,7 +642,7 @@ plot_connbidirect <- function(start, end, path) {
fill = direction), alpha = 0.5, show_guide = FALSE) +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "", labels = percent) +
+scale_y_continuous(name = "", labels = percent, limits = c(0, NA)) +
scale_colour_hue(name = "Medians and interquartile ranges",
breaks = c("both", "write", "read"),
labels = c("Both reading and writing", "Mostly writing",
@@ -946,7 +947,7 @@ plot_hidserv_dir_onions_seen <- function(start, end, path) {
geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
-scale_y_continuous(name = "") +
+scale_y_continuous(name = "", limits = c(0, NA), labels = formatter) +
ggtitle("Unique .onion addresses") +
labs(caption = copyright_notice)
ggsave(filename = path,
[tor-commits] [metrics-web/release] Give up on reshape in favor of tidyr.
commit 055229303076be6275ba64e9d2864d1ebbed004e
Author: Karsten Loesing
Date: Thu Mar 1 09:49:42 2018 +0100
Give up on reshape in favor of tidyr.
---
src/main/R/rserver/graphs.R | 14 +++---
src/main/R/rserver/rserve-init.R | 1 -
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/src/main/R/rserver/graphs.R b/src/main/R/rserver/graphs.R
index fd5201d..8847967 100644
--- a/src/main/R/rserver/graphs.R
+++ b/src/main/R/rserver/graphs.R
@@ -348,7 +348,7 @@ plot_networksize <- function(start, end, path) {
s <- rbind(s,
data.frame(date = as.Date(missing, origin = "1970-01-01"),
relays = NA, bridges = NA))
- networksize <- melt(s, id = "date")
+ networksize <- gather(s, variable, value, -date)
ggplot(networksize, aes(x = as.Date(date, "%Y-%m-%d"), y = value,
colour = variable)) + geom_line() +
scale_x_date(name = "", breaks = custom_breaks,
@@ -456,7 +456,7 @@ prepare_bandwidth <- function(start, end) {
plot_bandwidth <- function(start, end, path) {
b <- prepare_bandwidth(start, end)
- bandwidth <- melt(b, id = "date")
+ bandwidth <- gather(b, variable, value, -date)
ggplot(bandwidth, aes(x = as.Date(date, "%Y-%m-%d"),
y = value, colour = variable)) +
geom_line() +
@@ -544,7 +544,7 @@ prepare_dirbytes <- function(start, end, path) {
plot_dirbytes <- function(start, end, path) {
b <- prepare_dirbytes(start, end)
- dir <- melt(b, id = "date")
+ dir <- gather(b, variable, value, -date)
ggplot(dir, aes(x = as.Date(date, "%Y-%m-%d"), y = value,
colour = variable)) +
geom_line() +
@@ -699,7 +699,7 @@ plot_torperf_failures <- function(start, end, source,
server, filesize, path) {
torperf <- rbind(torperf,
data.frame(date = as.Date(missing, origin = "1970-01-01"),
timeouts = NA, failures = NA))
- torperf <- melt(torperf, id = "date")
+ torperf <- gather(torperf, variable, value, -date)
filesizes <- data.frame(filesizes = c("5mb", "1mb", "50kb"),
label = c("5 MiB", "1 MiB", "50 KiB"), stringsAsFactors = FALSE)
filesizeStr <- filesizes[filesizes$filesize == filesize, "label"]
@@ -734,7 +734,7 @@ prepare_connbidirect <- function(start, end) {
levels = c("both", "write", "read")),
quantile = paste("X", c$quantile, sep = ""),
fraction = c$fraction / 100)
- c <- cast(c, date + direction ~ quantile, value = "fraction")
+ c <- spread(c, quantile, fraction)
c
}
@@ -743,7 +743,7 @@ plot_connbidirect <- function(start, end, path) {
ggplot(c, aes(x = date, y = X0.5, colour = direction)) +
geom_line(size = 0.75) +
geom_ribbon(aes(x = date, ymin = X0.25, ymax = X0.75,
-fill = direction), alpha = 0.5, show_guide = FALSE) +
+fill = direction), alpha = 0.5, show.legend = FALSE) +
scale_x_date(name = "", breaks = custom_breaks,
labels = custom_labels, minor_breaks = custom_minor_breaks) +
scale_y_continuous(name = "", labels = percent, limits = c(0, NA)) +
@@ -787,7 +787,7 @@ prepare_bandwidth_flags <- function(start, end) {
b <- aggregate(list(advbw = b$advbw, bwhist = b$bwhist),
by = list(date = b$date, flag = b$flag), FUN = sum,
na.rm = TRUE, na.action = NULL)
- b <- melt(b, id.vars = c("date", "flag"), variable_name = "type")
+ b <- gather(b, type, value, -c(date, flag))
bandwidth <- b[b$value > 0, ]
dates <- seq(from = as.Date(start, "%Y-%m-%d"),
to = as.Date(end, "%Y-%m-%d"), by = "1 day")
diff --git a/src/main/R/rserver/rserve-init.R b/src/main/R/rserver/rserve-init.R
index ede7c73..b9a1d3b 100644
--- a/src/main/R/rserver/rserve-init.R
+++ b/src/main/R/rserver/rserve-init.R
@@ -1,7 +1,6 @@
##Pre-loaded libraries and graphing functions to speed things up
library("ggplot2")
-library("reshape")
library("RColorBrewer")
library("scales")
library(dplyr)
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Take out old news from Onionoo's specification.
commit 75258264b84b01e2d4e43b4642f493bb652da828 Author: Karsten Loesing Date: Thu Mar 1 14:49:54 2018 +0100 Take out old news from Onionoo's specification. --- src/main/resources/web/jsps/onionoo.jsp | 36 ++--- 1 file changed, 11 insertions(+), 25 deletions(-) diff --git a/src/main/resources/web/jsps/onionoo.jsp b/src/main/resources/web/jsps/onionoo.jsp index fc80677..0f29507 100644 --- a/src/main/resources/web/jsps/onionoo.jsp +++ b/src/main/resources/web/jsps/onionoo.jsp @@ -569,19 +569,18 @@ a family. -version updated +version # Return only relays or bridges running a Tor version that starts with the parameter value without leading "Tor". Searches are case-insensitive. -Extended to bridges on November 28, 2017. -host_name new +host_name # Return only relays with a domain name ending in the given (partial) @@ -590,13 +589,12 @@ Searches for subdomains of a specific domain should ideally be prefixed with a period, for example: ".csail.mit.edu". Non-ASCII host name characters must be encoded as punycode. Filtering by host name is case-insensitive. -Added on November 17, 2017. -recommended_version new +recommended_version # Return only relays and bridges running a Tor software version that is @@ -606,7 +604,6 @@ Uses the version in the consensus or bridge network status. Relays and bridges are not contained in either result, if the version they are running is not known. Parameter values are case-insensitive. -Added on November 28, 2017. @@ -740,14 +737,13 @@ Omitted if no major protocol changes are planned. -build_revision new +build_revision string optional # Git revision of the Onionoo instance's software used to write this response, which will be omitted if unknown. -Added on October 10, 2017. @@ -1455,7 +1451,7 @@ found. -version new +version string optional # @@ -1465,7 +1461,6 @@ by the directory authorities in the "v" line of the consensus. Omitted if either the directory authorities or the relay did not report which version the relay runs or if the relay runs an alternative Tor implementation. -Added on November 28, 2017. @@ -1486,7 +1481,7 @@ the relay did not report which version it runs. -effective_family updated +effective_family array of strings optional # @@ -1497,14 +1492,12 @@ These relays are part of this relay's family and they consider this relay to be part of their family. Omitted if empty or if descriptor containing this information cannot be found. -Removed the $ prefix from fingerprintson December 20, -2017. -alleged_family updated +alleged_family array of strings optional # @@ -1515,14 +1508,12 @@ These relays are part of this relay's family but they don't consider this relay to be part of their family. Omitted if empty or if descriptor containing this information cannot be found. -Removed the $ prefix from fingerprintson December 20, -2017. -indirect_family updated +indirect_family array of strings optional # @@ -1532,8 +1523,6 @@ mutual family relationship with this relay but that can be reached by following effective, mutual family relationships starting at this relay. Omitted if empty or if descriptor containing this information cannot be found. -Removed the $ prefix from fingerprintson December 20, -2017. @@ -1619,7 +1608,7 @@ contain measurement information. -unreachable_or_addresses new +unreachable_or_addresses array of strings optional # @@ -1640,7 +1629,6 @@ relays with unreachable addresses will be included here. Addresses are in arbitrary order. IPv6 hex characters are all lower-case. Omitted if empty. -Added on November 17, 2017. @@ -1801,7 +1789,7 @@ information cannot be found. -version new +version string optional # @@ -1811,13 +1799,12 @@ by the bridge in the "platform" line of its server descriptor. Omitted if not provided by the bridge, if the descriptor containing this information cannot be found, or if the bridge runs an alternative Tor implementation. -Added on November 28, 2017. -recommended_version new +recommended_version boolean optional # @@ -1827,7 +1814,6 @@ recommended by the directory authorities or not. Uses the bridge version in the bridge network status. Omitted if either the directory authorities did not recommend versions, or the bridge did not report which version it runs. -Added on November 28, 2017. ___ tor-commits mailing list tor-commits@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
[tor-commits] [metrics-web/release] Update news.json to version 227 of doc/MetricsTimeline.
commit a57dfb2d1604679cb9d00a1d8aaf9be724b69920
Author: Karsten Loesing
Date: Fri Feb 16 20:30:36 2018 +0100
Update news.json to version 227 of doc/MetricsTimeline.
---
src/main/resources/web/json/news.json | 88 +--
1 file changed, 73 insertions(+), 15 deletions(-)
diff --git a/src/main/resources/web/json/news.json
b/src/main/resources/web/json/news.json
index 6b8565a..c0d069e 100644
--- a/src/main/resources/web/json/news.json
+++ b/src/main/resources/web/json/news.json
@@ -2674,23 +2674,15 @@
{
"start": "2017-12-04",
"ongoing": true,
-"description": "DDoS attack uses a bug in the KIST scheduler to crash
relays by running them out of memory.",
+"description": "DDoS attack creates load on the network.",
"links": [
"https://lists.torproject.org/pipermail/tor-relays/2017-December/013669.html\;>tor-relays
thread",
"https://metrics.torproject.org/relayflags.html?start=2017-11-01=2018-01-31=Running=Exit=Fast=Guard=Stable=HSDir\;>relay
graph",
"https://lists.torproject.org/pipermail/tor-project/2017-December/001604.html\;>summary
post",
- "https://bugs.torproject.org/24665\;>ticket"
+ "https://bugs.torproject.org/24902\;>ticket"
]
},
{
-"start": "2017-12-12",
-"end": "2018-01-18",
-"protocols": [
- "meek"
-],
-"description": "Outage of the https://atlas.torproject.org/#details/C20658946DD706A7A2181159A1A04CD838570D04\;>meek.bamsoftware.com
(unthrottled for public use), https://atlas.torproject.org/#details/AA033EEB61601B2B7312D89B62AAA23DC3ED8A34\;>meek.bamsoftware.com:7443
(former meek-azure, now unused), and https://atlas.torproject.org/#details/D36B0328969EC57AB3085A4470882D99A09C0492\;>gaeuploader.meek.bamsoftware.com
(used by https://github.com/katherinelitor/GAEuploader\;>GAEuploader)
bridges."
- },
- {
"start": "2017-12-05",
"end": "2017-12-15",
"places": [
@@ -2702,6 +2694,14 @@
]
},
{
+"start": "2017-12-12",
+"end": "2018-01-18",
+"protocols": [
+ "meek"
+],
+"description": "Outage of the https://atlas.torproject.org/#details/C20658946DD706A7A2181159A1A04CD838570D04\;>meek.bamsoftware.com
(unthrottled for public use), https://atlas.torproject.org/#details/AA033EEB61601B2B7312D89B62AAA23DC3ED8A34\;>meek.bamsoftware.com:7443
(former meek-azure, now unused), and https://atlas.torproject.org/#details/D36B0328969EC57AB3085A4470882D99A09C0492\;>gaeuploader.meek.bamsoftware.com
(used by https://github.com/katherinelitor/GAEuploader\;>GAEuploader)
bridges."
+ },
+ {
"start": "2017-12-20",
"protocols": [
"ipv4",
@@ -2714,7 +2714,7 @@
},
{
"start": "2017-12-21",
-"description": "Release of tor 0.3.2.8-rc, intended to fix the bug that
was enabling an ongoing DDoS on relays.",
+"description": "Release of tor 0.3.2.8-rc, intended to fix the KIST bug
that enabled a DoS on relays by running them out of memory..",
"links": [
"https://lists.torproject.org/pipermail/tor-talk/2017-December/043844.html\;>announcement",
"https://bugs.torproject.org/24665\;>ticket"
@@ -2735,6 +2735,19 @@
]
},
{
+"start": "2018-01-01",
+"places": [
+ "ae"
+],
+"protocols": [
+ "relay"
+],
+"description": "User report that the UAE blocked Tor, bridges work.",
+"links": [
+ "https://bugs.torproject.org/25137#comment:17\;>comment"
+]
+ },
+ {
"start": "2018-01-05",
"ongoing": true,
"description": "Outage of the op-hk OnionPerf instance.",
@@ -2761,6 +2774,51 @@
]
},
{
+"start": "2018-01-20",
+"protocols": [
+ "relay"
+],
+"description": "Some more directory authorities upgrade to 0.3.2.9 which
enforces new requirements for the exit flag. ~50 relays lose the exit flag",
+"links": [
+ "https://lists.torproject.org/pipermail/tor-relays/2018-February/014478.html\;>tor-relays
post"
+]
+ },
+ {
+"start": "2018-02-08",
+"protocols": [
+ "ipv4",
+ "ipv6"
+],
+"description": "geoip and geoip6 databases updated to \"February 7 2018
Maxmind GeoLite2 Country\" (geoip-db-digest
FF83AD73DE7672C77EDFF4B241642C7C90F7, geoip6-db-digest
B1CDBFEB7C88F82EF3B5289CAFEED1321FA4693F).",
+"links": [
+ "https://gitweb.torproject.org/tor.git/commit/?id=f1278b7e573f01edb8bc3ab4d7fbfbd65b5d2c60\;>commit"
+]
+ },
+ {
+"start": "2018-02-10",
+"protocols": [
+ "relay"
+],
+"description": "Tor 0.3.3.2-alpha is released (containing important
denial-of-service migitations for relays)",
+"links": [
+ "https://blog.torproject.org/tor-0332-alpha-released-bugfixes-and-dos-prevention\;>blog"
+]
+ },
+ {
+"start": "2018-02-11",
+"protocols": [
+ "relay"
+],
+"description": "Tor 0.3.3.2-alpha reaches FreeBSD repositories"
+ },
+ {
+"start": "2018-02-12",
+"protocols": [
+ "relay"
+],
+
[tor-commits] [metrics-web/release] Preserve UTF-8 encoding of table data.
commit 50ae3c4579709fca8e95bfcab136a21de5acfe70
Author: Karsten Loesing
Date: Wed Feb 7 17:35:02 2018 +0100
Preserve UTF-8 encoding of table data.
Fixes #25167.
---
src/main/java/org/torproject/metrics/web/RObjectGenerator.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
index 06177bf..c7d0041 100644
--- a/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
+++ b/src/main/java/org/torproject/metrics/web/RObjectGenerator.java
@@ -14,6 +14,7 @@ import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
@@ -222,7 +223,7 @@ public class RObjectGenerator implements
ServletContextListener {
/* Write the table content to a map. */
List> result = new ArrayList<>();
try (BufferedReader br = new BufferedReader(new InputStreamReader(
-new ByteArrayInputStream(tableBytes {
+new ByteArrayInputStream(tableBytes), StandardCharsets.UTF_8))) {
String line = br.readLine();
if (line != null) {
List headers = new ArrayList<>(Arrays.asList(line.split(",")));
___
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits
