Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september, s30-o21a1|
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by karsten):

 * status:  needs_review => closed
 * resolution:   => fixed


Comment:

 On second thought, this bugfix doesn't really need another review. Merged
 to master, will release and deploy later today. Closing. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september, s30-o21a1|
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by karsten):

 * status:  merge_ready => needs_review


Comment:

 This has been merged and deployed, but I noticed a minor issue with paths
 and file names of written files. Please also review
 
[https://gitweb.torproject.org/user/karsten/collector.git/commit/?h=task-19332-2=14cff6502252a0bf70453581d11b8d05a6445ab8
 commit 14cff65 in my task-19332-2 branch] with a fix. I'd like to release
 and deploy that together with #31204.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september, s30-o21a1|
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by karsten):

 * status:  needs_review => merge_ready


Comment:

 Setting to merge_ready as per irl's statement during yesterday's meeting:
 ''"15:41:20  i think it's ok to merge it and i can review the test
 cases retroactively"''

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september, s30-o21a1|
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by karsten):

 * status:  needs_revision => needs_review


Comment:

 irl, please review [https://gitweb.torproject.org/user/karsten/metrics-
 lib.git/commit/?h=task-19332=2e6d689c62417e1277eca98805eda49da6cad386
 metrics-lib commit 2e6d689] and
 [https://gitweb.torproject.org/user/karsten/metrics-
 lib.git/commit/?h=task-19332=2e6d689c62417e1277eca98805eda49da6cad386
 CollecTor commit 2e6d689] with updates to the BridgeDB spec changes.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_revision
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september, s30-o21a1|
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by karsten):

 * status:  needs_review => needs_revision


Comment:

 Moving to needs_revision because of pending spec changes in #31780.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september,  s30-a1  |
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by gaba):

 * keywords:  metrics-roadmap-2019-q2, anti-censorship-roadmap-september =>
 metrics-roadmap-2019-q2, anti-censorship-roadmap-september,  s30-a1


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:  #31274   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by gaba):

 * parent:   => #31274


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:29 karsten]:
 > Replying to [comment:28 phw]:
 > > I'll follow up with the spec once it's done. I hope this is good
 enough to make progress in the meanwhile.
 >
 > Yes, that's a fine start. Thanks!
 [[br]]
 I forgot to mention that `CC` is actually `CC/EMAIL`.
 
[https://github.com/NullHypothesis/torspec/commit/8e49989519ca9f38955cc5f240e5c7cb6fb9a7c7
 #diff-8225f8d4c413573ee924f8e3fc19d521R441 Our draft spec] explains the
 details. Sorry about that.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:28 phw]:
 > I'll follow up with the spec once it's done. I hope this is good enough
 to make progress in the meanwhile.
 [[br]]
 For what it's worth, we now have a draft of the spec over at #31780.

 Also, the spec isn't carved in stone, so please let me know if you would
 like to add or improve anything and I'll get right to it.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  irl  |Sponsor:
 |  Sponsor30-must
-+-
Changes (by karsten):

 * status:  assigned => needs_review
 * reviewer:  cohosh => irl


Comment:

 Replying to [comment:28 phw]:
 > I'll follow up with the spec once it's done. I hope this is good enough
 to make progress in the meanwhile.

 Yes, that's a fine start. Thanks!

 I wrote some code. irl, please review
 [https://gitweb.torproject.org/user/karsten/metrics-
 lib.git/commit/?h=task-19332=820c246f0c10dbc89fcbfcb615473b3be555e330
 metrics-lib commit 820c246] and
 
[https://gitweb.torproject.org/user/karsten/collector.git/commit/?h=task-19332=3866164a09962e6ed640d70b623c089b680aa4d6
 CollecTor commit 3866164].

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:27 karsten]:
 > I probably asked this before and forgot the answer: where would I find
 the spec for parsing the format? Thanks in advance!
 [[br]]
 The spec is still on my todo list. Here's a quick-and-dirty summary, so
 you don't need to block on me:
 * A metrics file starts with `bridgedb-stats-end -MM-DD HH:MM:SS (SECS
 s)`
 * The second line determines the version of the format. It currently is
 `bridgedb-stats-version 1.0`
 * From here on, we have multiple `bridgedb-metric-count` lines. They are
 structured as follows:
   * `bridgedb-metric-count DIST.PROTO.CC.[success|fail].none NUM`
   * `DIST` is BridgeDB's distribution mechanism, which currently is
 `http`, `email`, or `moat`.
   * `PROTO` is the obfuscation protocol, which currently is `obfs2`,
 `obfs3`, `obfs4`, `scramblesuit`, or `fte`.
   * `CC` is our two-letter country code.
   * The second-to-last field is either `success` or `fail` depending on if
 the BridgeDB request succeeded or not.
   * The last field is currently `none` but will eventually be a anomaly
 score, perhaps normalised to [0, 1]. I would suggest to ignore it for now.
   * `NUM` is the approximate number of requests, rounded to the next
 multiple of 10.

 I'll follow up with the spec once it's done. I hope this is good enough to
 make progress in the meanwhile.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 Gah, I misspelled the directory name in one place. Fixed, files are coming
 in as they should. Thanks!

 I probably asked this before and forgot the answer: where would I find the
 spec for parsing the format? Thanks in advance!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Here's our latest rsync attempt:
 {{{
 1568738247:metrics:<36>Sep 17 16:37:27 collector-ssh-wrap[29644]: The
 SSH_ORIGINAL_COMMAND ('rsync --server -logDtpre.iLsfxC .
 /srv/collector.torproject.org/collector/in/bridgedb-stats/') is not on the
 whitelist
 1568738247:metrics:rsync: connection unexpectedly closed (0 bytes received
 so far) [sender]
 1568738247:metrics:rsync error: error in rsync protocol data stream (code
 12) at io.c(235) [sender=3.1.2]
 }}}

 It looks like the same issue as the one we recently solved in #31515.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 Replying to [comment:23 phw]:
 > Replying to [comment:22 karsten]:
 > > Okay, please let me know when I need to do something on
 colchicifolium's side.
 > [[br]]
 > The rsync is done from our side. We are already trying to sync data to
 `/srv/collector.torproject.org/collector/in/bridgedb-stats/`.

 Hmm, I changed something on colchicifolium's side to accept the rsync, but
 it doesn't seem to work. Is there anything in the log on your side?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  karsten
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-
Changes (by phw):

 * owner:  phw => karsten
 * status:  merge_ready => assigned


Comment:

 Replying to [comment:21 cohosh]:
 > These changes look good to me.
 [[br]]
 Merged to [https://gitweb.torproject.org/project/bridges/bridgedb-
 admin.git/commit/?id=7b534da219c25bb48fe511288f8e3289cd643d4f master].
 [[br]]
 > Since all logs (including previously rotated ones) are synced each time
 with rsync, is there a way to detect if old logs have been corrupted and
 are overwriting the previously synced logs? Not sure how we want to handle
 a case where logs that have previously been synced have changed for some
 reason, or what the easiest way to deal with this is.
 [[br]]
 I left the patch as it is according to Karsten's suggestion. I also
 reassigned the ticket to Karsten because BridgeDB's side is looking good
 now.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:22 karsten]:
 > Okay, please let me know when I need to do something on colchicifolium's
 side.
 [[br]]
 The rsync is done from our side. We are already trying to sync data to
 `/srv/collector.torproject.org/collector/in/bridgedb-stats/`.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 Replying to [comment:21 cohosh]:
 > These changes look good to me.

 Okay, please let me know when I need to do something on colchicifolium's
 side.

 > Since all logs (including previously rotated ones) are synced each time
 with rsync, is there a way to detect if old logs have been corrupted and
 are overwriting the previously synced logs? Not sure how we want to handle
 a case where logs that have previously been synced have changed for some
 reason, or what the easiest way to deal with this is.

 Thanks for thinking about such problems beforehand. In this case I think
 it's fine to just rsync what's on the BridgeDB host to colchicifolium. We
 can still decide on colchicifolium to not overwrite previously imported
 statistics, which I think is what we do with all other files. I'd say
 let's give it a try, and we can change this later if this turns out to be
 an issue.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  merge_ready
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-
Changes (by cohosh):

 * status:  needs_review => merge_ready


Comment:

 These changes look good to me.

 Since all logs (including previously rotated ones) are synced each time
 with rsync, is there a way to detect if old logs have been corrupted and
 are overwriting the previously synced logs? Not sure how we want to handle
 a case where logs that have previously been synced have changed for some
 reason, or what the easiest way to deal with this is.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  needs_review
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:  cohosh   |Sponsor:
 |  Sponsor30-must
-+-
Changes (by phw):

 * status:  needs_information => needs_review
 * reviewer:   => cohosh


Comment:

 Replying to [comment:19 karsten]:
 > Sure, that works, too. How about
 `/srv/collector.torproject.org/collector/in/bridgedb-stats/`? I'll have to
 edit a script on the receiving side, but feel free to set up the rsync on
 the sending side whenever you're ready.
 [[br]]
 Thanks. I updated the script on polyanthum's side. It will sync all
 available bridgedb-metrics.log files, including the rotated ones. The
 format of rotated files is the same as for assignments.log: `bridgedb-
 metrics.log-MMDD.gz`, e.g., `bridgedb-metrics.log-20190905.gz`. The
 file bridgedb-metrics.log is written once per day and also rotated once a
 day. For now, I configured logrotate to retain 30 rotated files, mostly as
 a precaution, so we don't lose data in case we run into trouble.

 All the changes I made are in the [https://gitweb.torproject.org/user/phw
 /bridgedb-admin.git/log/?h=task/19332 task/19332] branch of my bridgedb-
 admin repository. [https://gitweb.torproject.org/user/phw/bridgedb-
 admin.git/diff/?h=task%2F19332=master=3=1=0 Here's
 a patch]. Cecylia, can you please review these changes when you get a
 chance?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  needs_information
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 Sure, that works, too. How about
 `/srv/collector.torproject.org/collector/in/bridgedb-stats/`? I'll have to
 edit a script on the receiving side, but feel free to set up the rsync on
 the sending side whenever you're ready.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  needs_information
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-
Changes (by phw):

 * status:  assigned => needs_information


Comment:

 Replying to [comment:12 karsten]:
 > Replying to [comment:11 phw]:
 > > Replying to [comment:10 karsten]:
 > > > #9316 has been resolved recently which unblocks this ticket, AIUI.
 > > [[br]]
 > > Yes. How would you like me to expose the metrics file on BridgeDB's
 host? Should it be available over HTTPS? Or do you want me to rsync it to
 another host?
 >
 > Is there anything sensitive in the file that would have to be sanitized
 on the CollecTor host? If so, we should rsync it over ssh to
 colchicifolium. But if not, the preferred way would be to expose it on the
 BridgeDB host, so that others can fetch it, too.
 [[br]]
 Change of plan: Can we instead rsync BridgeDB's metrics to colchicifolium?
 Weasel isn't a fan of the idea of exposing BridgeDB's metrics on
 polyanthum. If CollecTor is archiving the metrics anyway, we might as well
 just sync them to colchicifolium.

 If you are ok with this, I just need a directory to sync the metrics to.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:  phw
 Type:  enhancement  | Status:
 |  assigned
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-
Changes (by gaba):

 * owner:  metrics-team => phw
 * status:  new => assigned


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, anti-   |  Actual Points:
  censorship-roadmap-september   |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-
Changes (by gaba):

 * keywords:  metrics-roadmap-2019-q2, ex-sponsor-19 => metrics-
 roadmap-2019-q2, anti-censorship-roadmap-september


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, ex- |  Actual Points:
  sponsor-19 |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:14 karsten]:
 > Any updates here? If possible, I'd like to work on this next week.
 Thanks!
 [[br]]
 Not yet. I'll try to get it done by early next week.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, ex- |  Actual Points:
  sponsor-19 |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 Any updates here? If possible, I'd like to work on this next week. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, ex- |  Actual Points:
  sponsor-19 |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:12 karsten]:
 > Replying to [comment:11 phw]:
 > > Replying to [comment:10 karsten]:
 > > > #9316 has been resolved recently which unblocks this ticket, AIUI.
 > > [[br]]
 > > Yes. How would you like me to expose the metrics file on BridgeDB's
 host? Should it be available over HTTPS? Or do you want me to rsync it to
 another host?
 >
 > Is there anything sensitive in the file that would have to be sanitized
 on the CollecTor host? If so, we should rsync it over ssh to
 colchicifolium. But if not, the preferred way would be to expose it on the
 BridgeDB host, so that others can fetch it, too.
 [[br]]
 There's nothing sensitive. We're doing the sanitisation ourselves and have
 [https://lists.torproject.org/pipermail/tor-dev/2019-July/013953.html
 published the data before]. I'll look into exposing the files over our
 apache.
 [[br]]
 > Here's another question, similar to the one about Snowflake stats: Would
 it be possible to expose more than just the latest BridgeDB statistics?
 Something like 7 or 14 days, or if it's not much data, everything until it
 gets too big?
 [[br]]
 Yes, that's definitely feasible. One week worth of data shouldn't be more
 than ~100 KB. I'll look into logrotate, so we can expose multiple weeks
 worth of data at any given time.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, ex- |  Actual Points:
  sponsor-19 |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 Replying to [comment:11 phw]:
 > Replying to [comment:10 karsten]:
 > > #9316 has been resolved recently which unblocks this ticket, AIUI.
 > [[br]]
 > Yes. How would you like me to expose the metrics file on BridgeDB's
 host? Should it be available over HTTPS? Or do you want me to rsync it to
 another host?

 Is there anything sensitive in the file that would have to be sanitized on
 the CollecTor host? If so, we should rsync it over ssh to colchicifolium.
 But if not, the preferred way would be to expose it on the BridgeDB host,
 so that others can fetch it, too.

 Here's another question, similar to the one about Snowflake stats: Would
 it be possible to expose more than just the latest BridgeDB statistics?
 Something like 7 or 14 days, or if it's not much data, everything until it
 gets too big?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, ex- |  Actual Points:
  sponsor-19 |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by phw):

 Replying to [comment:10 karsten]:
 > #9316 has been resolved recently which unblocks this ticket, AIUI.
 [[br]]
 Yes. How would you like me to expose the metrics file on BridgeDB's host?
 Should it be available over HTTPS? Or do you want me to rsync it to
 another host?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+-
 Reporter:  mrphs|  Owner:
 |  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2, ex- |  Actual Points:
  sponsor-19 |
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:
 |  Sponsor30-must
-+-

Comment (by karsten):

 #9316 has been resolved recently which unblocks this ticket, AIUI.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #19332 [Metrics/CollecTor]: Add a BridgeDB module (was: Publish BridgeDB stats)

[Tor Bug Tracker & Wiki]

#19332: Add a BridgeDB module
-+--
 Reporter:  mrphs|  Owner:  metrics-team
 Type:  enhancement  | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Metrics/CollecTor|Version:
 Severity:  Normal   | Resolution:
 Keywords:  metrics-roadmap-2019-q2  |  Actual Points:
Parent ID:   | Points:  8
 Reviewer:   |Sponsor:  Sponsor19
-+--
Changes (by irl):

 * status:  needs_information => new
 * parent:  #9316 =>
 * cc: ahf, dgoulet, phw (added)
 * points:  1 => 8
 * keywords:  metrics => metrics-roadmap-2019-q2
 * type:  defect => enhancement


Old description:

> While [https://lists.torproject.org/pipermail/tor-
> project/2016-June/000424.html talking about making a PT infographic] and
> interesting stats to point out, asn pointed out that there are no
> publicly available BridgeDB stats.
>
> This can be useful in many different fronts, such as seeing the balance
> between usage and available resources.

New description:

 While [https://lists.torproject.org/pipermail/tor-
 project/2016-June/000424.html talking about making a PT infographic] and
 interesting stats to point out, asn pointed out that there are no publicly
 available BridgeDB stats.

 This can be useful in many different fronts, such as seeing the balance
 between usage and available resources.

 This ticket is about adding a CollecTor module that will archive the stats
 exposed in #9316. The actual stats and format should be discussed in #9316
 and may benefit from the discussion in #29315.

--

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs