Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  closed
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:  fixed
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by dcf):

 * status:  needs_review => closed
 * resolution:   => fixed


Old description:

> After #21304, in the [http://meetbot.debian.net/tor-meeting/2019/tor-
> meeting.2019-04-18-20.01.log.html 2019-04-18] team checkin, we agreed to
> delete the pre-sanitization logs by 2019-06-01. Before then, we have a
> chance to extract any useful sanitized information from them.
>
> Cf. comment:8:ticket:30125 regarding the location of logs.
>
> === Broker
>
> On the broker, there are three files to delete:
>
>  /var/log/snowflake-broker/unstanitized.tar::
>Contains three files:
> * '''unsanitized.tar.1''', pre-rotation logs from 2017-07-15 to
> 2019-03-22 and rotated logs from 2017-03-22 to 2019-04-16.
> * '''@40005cb771341453839c.s''', logs from 2019-04-16 to
> 2019-04-17.
> * '''current''', empty file.
>  /var/log/snowflake-broker/mv:: Probably the result of a typo, it is a
> tar file that contains a single file '''unsanitized.tar''' which is
> identical to the '''unsanitized.tar.1''' inside '''/var/log/snowflake-
> broker/unstanitized.tar'''.
>  /var/log/snowflake-broker/x.tar:: contains a subset of what '''/var/log
> /snowflake-broker/mv''' contains.
>
> I think the most complete logs come from '''/var/log/snowflake-
> broker/unstanitized.tar''', extracting the contained
> '''unsanitized.tar.1''' and combining it with the contained
> '''@40005cb771341453839c.s'''.
>
> === snowflake-server
>
>  /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18 to
> 2019-04-11.
>
> === proxy-go
>
>  /home/snowflake-proxy/unsanitized_old.tar:: contains 31 individual log
> files:
>   * snowflake-proxy-17h.log
>   * snowflake-proxy-23h.log
>   * snowflake-proxy-29h.log
>   * snowflake-proxy-restartless.log
>   * snowflake-proxy-standalone-17h.log
>   * snowflake-proxy-standalone-23h.log
>   * snowflake-proxy-standalone-29h.log
>   * snowflake-proxy-10h.log.xz
>   * snowflake-proxy-17h.20180705.log.xz
>   * snowflake-proxy-17h.20181121.log.xz
>   * snowflake-proxy-1h.log.xz
>   * snowflake-proxy-23h.20180705.log.xz
>   * snowflake-proxy-23h.20181121.log.xz
>   * snowflake-proxy-29h.20180705.log.xz
>   * snowflake-proxy-29h.20181121.log.xz
>   * snowflake-proxy-2h.log.xz
>   * snowflake-proxy-a.log.xz
>   * snowflake-proxy-b.log.xz
>   * snowflake-proxy-c.log.xz
>   * snowflake-proxy-standalone-17h.20180705.log.xz
>   * snowflake-proxy-standalone-17h.20181121.log.xz
>   * snowflake-proxy-standalone-23h.20180705.log.xz
>   * snowflake-proxy-standalone-23h.20181121.log.xz
>   * snowflake-proxy-standalone-29h.20180705.log.xz
>   * snowflake-proxy-standalone-29h.20181121.log.xz
>   * snowflake-proxy-standalone-a.log.xz
>   * snowflake-proxy-standalone-b.log.xz
>   * snowflake-proxy-standalone-c.log.xz
>   * snowflake-proxy-test-10h.log.xz
>   * snowflake-proxy-test-1h.log.xz
>   * snowflake-proxy-test-2h.log.xz

New description:

 After #21304, in the [http://meetbot.debian.net/tor-meeting/2019/tor-
 meeting.2019-04-18-20.01.log.html 2019-04-18] team checkin, we agreed to
 delete the pre-sanitization logs by 2019-06-01. Before then, we have a
 chance to extract any useful sanitized information from them.

 Cf. comment:8:ticket:30125 regarding the location of logs.

 === Broker

 On the broker, there are three files to delete:

  /var/log/snowflake-broker/unstanitized.tar::
Contains three files:
 * '''unsanitized.tar.1''', pre-rotation logs from 2017-07-15 to
 2019-03-22 and rotated logs from 2017-03-22 to 2019-04-16.
 * '''@40005cb771341453839c.s''', logs from 2019-04-16 to
 2019-04-17.
 * '''current''', empty file.
  /var/log/snowflake-broker/mv:: Probably the result of a typo, it is a tar
 file that contains a single file '''unsanitized.tar''' which is identical
 to the '''unsanitized.tar.1''' inside '''/var/log/snowflake-
 broker/unstanitized.tar'''.
  /var/log/snowflake-broker/x.tar:: contains a subset of what '''/var/log
 /snowflake-broker/mv''' contains.

 I think the most complete logs come from '''/var/log/snowflake-
 broker/unstanitized.tar''', extracting the contained
 '''unsanitized.tar.1''' and combining it with the contained
 '''@40005cb771341453839c.s'''.

 === snowflake-server

  /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18 to
 2019-04-11.

 === proxy-go
  

Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+--
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  needs_review
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+--
Changes (by dcf):

 * status:  new => needs_review


Comment:

 I have prepared a candidate sanitized CSV file extracted from the
 sanitized logs. I've placed it in /var/log/snowflake-broker/broker.csv.xz
 for evaluation. It's 6 MB compressed but 1.8 GB uncompressed.

 The sanitized CSV looks like this:
 {{{
 timestamp,event,proxyid,clientid,additional
 2017-07-21 23:40:00,proxy-gets-none,157,,no-clients
 2017-07-21 23:40:00,proxy-polls,157,,
 2017-07-21 23:40:00,proxy-gets-none,149,,no-clients
 2017-07-21 23:40:00,proxy-polls,149,,
 2017-07-21 23:40:00,client-offers,,,
 2017-07-21 23:40:00,proxy-gets-offer,,,
 2017-07-21 23:40:00,proxy-answers,,,
 2017-07-21 23:40:00,client-gets-answer,,,
 2017-07-21 23:40:00,proxy-polls,160,,
 2017-07-21 23:40:00,proxy-gets-none,159,,no-clients
 2017-07-21 23:40:00,proxy-polls,159,,
 2017-07-21 23:40:00,proxy-gets-none,157,,no-clients
 2017-07-21 23:40:00,proxy-polls,157,,
 }}}

 Timestamps are truncated to multiples of 10 minutes. Client and proxy IDs
 are replaced by sequential integers.

 The `event` column can take on these values:
  * `start` the broker was restarted.
  * `client-offers` a client connects, sends an offer, and awaits an
 answer.
  * `client-gets-answer` a client receives a proxy's answer (successful
 broker match).
  * `client-gets-none` a client disconnects without receiving an answer,
 whether because of a timeout or because there were no proxies.
  * `proxy-polls` a proxy connects in order to receive an offer.
  * `proxy-gets-none` a proxy disconnects without receiving a client offer
 (no clients).
  * `proxy-gets-offer` a proxy receives a client offer.
  * `proxy-answers` a proxy sends an answer to the broker.
  * `error` an error; the most common is "http: TLS handshake error". Other
 possibilities are "http2: server: error", "http2: received GOAWAY", or a
 bad HTTP request. The `additional` column distinguishes these cases.

 Some of these have relations to each other. For example `proxy-polls` ≈
 `proxy-gets-none` + `proxy-gets-answer`.

 Using the sanitized CSV, I made a couple of graphs. The first shows shows
 the number of broker outcomes per day, where an outcome is one of the four
 possibilities:
  * A client and proxy are successfully linked up.
  * A proxy connects but doesn't get a client.
  * A client connects but doesn't get a proxy.
  * Some other error occurred.

 Click to embiggen.
 [[Image(broker-interactions.png,100%)]]

 The second graph shows the estimated number of proxies. This is just 10 ×
 `proxy-polls` / s. It's based on the assumption that each proxy polls
 every 10 s. The assumption doesn't hold when there are actually clients,
 but as you can see the estimate is pretty close to 3, which is the number
 of fallback proxy-go instances.

 Click to embiggen.
 [[Image(broker-estimated-proxies.png,100%)]]

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by dcf):

 * Attachment "broker-estimated-proxies.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by dcf):

 * Attachment "broker-interactions.png" added.


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by dcf):

 * Attachment "broker-logs.zip" added.

 Programs to process and visualize broker logs.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+
Changes (by dcf):

 * cc: arlolra, phw, cohosh (added)


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30693 [Circumvention/Snowflake]: Delete old unsanitized logs

[Tor Bug Tracker & Wiki]

#30693: Delete old unsanitized logs
-+
 Reporter:  dcf  |  Owner:  (none)
 Type:  task | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Circumvention/Snowflake  |Version:
 Severity:  Normal   | Resolution:
 Keywords:   |  Actual Points:
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+

Comment (by dcf):

 Replying to [ticket:30693 dcf]:
 > === snowflake-server
 >
 >  /var/log/tor/unsanitized.tar:: contains rotated logs dated 2019-02-18
 to 2019-04-11.

 This one is easy. We can just delete the file. We were never permanently
 storing these logs and they would have been automatically rotated and
 deleted anyway.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs