Re: [tor-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
---+-
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  closed
 Priority:  Medium |  Milestone:  CollecTor 1.7.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:  fixed
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer:  irl|Sponsor:
---+-
Changes (by karsten):

 * milestone:   => CollecTor 1.7.0


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
---+-
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  closed
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:  fixed
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer:  irl|Sponsor:
---+-
Changes (by karsten):

 * status:  merge_ready => closed
 * resolution:   => fixed


Comment:

 Great! Fixed in master. Closing. Thanks!

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
---+-
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  merge_ready
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer:  irl|Sponsor:
---+-
Changes (by irl):

 * status:  needs_review => merge_ready


Comment:

 Looks good to me.

 I don't think there is a need to combine the tar and xz steps into a
 single command, it's more readable as separate steps. There would probably
 be a speed up due to less disk I/O to be had if we really needed it.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
---+--
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  needs_review
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--

Comment (by iwakeh):

 Is it lighter on memory when doing two separate steps?  I.e. replace the
 tar and xz command by `tar -I "xz -9e" -chf ${TARBALLS[$i]}.tar.xz`?

 But the solution seems ok in general, no matter if the tar step
 is on or two lines.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
---+--
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  assigned
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal |   Keywords:
Actual Points: |  Parent ID:
   Points: |   Reviewer:
  Sponsor: |
---+--
 Today I wondered why some of our 2018-05 tarballs are over 3 days old even
 though we're creating new tarballs every 2 days. And I wondered even more
 after finding a seemingly completed run from yesterday. Here's what
 happened:

  - The May 19 run succeeded without issues.
  - The May 21 run was interrupted, probably due to a host reboot.
 Apparently, it finished compressing tarballs except for bridge extra-info
 descriptors, but it did not move any of them in place.
  - The May 23 run went through, but it did not compress any tarballs
 except for bridge extra-info descriptors.

 I think this is related to
 [https://gitweb.torproject.org/collector.git/tree/src/main/resources
 /create-tarballs.sh#n102 line 102 in the script]:

 {{{
 for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
   echo `date` "Creating" ${TARBALLS[$i]}'.tar'
   tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
   if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then   # <- this one
 echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
 xz -9e ${TARBALLS[$i]}.tar
   fi
 done
 }}}

 Explanation: there were still compressed files in the working directory
 from May 21, and as a result we did not attempt to compress new tarballs
 from May 23.

 Suggested fix: remove that `if` statement and compress the tarball
 regardless of whether a previous compressed tarball exists in the working
 directory using `xz -9e -f`:

 {{{
 diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources
 /create-tarballs.sh
 index cd16b2dc..d247c520 100755
 --- a/src/main/resources/create-tarballs.sh
 +++ b/src/main/resources/create-tarballs.sh
 @@ -99,10 +99,8 @@ done
  for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do
echo `date` "Creating" ${TARBALLS[$i]}'.tar'
tar chf ${TARBALLS[$i]}.tar ${TARBALLS[$i]}
 -  if [ ! -f ${TARBALLS[$i]}.tar.xz ]; then
 -echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
 -xz -9e ${TARBALLS[$i]}.tar
 -  fi
 +  echo `date` "Compressing" ${TARBALLS[$i]}'.tar'
 +  xz -9e -f ${TARBALLS[$i]}.tar
  done

  cd $OUTDIR/webstats/
 }}}

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #26193 [Metrics/CollecTor]: Tarballs are not compressed in a run following an aborted run

[Tor Bug Tracker & Wiki]

#26193: Tarballs are not compressed in a run following an aborted run
---+--
 Reporter:  karsten|  Owner:  karsten
 Type:  defect | Status:  needs_review
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID: | Points:
 Reviewer: |Sponsor:
---+--
Changes (by karsten):

 * status:  assigned => needs_review


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs