Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  metrics-team
 Type:  enhancement| Status:  assigned
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-2018   |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-

Comment (by karsten):

 Looks like we had two tickets for this issue. Here's what I wrote in
 #31866 which I'm now closing as duplicate:

 We're currently generating tarballs using the `tar` and `xz` command-line
 tools triggered by a cronjob. While this is very fast, it doesn't
 integrate that well with the rest of our code. For example, it would be
 much easier to extract descriptor types, publication times, and file
 digests for #31204 if tarball generation happened in Java.

 One possible issue might be that generated tarballs are larger or that
 compression takes longer. This is something I wanted to figure out early,
 which is why I ran some tests today:

 ||= Compression preset level == `xz` == XZ for Java 1.6 =||
 ||  1  || 269M || 1m27.522s || 269M || 1m22.905s ||
 ||  3  || 77M || 1m6.590s || 77M || 1m15.03s ||
 ||  6  || 30M || 3m8.426s || 30M || 4m54.837s ||
 ||  9  || 18M || 2m56.801s || 18M || 5m6.998s ||
 ||  9e  || 16M || 7m2.364s  NA ||

 We're currently using `xz -9e`, but I can't find this option in XZ for
 Java. The closest is compression preset level 9. That means that our
 tarballs would be 18M/16M = 12.5% larger and be created in
 306.998s/422.364s = 73% of the current time.

 Is it a blocker that our tarballs would be 12.5% larger? If so, we might
 try harder to configure XZ for Java in the same way as `xz -9e` operates,
 even though that would very likely increase tarball generation time.

 Not working on this at the moment, just leaving my thoughts here for
 discussion and for picking this up as time permits.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  metrics-team
 Type:  enhancement| Status:  assigned
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-2018   |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-

Comment (by iwakeh):

 The implementation should take the situation of #26193 into account and
 provide a solution, too.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  metrics-team
 Type:  enhancement| Status:  assigned
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-2018   |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-

Comment (by iwakeh):

 This create-tar code should call index-creation whenever reasonable (see
 patch in ticket #20351).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  metrics-team
 Type:  enhancement| Status:  assigned
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-2018   |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-
Changes (by iwakeh):

 * owner:  iwakeh => metrics-team
 * status:  accepted => assigned


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  iwakeh
 Type:  enhancement| Status:  accepted
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-2018   |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-
Changes (by iwakeh):

 * owner:  metrics-team => iwakeh
 * reviewer:  iwakeh =>
 * status:  assigned => accepted


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  metrics-team
 Type:  enhancement| Status:  assigned
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords:  metrics-2018   |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer:  iwakeh |Sponsor:
---+-
Changes (by iwakeh):

 * reviewer:   => iwakeh


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: Replace create-tarball.sh shell script with Java module (was: replace create-tarball.sh shell script with java module)

[Tor Bug Tracker & Wiki]

#20350: Replace create-tarball.sh shell script with Java module
---+-
 Reporter:  iwakeh |  Owner:  (none)
 Type:  enhancement| Status:  new
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-

Comment (by karsten):

 Capitalize some words in the summary.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: replace create-tarball.sh shell script with java module

[Tor Bug Tracker & Wiki]

#20350: replace create-tarball.sh shell script with java module
---+-
 Reporter:  iwakeh |  Owner:  (none)
 Type:  enhancement| Status:  new
 Priority:  Medium |  Milestone:  CollecTor 2.0.0
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-
Changes (by iwakeh):

 * milestone:   => CollecTor 2.0.0


Old description:

> This [https://gitweb.torproject.org/collector.git/tree/src/main/resources
> /create-tarballs.sh script's] should be transferred to java.
>
> The new `createtars` module should:
>
> * provide at least the functinality of the script
> * be configurable as other CollecTor modules
> * not impede other modules
>
> Please collect more features and functionality that the script
> can't/doesn't provide, but which should be part of this module in the
> comments below.

New description:

 This [https://gitweb.torproject.org/collector.git/tree/src/main/resources
 /create-tarballs.sh script's] should be transferred to java.

 The new `createtars` module should:

 * provide at least the functionality of the script
 * be configurable as other CollecTor modules
 * not impede other modules

 Please collect more features and functionality that the script
 can't/doesn't provide, but which should be part of this module in the
 comments below.

--

Comment:

 This module should also remove files from 'out' once they are archived.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #20350 [Metrics/CollecTor]: replace create-tarball.sh shell script with java module

[Tor Bug Tracker & Wiki]

#20350: replace create-tarball.sh shell script with java module
---+-
 Reporter:  iwakeh |  Owner:
 Type:  enhancement| Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal | Resolution:
 Keywords: |  Actual Points:
Parent ID:  #20518 | Points:
 Reviewer: |Sponsor:
---+-
Changes (by iwakeh):

 * parent:   => #20518


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #20350 [Metrics/CollecTor]: replace create-tarball.sh shell script with java module

[Tor Bug Tracker & Wiki]

#20350: replace create-tarball.sh shell script with java module
---+-
 Reporter:  iwakeh |  Owner:
 Type:  enhancement| Status:  new
 Priority:  Medium |  Milestone:
Component:  Metrics/CollecTor  |Version:
 Severity:  Normal |   Keywords:
Actual Points: |  Parent ID:
   Points: |   Reviewer:
  Sponsor: |
---+-
 This [https://gitweb.torproject.org/collector.git/tree/src/main/resources
 /create-tarballs.sh script's] should be transferred to java.

 The new `createtars` module should:

 * provide at least the functinality of the script
 * be configurable as other CollecTor modules
 * not impede other modules

 Please collect more features and functionality that the script
 can't/doesn't provide, but which should be part of this module in the
 comments below.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs