Re: Data patches during upgrade

[Naama Shoresh]

I understand.
Thanks.

On Tue, Jun 13, 2017 at 4:08 PM, Dewayne Richardson 
wrote:

> The reason the migrations manage "structure" is because of rollbacks with
> "goose Up and Down" (not saying "data" can't be rolled back, but it gets
> trickier, what if I screw up the "where" clause and inadvertently remove
> other data outside of the scope of the migration unintentionally?).
>
> Migrations managing only structure also helps with "separation".  I can
> apply "test" or "integration" data to the same structure, without having to
> run the migrations.  Test data doesn't always align with seed data because
> I might need to artificially manipulate the data differently depending on
> the test scenario I want to achieve.  If migrations have placed data in the
> database when I need to test, now I have to remove that static data before
> I even start.  Yes, we do that now with the test data but the test data is
> preditable.
>
> We have this problem today with the migrations (just managing the
> structure).  If I'm working in a branch and commit a migration with a
> migration file timestamp of yesterday, then you are also working on a
> different migration with a migration timestamp of today and commit and push
> before I do, then the timestamp sequencing gets misaligned.  Once goose
> runs it "bookmarks" your migration in the goose_db_version table in front
> of mine, which causes goose to ignore my migration, which forces me to
> rename that file to a later timestamp just so goose will see it.
>
> So, if we "interleave" seed data (over time), managing that data order on
> top of the structure management in goose will add yet another level of
> complexity.  This is the reason the db/admin.pl was created to help with
> this workflow.  I know it's not optimal, but these are the hurdles we hit
> as we were figuring this stuff out.
>
> -Dewayne
>
> On Mon, Jun 12, 2017 at 10:14 PM, Naama Shoresh  wrote:
>
> > Hi,
> >
> > I want to suggest a slightly different approach.
> > Goose is the brain managing the DB upgrade, right?
> > The data patches are part of the DB evolution, but today we can't use
> goose
> > to run them because we have seeds.sql in the middle.
> > What I suggest is turning seeds.sql into another migration script,
> > resulting in the following procedure:
> > 0) (In clean installations) Tables creation
> > 1) Goose migrations:
> > 1a) Schema changes
> > 1b) Data seeding (seeds.sql)
> > 1c) Data changes
> >
> > Going forward, I believe a data change migration script will be attached
> to
> > most schema changes, instead of populating the DB in the seeds.sql.
> >
> > The benefits come from the fact that for future changes, the order
> > presented above (1a, 1b, 1c) is not strict.
> > Future schema/data changes are expressed in a single migration script,
> > containing all relevant operations.
> > This ensures that whatever change is needed (schema/data/both), and
> > whatever the change depends on, it can be handled by a single Goose
> > migration script.
> >
> > What do you think?
> >
> >
> >
> > On Fri, Jun 9, 2017 at 6:35 PM, Dewayne Richardson 
> > wrote:
> >
> > > Yea, it's just a new feature to admin.pl to support data conversions,
> to
> > > keep the migrations clean.  Derek and I have been working through it.
> > >
> > > -Dew
> > >
> > > On Thu, Jun 8, 2017 at 7:40 AM, Jeremy Mitchell  >
> > > wrote:
> > >
> > > > This seems to make sense to me but honestly, I'd probably defer to
> > > Dewayne.
> > > >
> > > > In theory, it would be nice if migrations only included "structural"
> > > > changes (new tables, columns, changing column types or not  null,
> etc)
> > > and
> > > > seeds focused on the "base" (or the minimum required) static data
> > > required
> > > > of TO (types, statuses, roles, etc) and then yea, putting data fixing
> > or
> > > > data massaging as the last step makes sense to me. But you know what
> > they
> > > > say about theory...
> > > >
> > > > +1
> > > >
> > > > Jeremy
> > > >
> > > > On Wed, Jun 7, 2017 at 8:41 AM, Gelinas, Derek <
> > > derek_geli...@comcast.com>
> > > > wrote:
> > > >
> > > > > I'm adding a feature to traffic ops that creates a new column in
> > > > > steering_target called type, that is populated with type ids from
> the
> > > > type
> > > > > table.  Using admin.pl upgrade, the column is created in
> migrations,
> > > and
> > > > > the two types for this table are populated by seeds.sql.  None of
> > this
> > > is
> > > > > out of the ordinary.  Unfortunately I also need to populate the
> type
> > > > column
> > > > > based on data that isn't in there until after seeds.sql is run, so
> I
> > > > can't
> > > > > place this into the migration.  Seeds.sql needs to run after the
> > > > migration
> > > > > due to any structural changes that happen there.
> > > > >
> > > > > Dewayne and I have discussed this a bit this morning, and we're
> > > thinking
> > > > > the best 

Re: [VOTE] Release Apache Traffic Control 2.0.0-incubating (RC6)

[Dewayne Richardson]

+1

- gpg sig is good
- checksums match source tarball
- tarball has correct name and structure
- rpms build from tarball using 'pkg' command
- installed using those rpms and ran postinstall on a clean Centos7 VM
- imported default profiles from here:
http://trafficcontrol.incubator.apache.org/downloads/profiles/2.0.x/default/
- some basic traffic_ops UI functionality


-Dewayne

On Tue, Jun 20, 2017 at 2:04 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Correction to the original vote deadline:
> Instead of Friday June 21, 2017, I meant that the vote will run until
> Wednesday June 21, 2017 (72 hrs)
>
> Please get your votes in within the next day or so- we need at least one
> more +1 vote to pass the release on to the IPMC
>
> Thanks,
> Eric
>
>
>
> > On Jun 19, 2017, at 5:07 PM, Dave Neuman  wrote:
> >
> > +1
> > Things I tested (Centos 7.2 VM):
> > -- verified md5
> > -- verified gpg sig
> > -- built all RPMs from source tar
> > -- Traffic Stats installs, starts, and runs
> > -- Traffic Router installs, starts, serves digs and curls
> > -- Traffic Ops can be installed and started.
> >
> > I found a couple issues that should be addressed before the next release:
> > [1] Traffic Stats install can't create traffic_stats user.
> >
> > Thanks,
> > Dave
> >
> > [1] https://issues.apache.org/jira/browse/TC-397
> >
> >
> > On Mon, Jun 19, 2017 at 12:39 PM, Dan Kirkwood 
> wrote:
> >
> >> +1
> >>
> >> I checked:
> >> - git tag verified
> >> - gpg sig is good
> >> - checksums match source tarball
> >> - tarball has correct name and structure
> >> - rpms build from tarball
> >> - traffic_ops installation and postinstall on a clean Centos7 VM
> >> - some basic traffic_ops UI functionality
> >> - docs no longer mention build area to download rpms
> >>
> >> One caveat -- gpg signature is not signed and so not in the web of
> >> trust,  but according to
> >> http://www.apache.org/dev/release-distribution.html#sigs-and-sums :
> >>
> >> "Signing keys SHOULD be linked into a strong web of trust."
> >>
> >> We should get Eric's key signed at the earliest opportunity,  but it's
> >> not a requirement for the release.
> >>
> >> On Fri, Jun 16, 2017 at 10:31 AM, Eric Friedrich (efriedri)
> >>  wrote:
> >>> Hello All,
> >>>
> >>> I've prepared the next candidate release for incubator-trafficcontrol
> >> v2.0.0 (RC6)
> >>>
> >>> Changes since 1.8.1:
> >>> https://github.com/apache/incubator-trafficcontrol/
> >> compare/RELEASE-1.8.1...RELEASE-2.0.0-RC6 >> github.com/apache/incubator-trafficcontrol/compare/
> >> RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC5>
> >>>
> >>> This corresponds to git:
> >>> Hash: 85d14ebe2a4ac71236f86f70349481d2b3dc784b
> >>> Tag: RELEASE-2.0.0-RC6
> >>>
> >>> Which can be verified with the following:git tag -v
> RELEASE-2.0.0-RC6
> >>>
> >>> My code signing key is available here:
> >>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
> >>>
> >>> and here:
> >>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
> >>>
> >>> Make sure you refresh from a key server to get all relevant signatures.
> >>>
> >>> The source .tar.gz file, pgp signature (.asc signed with my key from
> >>> above), and md5 and sha512 checksums are provided here:
> >>> https://dist.apache.org/repos/dist/dev/incubator/
> >> trafficcontrol/2.0.0/RC6
> >>>
> >>>
> >>> Docs are available here: https://trafficcontrol.apache.org/docs/2.0.x/
> >>>
> >>>
> >>> The vote will remain open until Friday, June 21, 2017.
> >>>
> >>>
> >>> Thanks,
> >>> Eric Friedrich
> >>
>
>

Re: [VOTE] Release Apache Traffic Control 2.0.0-incubating (RC6)

[Eric Friedrich (efriedri)]

Correction to the original vote deadline: 
Instead of Friday June 21, 2017, I meant that the vote will run until Wednesday 
June 21, 2017 (72 hrs)

Please get your votes in within the next day or so- we need at least one more 
+1 vote to pass the release on to the IPMC

Thanks,
Eric



> On Jun 19, 2017, at 5:07 PM, Dave Neuman  wrote:
> 
> +1
> Things I tested (Centos 7.2 VM):
> -- verified md5
> -- verified gpg sig
> -- built all RPMs from source tar
> -- Traffic Stats installs, starts, and runs
> -- Traffic Router installs, starts, serves digs and curls
> -- Traffic Ops can be installed and started.
> 
> I found a couple issues that should be addressed before the next release:
> [1] Traffic Stats install can't create traffic_stats user.
> 
> Thanks,
> Dave
> 
> [1] https://issues.apache.org/jira/browse/TC-397
> 
> 
> On Mon, Jun 19, 2017 at 12:39 PM, Dan Kirkwood  wrote:
> 
>> +1
>> 
>> I checked:
>> - git tag verified
>> - gpg sig is good
>> - checksums match source tarball
>> - tarball has correct name and structure
>> - rpms build from tarball
>> - traffic_ops installation and postinstall on a clean Centos7 VM
>> - some basic traffic_ops UI functionality
>> - docs no longer mention build area to download rpms
>> 
>> One caveat -- gpg signature is not signed and so not in the web of
>> trust,  but according to
>> http://www.apache.org/dev/release-distribution.html#sigs-and-sums :
>> 
>> "Signing keys SHOULD be linked into a strong web of trust."
>> 
>> We should get Eric's key signed at the earliest opportunity,  but it's
>> not a requirement for the release.
>> 
>> On Fri, Jun 16, 2017 at 10:31 AM, Eric Friedrich (efriedri)
>>  wrote:
>>> Hello All,
>>> 
>>> I've prepared the next candidate release for incubator-trafficcontrol
>> v2.0.0 (RC6)
>>> 
>>> Changes since 1.8.1:
>>> https://github.com/apache/incubator-trafficcontrol/
>> compare/RELEASE-1.8.1...RELEASE-2.0.0-RC6> github.com/apache/incubator-trafficcontrol/compare/
>> RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC5>
>>> 
>>> This corresponds to git:
>>> Hash: 85d14ebe2a4ac71236f86f70349481d2b3dc784b
>>> Tag: RELEASE-2.0.0-RC6
>>> 
>>> Which can be verified with the following:git tag -v RELEASE-2.0.0-RC6
>>> 
>>> My code signing key is available here:
>>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
>>> 
>>> and here:
>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>>> 
>>> Make sure you refresh from a key server to get all relevant signatures.
>>> 
>>> The source .tar.gz file, pgp signature (.asc signed with my key from
>>> above), and md5 and sha512 checksums are provided here:
>>> https://dist.apache.org/repos/dist/dev/incubator/
>> trafficcontrol/2.0.0/RC6
>>> 
>>> 
>>> Docs are available here: https://trafficcontrol.apache.org/docs/2.0.x/
>>> 
>>> 
>>> The vote will remain open until Friday, June 21, 2017.
>>> 
>>> 
>>> Thanks,
>>> Eric Friedrich
>> 

cachekey proposal

[Gelinas, Derek]

I've created a PR for cachekey support in Traffic Ops 2+.  I wanted to email 
the list because it's a departure from the norm for plugin configuration.  
Cachekey has several options available requiring multiple pparam entries.  
Rather than try and shoehorn that into the db with a single "cachekey options" 
line, I've created code to interpret parameters for the ds profile.  When 
parameters assigned to that delivery service's profile are found for 
cachekey.config, a pparam entry is created for each parameter and its value in 
the remap.config.  For example, the following parameters, when assigned to a 
delivery service profile:


Parameter name - "ua-capture"
Parameter config file - "cachekey.config"
Parameter value - "(Mozilla/[^\s])."

Parameter name - "include-headers"
Parameter config file - "cachekey.config"
Parameter value - "H1,H2"

Will create this entry in the remap.config for that delivery service.

@plugin=cachekey.so 
@pparam=--ua-capture=(Mozilla/[^\s]). 
@pparam=--include-headers=H1,H2

For anyone unfamiliar with delivery service profiles, these were created in 2.0 
as a way to add MSO parameters for 6.x trafficserver support.  They are useful 
for any situation in which you have a configuration that isn't necessarily used 
often and is more complex than a single text field/db column would allow.

Longer term, I'm thinking that a sub-interface could be added to the delivery 
service page (similar to the server page, or steering, etc) where these options 
could be more explicitly spelled out and modified more intuitively, but the 
full functionality is nonetheless there with this small change.

As always, I welcome your thoughts.

Derek

Re: DSCP marking from Mid->Edge

[Durfey, Ryan]

As an aside we also struggle with separating mid tier to origin logs for 
services using the same origin value.  Currently we are forced to create a 
cname for one of the services so the origin fqdns are different between the two 
services.  Marking services heading from mid to origin with a service value 
marker would also potentially solve that issue which would be helpful.

Ryan DurfeyM | 303-524-5099
CDN Support (24x7): 866-405-2993 or 
cdn_supp...@comcast.com


From: "Mike Sandman (misandma)" 
Reply-To: "dev@trafficcontrol.incubator.apache.org" 

Date: Monday, June 19, 2017 at 5:05 PM
To: "dev@trafficcontrol.incubator.apache.org" 

Subject: Re: DSCP marking from Mid->Edge

I think we do want it per DS (and not even per origin) since the asking 
customer will potentially have two DSes pointing to the same origin where one 
is for in-home traffic and one is for mobile, and they may want different QoS 
for each.

Thanks,
Mike Sandman

On 6/19/17, 5:00 PM, "Steve Malenfant" 
> wrote:

Mike,

Is there a requirement to have it configured per delivery service? If not,
would enabling global header rewrite in plugin.config would work for a
specific DSCP value?

Steve

On Mon, Jun 19, 2017 at 1:28 PM, Mike Sandman (misandma) 

> wrote:

> Hi all,
>
> I’m beginning work on a feature to enable DSCP marking for traffic between
> the mid and edge cache. So far the high level proposed design is:
>
>
> 1.   DSCP marking from Mid to Edge will be configurable on a per-DS
> basis (like DSCP marking from Edge to Client is today). This means the 
Mids
> will need to know what DS a particular request corresponds to (we have a
> use case for two DSes pointing to the same origin)
>
> 2.   Therefore, the Edge will need to write a new header
> (Forwarded:host= as per RFC 
7239)
> into the request that is sent up to the Mid
>
> 3.   The Mid will use a remap rule (like the Edge does today) to apply
> a DSCP config to responses where the request has a matching Forwarded 
header
>
> 4.   To accomplish this, Traffic Ops will need to generate header
> rewrite rules on the Edge and remap.config on the Mid to match this
> configuration scheme
>
> Any issues or improvements to this? Or will this work?
>
> Thanks,
> Mike Sandman
>