This is the records.config setting: proxy.config.url_remap.pristine_host_hdr. The vhost config on the origin would need to match the CNAME.
> On Jan 31, 2019, at 10:30 PM, John Rushford <[email protected]> wrote: > > origin virtual hosting relies on the Host header. So if the request from the > client to the reverse > proxy has a Host header and maintain_pristine_host_hdr is not enabled, remap > will rewrite the Host header along with the request url. So in this case, > remapping to > a CNAME should work. > >> On Jan 29, 2019, at 6:02 PM, Steve Malenfant <[email protected]> wrote: >> >> CNAME might not necessary work if the origin is using virtual hosting. It >> would need to be configured as well. Might work with some origin? But not >> all of them. >> >> On Tue, Jan 29, 2019 at 7:40 PM Fieck, Brennan <[email protected]> >> wrote: >> >>> The issue is inherent with the structure of ATS configuration files. If I >>> understand correctly - someone correct me if I don't - >>> you can't have more than one rule for a "primary destination" in the >>> parent.config. Or, you can, but only one of them will >>> wind up catching all of your matches. Which means that if you have >>> multiple Delivery Services that use the same origin FQDN >>> (dest_domain) then you get proper behavior out of ATS if and only if the >>> resulting rules would be identical. I'd highly >>> recommend that you not move forward using duplicate origin FQDNs on your >>> Delivery Services, because it probably won't >>> do what you want. >>> >>> I'm not sure what you mean about the cachekeys. The Delivery Service URLs >>> should still be unique, so you shouldn't run >>> into a collision in the cache keys. >>> >>> Origins are an object. They're also a cachegroup type. And a profile type. >>> And a server type. And a URL field on a Delivery Service. >>> >>> The solution that's been used in the past to use the same actual origin >>> machine to service multiple Delivery Services safely >>> is to add a "Canonical Name" DNS record to the ATC internal zone for each >>> Delivery Service. The idea is to have a unique name >>> for each Delivery Service, but they all resolve to the same IP address. >>> ________________________________________ >>> From: Steve Malenfant <[email protected]> >>> Sent: Tuesday, January 29, 2019 4:32 PM >>> To: [email protected] >>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery Services >>> produces indeterminate parent.config >>> >>> We use and will use the same origin for multiple DS. It’s an inherent part >>> of the design of some part of our delivery. >>> >>> Wouldn’t be less of a problem using cachekeys for different delivery >>> services? >>> >>> I would rather see origin as an object and not just a simple URL. I’ve seen >>> this in the past in other systems. >>> >>> I don’t believe we can create multiple http services endpoint for a single >>> DS like we can for DNS type delivery service. This would solve it for us. >>> >>> Steve >>> >>> On Thu, Jan 24, 2019 at 11:22 AM Gray, Jonathan <[email protected] >>>> >>> wrote: >>> >>>> Even that has a strong element of confusion when in a self-service >>>> multi-tenancy world. Multiple tenants can use the same origins for >>> things >>>> like akami, aws, internal load balancers and any communication about the >>>> origin already being used will be confusing. >>>> >>>> Jonathan G >>>> >>>> >>>> On 1/23/19, 7:37 PM, "Gelinas, Derek" <[email protected]> >>> wrote: >>>> >>>> Perhaps we could solve both issues and just allow it to happen on DS >>>> config but throw up a message when it is saved that says "hey we're going >>>> to do this but it's really not without an element of risk and ats isn't >>>> designed to work properly with multiple identical origins. We recommend >>> you >>>> use an alternative fqdn blah blah blah." >>>> >>>> Something along those lines, anyway. >>>> >>>> Derek >>>> >>>>> On Jan 23, 2019, at 7:18 PM, Rawlin Peters < >>> [email protected]> >>>> wrote: >>>>> >>>>> Simply prohibiting duplicate origin FQDNs has been vetoed now, so >>> we >>>>> can't really move forward with that unless the vetoers change their >>>>> minds and remove their -1s. >>>>> >>>>> Currently that leaves us with checking at DS create/update whether >>> or >>>>> not it would conflict with a shared origin due to different DS >>> types, >>>>> qstring config, etc. I am -1 on that idea since I think it will be >>>>> difficult to maintain and enumerate all the possible conflicting >>>>> cases, will present confusing error messages to the user to which >>> the >>>>> solution would just be "create a CNAME record" anyways, and that >>> the >>>>> ATS parent.config should really be per-remap rather than "global". >>> I >>>>> know there has been some discussion in the ATS community about >>> making >>>>> parent.config per-remap, but I don't have much information there. >>>>> >>>>> That is to say, I don't think there is a great solution to this >>>>> problem right now that we can all agree upon, but maybe our efforts >>>>> would be better spent in the ATS community by making a per-remap >>>>> parent.config. Then the duplicate origin problem in Traffic Control >>>>> would go away. >>>>> >>>>> Another option could be to have some kind of generic rules engine >>>> that >>>>> would allow an organization to create some set of rules that DSes >>>>> would have to abide by. For example, no duplicate origins, an >>> origin >>>>> can't be in a list of known malicious domains, you can't use the DS >>>>> type "HTTP_NO_CACHE", maxDnsAnswers must be < 5, TTLs must be > 30 >>>> and >>>>> < 6000, protocol must be HTTPS, intitialDispersion must be < 10, >>> DSCP >>>>> must be 40, etc. It seems like for something truly self-service >>> you'd >>>>> probably need to set up some rules for your users to abide by that >>>>> might not necessarily apply to ALL organizations. Just a thought. I >>>>> don't think the duplicate origins problem alone would warrant >>>>> something like that, but if it also provided a solution for the >>>>> self-service effort, maybe it would be worth it. >>>>> >>>>> - Rawlin >>>>> >>>>> On Wed, Jan 23, 2019 at 4:15 AM Gelinas, Derek >>>>> <[email protected]> wrote: >>>>>> >>>>>> That's the problem. We either need to be smart enough to recognize >>>> that in our config and warn the user or just prevent it and suggest to >>> the >>>> user that they use a different fqdn. >>>>>> >>>>>> My vote is for preventing it at this point. >>>>>> >>>>>>> On Jan 23, 2019, at 12:50 AM, Gray, Jonathan < >>>> [email protected]> wrote: >>>>>>> >>>>>>> It's not just differing types, it's anything that affects the >>>> parent.config which also includes the, qstring configuration, MSO >>>> parents/config, etc. as well. >>>>>>> >>>>>>> Jonathan G >>>>>>> >>>>>>> >>>>>>> On 1/22/19, 2:33 PM, "Dave Neuman" <[email protected]> wrote: >>>>>>> >>>>>>> -1 on the one DS to Origin limitation. I think there are legit >>>> use cases >>>>>>> for it. >>>>>>> I would be +1 on the idea if we include type though. Two DSs >>>> can share the >>>>>>> same origin as long as they are of the same type (DNS, HTTP, >>>> HTTP_LIVE), >>>>>>> etc. You shouldn't be able to have two DSs with the same >>> origin >>>> but >>>>>>> different types. >>>>>>> >>>>>>> On Mon, Jan 14, 2019 at 7:53 AM Fieck, Brennan < >>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I don't see this as a complicated use case or a barrier to >>> entry, >>>> an >>>>>>>> extremely basic introduction to ATC would only make use of one >>> DS >>>>>>>> (shameless CDN-in-a-Box plug). >>>>>>>> It seems far more complicated for a user to be using many DSes >>> to >>>> serve a >>>>>>>> single origin - we can easily make it clear in the docs that the >>>> two have a >>>>>>>> 1:1 relationship, with a footnote about CNAME records for users >>>> with the >>>>>>>> knowledge, resources, and need for such workarounds. >>>>>>>> ________________________________________ >>>>>>>> From: Mark Torluemke <[email protected]> >>>>>>>> Sent: Friday, January 11, 2019 1:21 PM >>>>>>>> To: [email protected] >>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery >>>> Services >>>>>>>> produces indeterminate parent.config >>>>>>>> >>>>>>>>>> 1. Prohibit creating new delivery services that would share an >>>>>>>> existing origin and prohibit updating a delivery service to a >>>> shared >>>>>>>> origin >>>>>>>> In case my position has been lost, I'm still -1 on this. :) IMO, >>>> we >>>>>>>> shouldn't let the few complicated use cases increase the >>> barrier >>>> to entry >>>>>>>> (to using ATC) for the masses. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 11, 2019 at 11:53 AM Rawlin Peters < >>>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Alright, I'm trying to sum up this discussion so far since it >>>> seems >>>>>>>>> like everyone went on vacation and didn't really get a chance >>> to >>>> wrap >>>>>>>>> this one up: >>>>>>>>> - duplicate origins cause undefined behavior >>>>>>>>> - we need a way to migrate to a future that is free of >>> duplicate >>>>>>>>> origins in Traffic Control >>>>>>>>> - we need a visual and easy way to determine if Traffic Ops >>>> currently >>>>>>>>> contains duplicate origins, so that operators are incentivized >>>> to fix >>>>>>>>> them rather than let it slide indefinitely >>>>>>>>> - operators should have a fair amount of time to fix their >>>> duplicate >>>>>>>>> origins >>>>>>>>> >>>>>>>>> I believe this is what we've mostly agreed upon but haven't >>>> clearly voted >>>>>>>>> on: >>>>>>>>> >>>>>>>>> In release N we will: >>>>>>>>> 1. Prohibit creating new delivery services that would share an >>>>>>>>> existing origin and prohibit updating a delivery service to a >>>> shared >>>>>>>>> origin >>>>>>>>> 2. Add some kind of visual indicator that duplicate origins >>> are a >>>>>>>>> problem that need to be fixed before release N+1; otherwise, an >>>>>>>>> upgrade to N+1 will be prohibited. >>>>>>>>> >>>>>>>>> In release N+1 we will: >>>>>>>>> 3. Include a DB migration that adds a uniqueness constraint on >>>> origin >>>>>>>>> FQDN, removing the API-level checks for that. >>>>>>>>> 4. Prevent an upgrade to N+1 if duplicate origins are found >>> (this >>>>>>>>> might occur as a byproduct of step 3). >>>>>>>>> >>>>>>>>> I am +1 on this plan and believe this would hit on all the >>>> summarized >>>>>>>>> points above. Please provide a clear vote on this plan so that >>>> we can >>>>>>>>> dive deeper in the details (i.e. what release 'N' is, the best >>>> visual >>>>>>>>> indicator for step 2, and a friendly way to handle step 4). >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Rawlin >>>>>>>>> >>>>>>>>> On Wed, Dec 19, 2018 at 3:39 PM Jeremy Mitchell < >>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Not sure TP is the right place for a warning like "clean up >>> this >>>>>>>>>> 'duplicate' origin or your next upgrade will fail". Most users >>>> of our >>>>>>>>>> system will be like "not my problem". >>>>>>>>>> >>>>>>>>>> On Wed, Dec 19, 2018 at 11:58 AM Fieck, Brennan < >>>>>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Probably. It would impact load times by a bit, but the page >>>> for an >>>>>>>>>>> individual object is not our bottleneck. >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: Robert Butts <[email protected]> >>>>>>>>>>> Sent: Wednesday, December 19, 2018 11:50 AM >>>>>>>>>>> To: [email protected] >>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>> Delivery >>>>>>>>> Services >>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>> >>>>>>>>>>>> - Including a warning on startup and an API constraint >>>> preventing >>>>>>>>> adding >>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate >>>>>>>>>>>> - Adding a database constraint immediately into master that >>>> won't >>>>>>>> be >>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0 >>>>>>>>>>> >>>>>>>>>>> +1 >>>>>>>>>>> >>>>>>>>>>> I understand Jonathan's objection, but at some point, we have >>>> to be >>>>>>>>> able to >>>>>>>>>>> move forward. This is a good compromise: deprecate, then >>>> remove. That >>>>>>>>> gives >>>>>>>>>>> people a full major version to fix their data. >>>>>>>>>>> >>>>>>>>>>> I would be ideal if it were more than just a logged warning, >>>> though. >>>>>>>>> Can we >>>>>>>>>>> add a big red banner in Traffic Portal, on the Delivery >>>> Service page >>>>>>>>> for >>>>>>>>>>> any DS with a duplicate origin, telling users to fix it, and >>>> that >>>>>>>> they >>>>>>>>>>> won't be able to upgrade to the next major version until it's >>>> fixed? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Dec 19, 2018 at 10:57 AM Fieck, Brennan < >>>>>>>>> [email protected] >>>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> So it seems like nobody has a problem with the "how" - >>>> disallowing >>>>>>>>>>>> duplicate origin FQDNs on Delivery Services - but we never >>>> reached >>>>>>>> a >>>>>>>>>>>> consensus on "when". >>>>>>>>>>>> >>>>>>>>>>>> I stand by my previous proposal: >>>>>>>>>>>> - Including a warning on startup and an API constraint >>>> preventing >>>>>>>>> adding >>>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate >>>>>>>>>>>> - Adding a database constraint immediately into master that >>>> won't >>>>>>>> be >>>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0 >>>>>>>>>>>> ________________________________________ >>>>>>>>>>>> From: Rawlin Peters <[email protected]> >>>>>>>>>>>> Sent: Tuesday, December 18, 2018 4:59 PM >>>>>>>>>>>> To: [email protected] >>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>> Delivery >>>>>>>>> Services >>>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>>> >>>>>>>>>>>> Also, building more around DS types will make it even harder >>>> to get >>>>>>>>>>>> away from DS types in the future too, which I know is >>>> something >>>>>>>> we've >>>>>>>>>>>> discussed on this mailing list before. It also adds to the >>>> overhead >>>>>>>>> of >>>>>>>>>>>> Delivery Service Topologies, since a lot of the DS types >>> won't >>>>>>>>>>>> carryover into that world. >>>>>>>>>>>> >>>>>>>>>>>> - Rawlin >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 18, 2018 at 2:42 PM Fieck, Brennan >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> +1. >>>>>>>>>>>>> If there's a simple way to work around duplicate origins >>>> being >>>>>>>>>>>> prohibited, >>>>>>>>>>>>> then we should rely on that instead of "enumerating all >>> those >>>>>>>>> possible >>>>>>>>>>>> conflicting >>>>>>>>>>>>> settings, which are not only highly complex and confusing, >>>> but >>>>>>>> also >>>>>>>>>>>> further >>>>>>>>>>>>> entrench us in only supporting ATS as a caching proxy >>>> (hurting >>>>>>>>> efforts >>>>>>>>>>> to >>>>>>>>>>>>> integrate e.g. Grove, nginx etc.) >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> From: Rawlin Peters <[email protected]> >>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 2:20 PM >>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>> Delivery >>>>>>>>>>>> Services produces indeterminate parent.config >>>>>>>>>>>>> >>>>>>>>>>>>> There are a number of different DS settings at play that >>> can >>>>>>>>>>>>> potentially cause conflicts. The question is: do we want to >>>> get >>>>>>>>> into >>>>>>>>>>>>> the business of enumerating all those possible conflicting >>>>>>>>> settings or >>>>>>>>>>>>> just simply prohibit duplicate origins altogether? I think >>>> we can >>>>>>>>> dig >>>>>>>>>>>>> in and get that "sufficiently advanced sql query" to check >>>> for >>>>>>>>>>>>> conflicting origins, but is that something we want to carry >>>> along >>>>>>>>> for >>>>>>>>>>>>> the foreseeable future? Aren't CNAMEs relatively cheaper >>> than >>>>>>>>>>>>> developing and maintaining that code and the mental >>> overhead >>>>>>>>> required >>>>>>>>>>>>> in understanding why you're getting an error that says your >>>>>>>>> requested >>>>>>>>>>>>> DS would cause an origin conflict? I think at the point >>>> you've >>>>>>>>>>>>> requested a DS that would create a conflict, you've chosen >>>> those >>>>>>>>>>>>> settings for a reason and would probably prefer to just >>>>>>>> create/use >>>>>>>>> a >>>>>>>>>>>>> CNAME in your new DS and keep the rest of your settings the >>>> same. >>>>>>>>>>>>> >>>>>>>>>>>>> Thinking in terms of errors, I'm imagining: >>>>>>>>>>>>> "cannot create delivery service: origin fqdn ' >>>> foo.example.com' >>>>>>>>> already >>>>>>>>>>>> in use" >>>>>>>>>>>>> vs >>>>>>>>>>>>> "cannot create delivery service: origin fqdn ' >>>> foo.example.com' >>>>>>>>> already >>>>>>>>>>>>> in use as type DNS_LIVE_NATNL, which is incompatible with >>>> your >>>>>>>>> chosen >>>>>>>>>>>>> type of HTTP_NO_CACHE" >>>>>>>>>>>>> >>>>>>>>>>>>> At that point you'd probably say to yourself, "uh, I need >>>>>>>>>>>>> HTTP_NO_CACHE, so what am I supposed to do now?" >>>>>>>>>>>>> >>>>>>>>>>>>> As a lazy developer I'm +1 on prohibiting duplicate origin >>>> fqdns >>>>>>>>>>>>> because the resulting code will be simpler, but I think >>>>>>>> eliminating >>>>>>>>>>>>> the mental overhead for operators could be worthwhile too. >>>> If we >>>>>>>>> can >>>>>>>>>>>>> agree on an end state of prohibiting duplicate origins >>>>>>>> altogether, >>>>>>>>> we >>>>>>>>>>>>> can start working on a design to smoothly transition us to >>>> that >>>>>>>>> point. >>>>>>>>>>>>> Are we willing to live with "just CNAME your origin fqdn" >>> as >>>> the >>>>>>>>>>>>> standard solution to duplicates? >>>>>>>>>>>>> >>>>>>>>>>>>> - Rawlin >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 18, 2018 at 1:27 PM Gelinas, Derek >>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> The only situation in which they can share origins is if >>> a) >>>> the >>>>>>>>>>>> origins are shared in an MSO configuration but still have >>>> different >>>>>>>>>>> defined >>>>>>>>>>>> origin fields in the delivery service, or if they're >>> assigned >>>> to >>>>>>>>>>> completely >>>>>>>>>>>> different cachegroups. It's when two delivery services >>> share >>>> the >>>>>>>>> same >>>>>>>>>>>> edges that there's an issue, because you end up with >>>> parent.config >>>>>>>>>>> issues. >>>>>>>>>>>> Actually you could even get away with it in mids as long as >>>> you >>>>>>>>> weren't >>>>>>>>>>>> doing anything like MSO to it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Could get messy real fast, though. Best to just create a >>>>>>>> second >>>>>>>>>>> FQDN. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Derek >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 12/18/18, 3:23 PM, "Fieck, Brennan" < >>>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> So no two Delivery Services may share an origin >>>> *regardless >>>>>>>>> of >>>>>>>>>>>> cache hierarchy* ? I've been told that DNS Delivery Services >>>> can >>>>>>>>> have the >>>>>>>>>>>> same origin as HTTP Delivery Services because they obey the >>>> same >>>>>>>>> cache >>>>>>>>>>>> hierarchy. You're saying that would still produce invalid >>>> output >>>>>>>>> and/or >>>>>>>>>>> is >>>>>>>>>>>> explicitly disallowed by ATS? >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> From: Robert Butts <[email protected]> >>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 1:09 PM >>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>>>>>>> Delivery >>>>>>>>>>>> Services produces indeterminate parent.config >>>>>>>>>>>>>> >>>>>>>>>>>>>>> can you give an example of what parent.config looks like >>>>>>>>> when 2 >>>>>>>>>>>> ds's share >>>>>>>>>>>>>> an origin and have different a different topology? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Answering because I encountered this directly, when >>>>>>>> rewriting >>>>>>>>>>>> parent.config. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For example: Suppose you have one Delivery Service: >>>>>>>>>>>>>> XML_ID: foo >>>>>>>>>>>>>> Type: HTPT_LIVE_NATL >>>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass >>>> up >>>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net >>>>>>>>>>>>>> >>>>>>>>>>>>>> And another Delivery Service: >>>>>>>>>>>>>> XML_ID: bar >>>>>>>>>>>>>> Type: HTPT_LIVE_NATL >>>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass >>>> up >>>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net >>>>>>>>>>>>>> >>>>>>>>>>>>>> ATS only supports unique `dest_domain` entries in >>>>>>>>> parent.config. >>>>>>>>>>>> Therefore, >>>>>>>>>>>>>> the parent.config generated for a server assigned to >>> both >>>>>>>> of >>>>>>>>>>> these >>>>>>>>>>>> Delivery >>>>>>>>>>>>>> Services with either be: >>>>>>>>>>>>>> >>>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=true >>>>>>>>>>>>>> >>>>>>>>>>>>>> Or >>>>>>>>>>>>>> >>>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=false >>>>>>>>>>>> qstring=consider >>>>>>>>>>>>>> >>>>>>>>>>>>>> Right now, it's arbitrary which Perl Traffic Ops >>> inserts, >>>>>>>> and >>>>>>>>>>> Perl >>>>>>>>>>>> provides >>>>>>>>>>>>>> no warning or error of any kind (the pending Go >>>>>>>>> parent.config PR >>>>>>>>>>>> logs an >>>>>>>>>>>>>> error). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Whichever is arbitrarily inserted, the resulting remaps >>>> for >>>>>>>>> the >>>>>>>>>>>> other >>>>>>>>>>>>>> delivery service will be wrong. Either `foo` requests >>> will >>>>>>>>> drop >>>>>>>>>>>> the query >>>>>>>>>>>>>> string when they shouldn't, and go to the mid when they >>>>>>>>>>> shouldn't; >>>>>>>>>>>> or `bar` >>>>>>>>>>>>>> requests will use the query string and skip the mid when >>>> it >>>>>>>>>>>> shouldn't. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Does that make sense? The only correct solution, is to >>>>>>>>> somehow >>>>>>>>>>>> prevent >>>>>>>>>>>>>> different DSes having the same origin, and tell tenants >>>>>>>> they >>>>>>>>> must >>>>>>>>>>>> use >>>>>>>>>>>>>> CNAMEs if they need. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This isn't a bug in Traffic Control. ATS fundamentally >>>>>>>>> doesn't >>>>>>>>>>>> support >>>>>>>>>>>>>> multiple remap rules with the same parent FQDN with >>>>>>>> different >>>>>>>>>>>>>> configurations. Hence, Traffic Control needs to prohibit >>>>>>>>> that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 12:24 PM Jeremy Mitchell < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> brennan, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> can you give an example of what parent.config looks like >>>>>>>>> when 2 >>>>>>>>>>>> ds's share >>>>>>>>>>>>>>> an origin and have different a different topology? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> jeremy >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 11:39 AM Fieck, Brennan < >>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To be clear, the "Warning" I'm talking about would >>>>>>>>> happen at >>>>>>>>>>>> startup, but >>>>>>>>>>>>>>>> I'd like a UI-only constraint to come with that to >>>>>>>>> disallow >>>>>>>>>>>> using the API >>>>>>>>>>>>>>>> to bind the same origin to multiple Delivery Services >>>>>>>>> with >>>>>>>>>>>> varying >>>>>>>>>>>>>>>> topography requirements. It wouldn't change the >>>>>>>> existing >>>>>>>>>>> data, >>>>>>>>>>>> but >>>>>>>>>>>>>>> prevent >>>>>>>>>>>>>>>> users from creating more bad data. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> "warning" doesn't really sufficiently describe that, my >>>>>>>>> bad. >>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>> From: Fieck, Brennan <[email protected]> >>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 11:24 AM >>>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>>>>>>>>>> Delivery Services >>>>>>>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Well the cost of fixing this bug is a constraint on the >>>>>>>>> data. >>>>>>>>>>>> Unless we >>>>>>>>>>>>>>>> make it a UI-only constraint - which I'm personally >>>>>>>>> against - >>>>>>>>>>>> there must >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>> some point in the future where ATC cannot reasonably be >>>>>>>>>>>> expected to work >>>>>>>>>>>>>>>> with data that violates that constraint. The question >>>>>>>> is >>>>>>>>> when >>>>>>>>>>>> that should >>>>>>>>>>>>>>>> occur, which should likely happen at a minor version >>>>>>>>> release. >>>>>>>>>>>> Minor not >>>>>>>>>>>>>>>> major because it doesn't involve a change in data >>>>>>>>> structures, >>>>>>>>>>>> merely >>>>>>>>>>>>>>>> relationships between them - in my opinion that's a >>>>>>>> minor >>>>>>>>>>>> version change >>>>>>>>>>>>>>>> but that's definitely up for debate. With several >>>>>>>> release >>>>>>>>>>>> candidates for >>>>>>>>>>>>>>>> 3.0.0 that _doesn't_ include this restriction already >>>>>>>> in >>>>>>>>> the >>>>>>>>>>>> wild, I >>>>>>>>>>>>>>>> wouldn't recommend putting it in there. That means to >>>>>>>>> fix the >>>>>>>>>>>> bug as soon >>>>>>>>>>>>>>>> as possible it should go in 3.1.0 which should be the >>>>>>>>> target >>>>>>>>>>>> of "master" >>>>>>>>>>>>>>>> after the 3.0.0 release is cut from it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So I'd recommend immediately implementing the >>>>>>>> constraint >>>>>>>>> in >>>>>>>>>>>> master with a >>>>>>>>>>>>>>>> refusal to upgrade with bad data, and backport a >>>>>>>> warning >>>>>>>>>>> about >>>>>>>>>>>> the future >>>>>>>>>>>>>>>> behavior into 3.0.0 or as part of a 3.0.1 provided we >>>>>>>> had >>>>>>>>>>> more >>>>>>>>>>>> changes >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>> would warrant a micro version bump. >>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>> From: Gray, Jonathan <[email protected]> >>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 9:34 AM >>>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>>>>>>>>>> Delivery Services >>>>>>>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -1 Holding an ATC upgrade hostage to data cleanup seems >>>>>>>>> like >>>>>>>>>>> a >>>>>>>>>>>> bad idea. >>>>>>>>>>>>>>>> The issue isn't great, but it's also not new. We >>>>>>>> should >>>>>>>>>>> allow >>>>>>>>>>>> teams to >>>>>>>>>>>>>>> fix >>>>>>>>>>>>>>>> their data at their normal paces if it doesn't create >>>>>>>>>>>> significant >>>>>>>>>>>>>>> overhead >>>>>>>>>>>>>>>> or an inherant blocker for new functionality or >>>>>>>>> correction of >>>>>>>>>>>> other major >>>>>>>>>>>>>>>> problems imho. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jonathan G >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 12/18/18, 9:28 AM, "Fieck, Brennan" < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Another option is we could detect collisions at >>>>>>>>> startup >>>>>>>>>>>> and simply >>>>>>>>>>>>>>>> refuse to continue with the upgrade until the data is >>>>>>>>> fixed. >>>>>>>>>>>> That would >>>>>>>>>>>>>>>> allow people using the now-unsupported data format to >>>>>>>>>>> continue >>>>>>>>>>>> to use >>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>> old versions of Traffic Ops without wrecking their >>>>>>>>> database, >>>>>>>>>>>> but also >>>>>>>>>>>>>>>> provide an incentive to clean up the data. >>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>> From: Gray, Jonathan <[email protected]> >>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 5:12 AM >>>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to >>>>>>>>> Multipe >>>>>>>>>>>> Delivery >>>>>>>>>>>>>>>> Services produces indeterminate parent.config >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'm generally a fan of constrain your data in your >>>>>>>>>>>> database, but not >>>>>>>>>>>>>>>> necessarily exclusively. I see this as a one-way >>>>>>>>>>>> cleanup/conversion so >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> doesn't need to be configurable; otherwise you have to >>>>>>>>> ask >>>>>>>>>>> the >>>>>>>>>>>> question >>>>>>>>>>>>>>>> what happens if someone turns it off. That said, >>>>>>>>> something >>>>>>>>>>> in >>>>>>>>>>>> the UI >>>>>>>>>>>>>>> layer >>>>>>>>>>>>>>>> would be nice to prevent spending significant >>>>>>>> quantities >>>>>>>>> of >>>>>>>>>>>> time >>>>>>>>>>>>>>> building a >>>>>>>>>>>>>>>> complex DS only to have it fail to post for reasons >>>>>>>> that >>>>>>>>>>> could >>>>>>>>>>>> have been >>>>>>>>>>>>>>>> known earlier. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The way my brain works in this case: >>>>>>>>>>>>>>>> If !unique_constraint_exists_query() >>>>>>>>>>>>>>>> If has_duplicates_query() >>>>>>>>>>>>>>>> show_warning() >>>>>>>>>>>>>>>> else >>>>>>>>>>>>>>>> add_unique_constraint() >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> to which the API and UI configuration could also >>>>>>>>> make use >>>>>>>>>>>> of >>>>>>>>>>>>>>>> unique_constraint_exists_query() to drive additional >>>>>>>>> layer >>>>>>>>>>>> constraints if >>>>>>>>>>>>>>>> desired. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jonathan G >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 12/17/18, 1:11 PM, "Rawlin Peters" < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That is an interesting idea...detect at TO >>>>>>>>> startup >>>>>>>>>>>> whether or not >>>>>>>>>>>>>>>> there are duplicate origins and operate in a >>>>>>>>> "prevent >>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> origins" state if no duplicates are found or >>>>>>>>> "prevent >>>>>>>>>>>> conflicting >>>>>>>>>>>>>>>> DS >>>>>>>>>>>>>>>> topologies" state if duplicates are found? So >>>>>>>>> once >>>>>>>>>>>> operators have >>>>>>>>>>>>>>>> replaced all the duplicate origins with CNAMEs, >>>>>>>>> TO >>>>>>>>>>> will >>>>>>>>>>>>>>> essentially >>>>>>>>>>>>>>>> operate in a "prohibit all duplicate origins" >>>>>>>>> state. >>>>>>>>>>>> That would >>>>>>>>>>>>>>>> probably make for a simpler transition, but I'd >>>>>>>>> want >>>>>>>>>>>> to remove >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>> logic in a following release that strictly >>>>>>>>> prohibits >>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> origins >>>>>>>>>>>>>>>> (assuming that the community agrees we should >>>>>>>>>>> prohibit >>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> origins altogether). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As for DB constraints vs UI, I was thinking >>>>>>>> those >>>>>>>>>>>> DS-type >>>>>>>>>>>>>>>> constraints >>>>>>>>>>>>>>>> I pointed out would live in the API. It would >>>>>>>>>>>> basically be added >>>>>>>>>>>>>>>> validation in the deliveryservices POST/PUT >>>>>>>>> endpoint >>>>>>>>>>>> that checks >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> DB for existing DSes that conflict with the >>>>>>>>> requested >>>>>>>>>>>> DS. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Rawlin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 12:35 PM Gray, Jonathan >>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> These kinds of conditions should be >>>>>>>> detectable >>>>>>>>>>> with a >>>>>>>>>>>>>>>> sufficiently advanced SQL query. Is it possible to add >>>>>>>>> the >>>>>>>>>>>> constraint if >>>>>>>>>>>>>>>> it passes and emit a warning during TO startup >>>>>>>> otherwise? >>>>>>>>>>>> That would let >>>>>>>>>>>>>>>> you know the condition exists at startup but not >>>>>>>> getting >>>>>>>>> in >>>>>>>>>>>> your way and >>>>>>>>>>>>>>>> keep you out of trouble once you've cleaned up. We >>>>>>>> made >>>>>>>>> a >>>>>>>>>>>> mistake early >>>>>>>>>>>>>>>> on, but this would acknowledge it was bad and encourage >>>>>>>>> it to >>>>>>>>>>>> be fixed at >>>>>>>>>>>>>>>> the speed of operations teams. Also this puts the >>>>>>>>> constraint >>>>>>>>>>>> in the >>>>>>>>>>>>>>>> database rather than the UI which is really where the >>>>>>>>>>>> contention is for >>>>>>>>>>>>>>>> usability. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jonathan G >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 12/17/18, 11:38 AM, "Rawlin Peters" < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We occasionally discuss this issue but >>>>>>>>> haven't >>>>>>>>>>>> tackled it >>>>>>>>>>>>>>>> yet. I think >>>>>>>>>>>>>>>>> the main issue is just that duplicate >>>>>>>>> origins >>>>>>>>>>>> have been >>>>>>>>>>>>>>>> allowed since >>>>>>>>>>>>>>>>> the beginning, and now everyone's Traffic >>>>>>>>> Ops >>>>>>>>>>>> could be >>>>>>>>>>>>>>>> littered with >>>>>>>>>>>>>>>>> duplicate origins. Also, depending on the >>>>>>>>>>> config >>>>>>>>>>>> of the >>>>>>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>>> delivery services, the origins might not >>>>>>>>> be in >>>>>>>>>>>> conflict at >>>>>>>>>>>>>>>> all (if >>>>>>>>>>>>>>>>> they don't have different topology >>>>>>>>>>> constraints). >>>>>>>>>>>> I would >>>>>>>>>>>>>>>> love for us >>>>>>>>>>>>>>>>> to just add a uniqueness constraint, but >>>>>>>>> there >>>>>>>>>>>> would need >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> be a fair >>>>>>>>>>>>>>>>> amount of warning to the community before >>>>>>>>> doing >>>>>>>>>>>> so and >>>>>>>>>>>>>>> might >>>>>>>>>>>>>>>>> invalidate a significant amount of valid >>>>>>>>> use >>>>>>>>>>>> cases. >>>>>>>>>>>>>>>> Operators would >>>>>>>>>>>>>>>>> need time to make DNS CNAME records for >>>>>>>> the >>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> origins and >>>>>>>>>>>>>>>>> update their DSes to use the different >>>>>>>>> CNAMEs. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think as a good first step to >>>>>>>>> eliminating the >>>>>>>>>>>> use of >>>>>>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>>> origins altogether, we should identify >>>>>>>>> which >>>>>>>>>>>> "topology >>>>>>>>>>>>>>>> constraints" >>>>>>>>>>>>>>>>> actually cause conflicting config when >>>>>>>> used >>>>>>>>>>> with >>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> origins and >>>>>>>>>>>>>>>>> prevent creating DSes with duplicate >>>>>>>>> origins >>>>>>>>>>> _if >>>>>>>>>>>> it would >>>>>>>>>>>>>>>> cause a >>>>>>>>>>>>>>>>> conflict with an existing DS that uses >>>>>>>> the >>>>>>>>> same >>>>>>>>>>>> origin_. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For instance, I believe an HTTP and >>>>>>>>> DNS-type DS >>>>>>>>>>>> can live >>>>>>>>>>>>>>>> happily >>>>>>>>>>>>>>>>> side-by-side using the same origin >>>>>>>>> (probably >>>>>>>>>>>> need different >>>>>>>>>>>>>>>>> routing_names?), but scenarios like HTTP >>>>>>>>> and >>>>>>>>>>>> HTTP_LIVE, or >>>>>>>>>>>>>>>> DNS and >>>>>>>>>>>>>>>>> HTTP_NO_CACHE sharing the same origin >>>>>>>> will >>>>>>>>>>> cause >>>>>>>>>>>> conflicts >>>>>>>>>>>>>>>> for sure. >>>>>>>>>>>>>>>>> So maybe we can start by making sure the >>>>>>>> DS >>>>>>>>>>>> types "match" >>>>>>>>>>>>>>>> when using >>>>>>>>>>>>>>>>> the same origin: >>>>>>>>>>>>>>>>> HTTP + DNS: possibly good, if they have >>>>>>>>>>>> different routing >>>>>>>>>>>>>>>> names? >>>>>>>>>>>>>>>>> HTTP_LIVE + HTTP_LIVE_NATNL: bad >>>>>>>>>>>>>>>>> HTTP_NO_CACHE + [any other type]: bad >>>>>>>>>>>>>>>>> HTTP_LIVE + HTTP: bad >>>>>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> There are most likely other conflict >>>>>>>>> scenarios >>>>>>>>>>>> that don't >>>>>>>>>>>>>>>> involve the >>>>>>>>>>>>>>>>> DS types, but I think this would be a >>>>>>>> good >>>>>>>>>>>> start. In the >>>>>>>>>>>>>>>> future with >>>>>>>>>>>>>>>>> Delivery Service Topologies (aka Flexible >>>>>>>>>>>> Cachegroups aka >>>>>>>>>>>>>>>> Bring Your >>>>>>>>>>>>>>>>> Own Topology), we might be able to >>>>>>>> prohibit >>>>>>>>>>>> assigning a DS >>>>>>>>>>>>>>>> to a >>>>>>>>>>>>>>>>> Topology if the DS's origin is already >>>>>>>>> used by >>>>>>>>>>>> another DS >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>> different Topology. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - Rawlin >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 10:52 AM Fieck, >>>>>>>>> Brennan >>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> As some of you may be aware, >>>>>>>>> `parent.config` >>>>>>>>>>>> files >>>>>>>>>>>>>>>> generated by Traffic Ops can vary wildly when an origin >>>>>>>>> is >>>>>>>>>>>> assigned to >>>>>>>>>>>>>>>> multiple Delivery Services. This results in undefined >>>>>>>>>>>> behavior. I'm told >>>>>>>>>>>>>>>> that the conflict only happens when two Delivery >>>>>>>> Services >>>>>>>>>>> with >>>>>>>>>>>> different >>>>>>>>>>>>>>>> "topology requirements" use the same origin, whatever >>>>>>>>> that >>>>>>>>>>>> means (content >>>>>>>>>>>>>>>> routing type?). Regardless, the issue should be >>>>>>>>> addressed. >>>>>>>>>>> The >>>>>>>>>>>> obvious >>>>>>>>>>>>>>>> solution is to put in place a database constraint that >>>>>>>>>>>> prevents an origin >>>>>>>>>>>>>>>> from being assigned to more that one Delivery Service >>>>>>>>> with >>>>>>>>>>> API >>>>>>>>>>>> checks in >>>>>>>>>>>>>>>> place that would provide helpful error messages when an >>>>>>>>>>>> attempt is made >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> violate the constraint. However, would that mess with >>>>>>>>> things >>>>>>>>>>>> like >>>>>>>>>>>>>>>> Multi-Site Origin? Or is it just not viable for some >>>>>>>>> other >>>>>>>>>>>> reason? If it >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> a good solution, I'm prepared to work on a fix that >>>>>>>>> utilizes >>>>>>>>>>>> it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>> >>>> >>>> >>> >
