Re: TeamCity having problems

2019-03-01 Thread Clint Wylie
After adding a maven clean step after the newly introduced maven install step to clean up the 'target' directories leftover from compiling, it looks like team city has passed for the first time in ~3 days! On Wed, Feb 27, 2019 at 6:57 PM Gian Merlino wrote: > It seems to be broken in a different

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread David Glasser
Thanks, I've opened a proposal and taken Gian's suggestion into account. https://github.com/apache/incubator-druid/issues/7180 On Fri, Mar 1, 2019 at 4:20 PM Gian Merlino wrote: > To me this seems like a lot of effort to go through just to detect cases > where servers from two different clusters

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread Gian Merlino
To me this seems like a lot of effort to go through just to detect cases where servers from two different clusters are misconfigured to read each others' files or talk to each other by accident. I wonder if there's an easier way to do it. Maybe keep the cluster name idea, but write it to a marker f

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread Jihoon Son
Thanks for additional details. It sounds pretty straightforward and maybe it's better than poking database every time. It would be worth to start a discussion on Github by raising a proposal if you think it's valuable. Jihoon On Fri, Mar 1, 2019 at 2:17 PM David Glasser wrote: > Makes sense. >

Podling Report Reminder - March 2019

2019-03-01 Thread jmclean
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 20 March 2019, 10:30 am PDT. The report for your podling will form a

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread David Glasser
Makes sense. To elaborate a bit more on my "cluster name" concept, I actually think it would be pretty straightforward: - Add something like `druid.cluster.name=staging`. - To be compatible with existing data, also add something like `druid.cluster.allowSegmentsFromClusters=["", "dev"]`. Note tha

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread Jihoon Son
The broker learns from historicals and tasks even though recently a PR has been merged to keep published segments in memory ( https://github.com/apache/incubator-druid/pull/6901) in brokers. Probably it makes sense to filter out segments in brokers too if they are from historicals and not in the me

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread David Glasser
That makes sense. Does the coordinator's decisions about what segments are 'used' affect the broker's choices for routing queries, or does it just learn about things directly from historicals/ingestion tasks (via... zookeeper?) --dave On Fri, Mar 1, 2019 at 1:15 PM Jihoon Son wrote: > Hi Dave,

Re: Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread Jihoon Son
Hi Dave, I think the third option sounds most reasonable to fix this issue. Though the second option sounds useful in general. And yes, it wouldn't be easy to refuse to announce unknown segments in historicals. I think it makes more sense to check only in the coordinator because it's the only node

Re: Auto-closing old PRs

2019-03-01 Thread Dylan Wylie
Yeah, any comments/commits/activity on a PR reset the threshold. e.g. https://github.com/apache/incubator-druid/pull/6768 On Thu, 28 Feb 2019 at 23:05, Gian Merlino wrote: > I thought the bot uses a threshold of 60 days with absolutely no activity > (not 60 days since opening or anything like

Re: Downloading binaries of previous Druid versions

2019-03-01 Thread Gian Merlino
They are available at URLs like http://static.druid.io/artifacts/releases/druid-0.12.3-bin.tar.gz (not listed anywhere, but you should be able to guess them). They aren't available from the Apache mirrors, since they are not Apache releases. On Thu, Feb 28, 2019 at 9:33 PM Eyal Yurman wrote: > H

Namespacing segments, or preventing unknown segments from wreaking havoc

2019-03-01 Thread David Glasser
(I sent this message to druid-user last week and got no response. Since it is proposing making improvements to Druid, I thought maybe it would be appropriate to resend here. Hope that's OK.) We had a big outage in our Druid cluster last week. We run our Druid servers in Kubernetes, and our histor