[GitHub] jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour
jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour URL: https://github.com/apache/incubator-druid/issues/5979#issuecomment-404289501 @erankor thanks! I raised https://github.com/apache/incubator-druid/issues/5994. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
[GitHub] jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour
jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour URL: https://github.com/apache/incubator-druid/issues/5979#issuecomment-404276658 @erankor 1. That SQL is to list all _used_ (live) segments falling in an interval, and it can be called 1) whenever a new segment is allocated in KIS and 2) when a taskLock is acquired in batch ingestion (like index or merge tasks). a. Yes, you can delete them, but I would recommend to use the kill task rather than manually deleting rows. You can set `druid.coordinator.kill.on` to true for coordinators to periodically kill segments. Please check http://druid.io/docs/0.10.0/configuration/coordinator.html#coordinator-operation for details. b. I haven't tested, but sounds making sense. 2. The `druid_tasklogs` is not actually being used, so you can freely delete all entries in that table. Please check https://github.com/apache/incubator-druid/issues/5859. For `druid_tasks`, yes you can delete old entries from that table. One thing I'm curious is, do you have an actual issue of too large `druid_tasks` table? or are you just checking all tables? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
[GitHub] jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour
jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour URL: https://github.com/apache/incubator-druid/issues/5979#issuecomment-403952075 > I won't have any job in queue for that long, my main concern is whether it's ok to update this table directly while the overlord is running. @erankor yes, it should be fine if you can make sure to not have any _running_ tasks of created_date older than a week. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
[GitHub] jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour
jihoonson commented on issue #5979: Kafka Indexing Service lagging every hour URL: https://github.com/apache/incubator-druid/issues/5979#issuecomment-403947611 @erankor good to hear that you've solved! FYI, a new feature for automatic pendingSegments cleanup (https://github.com/apache/incubator-druid/pull/5149) has added in 0.12.0. This may make easier to handle the growing pendingSegments table. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
