To clarify, PubsubIO does not clean up auto-created subscriptions, and SQL doesn't compensate for that.
On Wed, Feb 6, 2019 at 8:45 AM Mikhail Gryzykhin <mig...@google.com> wrote: > Thank you for quick response Andrew. > > I'll cleanup these. I'll keep the bug open and assign it to @Kenneth > Knowles <k...@google.com> who's working on SQK for follow up: we need a > way to automatically cleanup topics. > > Current suggestions: > 1. Make SQL cleanup created topics > 2. Cleanup topics created by SQL in tests > 3. Increase quota so that topics have enough time to be cleaned up > automatically. > > Regards, > --Mikhail > > Have feedback <http://go/migryz-feedback>? > > > On Wed, Feb 6, 2019 at 8:05 AM Andrew Pilloud <apill...@google.com> wrote: > >> SQL doesn't cleanup pubsub subscriptions. Feel free to delete those. >> >> Andrew >> >> On Wed, Feb 6, 2019, 8:01 AM Mikhail Gryzykhin <mig...@google.com> wrote: >> >>> +Kenneth Knowles <k...@google.com> you're working on SQL recently, so >>> might provide some info. >>> >>> I see a lot of topics of format >>> rojects/apache-beam-testing/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721 >>> <https://pantheon.corp.google.com/cloudpubsub/topics/integ-test-PubsubJsonIT-testSQLLimit-2018-08-27-16-55-44-342-start-7392789257486934721?project=apache-beam-testing>. >>> Seems we do not cleanup properly. >>> >>> Is it safe to cleanup topics with this name? >>> >>> --Mikhail >>> >>> Have feedback <http://go/migryz-feedback>? >>> >>> >>> On Wed, Feb 6, 2019 at 7:46 AM Mikhail Gryzykhin <mig...@google.com> >>> wrote: >>> >>>> Minor UPD: >>>> As expected it fails most of our test jobs, since we use Pub/Subs in >>>> many tests. >>>> >>>> --Mikhail >>>> >>>> Have feedback <http://go/migryz-feedback>? >>>> >>>> >>>> On Wed, Feb 6, 2019 at 7:38 AM Mikhail Gryzykhin <mig...@google.com> >>>> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> Our python pipelines failed with limit exceeded error >>>>> <https://builds.apache.org/job/beam_PostCommit_Python_Verify/7313/console> >>>>> : >>>>> >>>>> ResourceExhausted: 429 Your project has exceeded a limit: >>>>> (type="topics-per-project", current=10000, maximum=10000). >>>>> >>>>> >>>>> Does anyone know if there were new tests that use topics added >>>>> recently? >>>>> >>>>> I tried to see list of topics, but UI fails >>>>> <https://pantheon.corp.google.com/cloudpubsub/topicList?project=apache-beam-testing> >>>>> to load. Will see if I can use APIs to investigate. >>>>> >>>>> If anyone has good insight, please, pick up BEAM-6610 >>>>> <https://issues.apache.org/jira/browse/BEAM-6610>. >>>>> >>>>> --Mikhail >>>>> >>>>> Have feedback <http://go/migryz-feedback>? >>>>> >>>>
