OK got it!
Thanks! On Tue, 9 Mar 2021 at 21:17, Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > That property decides how many log files (log file is created per batch > per type - types are like offsets, commits, etc.) to retain on the > checkpoint. > > Unless you're struggling with a small files problem on checkpoint, you > wouldn't need to tune the value. I guess that's why the configuration is > marked as "internal" meaning just some admins need to know about such > configuration. > > On Wed, Mar 10, 2021 at 3:58 AM German Schiavon <gschiavonsp...@gmail.com> > wrote: > >> Hey Maxim, >> >> ok! I didn't see them. >> >> Is this property documented somewhere? >> >> Thanks! >> >> On Tue, 9 Mar 2021 at 13:57, Maxim Gekk <maxim.g...@databricks.com> >> wrote: >> >>> Hi German, >>> >>> It is used at least at: >>> 1. >>> https://github.com/apache/spark/blob/a093d6feefb0e086d19c86ae53bf92df12ccf2fa/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala#L56 >>> 2. >>> https://github.com/apache/spark/blob/e7e016192f882cfb430d706c2099e58e1bcc014c/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala#L84 >>> >>> Maxim Gekk >>> >>> Software Engineer >>> >>> Databricks, Inc. >>> >>> >>> On Tue, Mar 9, 2021 at 3:27 PM German Schiavon <gschiavonsp...@gmail.com> >>> wrote: >>> >>>> Hello all, >>>> >>>> I wanted to ask if this property is still active? I can't find it in >>>> the doc https://spark.apache.org/docs/latest/configuration.html or >>>> anywhere in the code(only in Tests). >>>> >>>> If so, should we remove it? >>>> >>>> val MIN_BATCHES_TO_RETAIN = >>>> buildConf("spark.sql.streaming.minBatchesToRetain") >>>> .internal() >>>> .doc("The minimum number of batches that must be retained and made >>>> recoverable.") >>>> .version("2.1.1") >>>> .intConf >>>> .createWithDefault(100) >>>> >>>>
