Re: [UPDATE] Beam 2.25.0 release progress update

[Kenneth Knowles]

Have we already discussed the possibility of adding all committers to this
account? (sorry if I missed a prior discussion)

Kenn

On Mon, Oct 5, 2020 at 3:43 PM Robin Qiu  wrote:

> One thing that could block my work soon is that I am still not added to
> the beammaintainers team in Docker Hub. Pablo has helped me created a infra
> ticket (https://issues.apache.org/jira/browse/INFRA-20900) but it has not
> made progress. Could someone help with expediting this? Thanks!
>
> On Mon, Oct 5, 2020 at 3:39 PM Robin Qiu  wrote:
>
>> Update: the release blockers are all resolved and the fixes are
>> cherry-picked into the release branch. I will work on creating a release
>> candidate soon.
>>
>> On Fri, Sep 25, 2020 at 11:36 AM Robin Qiu  wrote:
>>
>>> Update: here is the list of release blockers to be resolved
>>> https://issues.apache.org/jira/projects/BEAM/versions/12347147
>>>
>>> On Thu, Sep 24, 2020 at 5:00 PM Robin Qiu  wrote:
>>>
 I see. Thanks for letting me know!

 On Thu, Sep 24, 2020 at 4:59 PM Robert Bradshaw 
 wrote:

> FYI, I'm waiting for Daniel's CLs to go in before starting my own.
>
> On Thu, Sep 24, 2020 at 9:29 AM Robert Bradshaw 
> wrote:
>
>> OK, I'll get that started.
>>
>> On Thu, Sep 24, 2020 at 9:26 AM Robin Qiu  wrote:
>>
>>> No I haven't. Could you please do that? I wanted to sync with you
>>> yesterday but I saw you were ooo.
>>>
>>> On Thu, Sep 24, 2020 at 9:03 AM Robert Bradshaw 
>>> wrote:
>>>
 [just you]

 Have you started an import at this commit, or should I?

 On Wed, Sep 23, 2020 at 5:51 PM Robin Qiu 
 wrote:

> Update: The 2.25.0 release branch has been cut.
>
>
> On Wed, Sep 23, 2020 at 3:34 PM Robin Qiu 
> wrote:
>
>> Hello everyone,
>>
>> I am starting this email thread to share with you updates on Beam
>> 2.25.0 release.
>>
>> Update: I plan to cut the release branch in a few hours,
>> according to the release calendar [1].
>>
>> Thanks,
>> Robin
>>
>> [1]
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>>
>>
>>
>
>

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

[Kenneth Knowles]

On Mon, Oct 5, 2020 at 2:44 PM Luke Cwik  wrote:

> For the 2.25 release the Java Direct, Flink, Jet, Samza, Twister2 will use
> SDF powered Read transforms. Users can opt-out
> with --experiments=use_deprecated_read.
>

Huzzah! In our release notes maybe be clear about the expectations for
users:

 - semantics are expected to be the same: file bugs for any change in
results
 - perf may vary: file bugs or write to user@

I was unable to get Spark done for 2.25 as I found out that Spark streaming
> doesn't support watermark holds[1]. If someone knows more about the
> watermark system in Spark I could use some guidance here as I believe I
> have a version of unbounded SDF support written for Spark (I get all the
> expected output from tests, just that watermarks aren't being held back so
> PAssert fails).
>

Spark's watermarks are not comparable to Beam's. The rule as I understand
it is that any data that is later than `max(seen timestamps) -
allowedLateness` is dropped. One difference is that dropping is relative to
the watermark instead of expiring windows, like early versions of Beam. The
other difference is that it track the latest event (some call it a "high
water mark" because it is the highest datetime value seen) where Beam's
watermark is an approximation of the earliest (some call it a "low water
mark" because it is a guarantee that it will not dip lower). When I chatted
about this with Amit in the early days, it was necessary to implement a
Beam-style watermark using Spark state. I think that may still be the case,
for correct results.

Also, I started a doc[2] to produce an updated blog post since the original
> SplittableDoFn blog from 2017 is out of date[3]. I was thinking of making
> this a new blog post and having the old blog post point to it. We could
> also remove the old blog post and or update it. Any thoughts?
>

New blog post w/ pointer from the old one.

Finally, I have a clean-up PR[4] that pushes the Read -> primitive Read
> expansion into each of the runners instead of having it within Read
> transform within beam-sdks-java-core.
>

Approved! I did CC a bunch of runner authors already. I think the important
thing is if a default changes we should be sure everyone is OK with the
perf changes, and everyone is confident that no incorrect results are
produced. The abstractions between sdk-core, runners-core-*, and individual
runners is important to me:

 - The SDK's job is to produce a portable, un-tweaked pipeline so moving
flags out of SDK core (and IOs) ASAP is super important.
 - The runner's job is to execute that pipeline, if they can, however they
want. If a runner wants to run Read transforms differently/directly that is
fine. If a runner is incapable of supporting SDF, then Read is better than
nothing. Etc.
 - The runners-core-* job is to just be internal libraries for runner
authors to share code, and should not make any decisions about the Beam
model, etc.

Kenn

1: https://github.com/apache/beam/pull/12603
> 2: http://doc/1kpn0RxqZaoacUPVSMYhhnfmlo8fGT-p50fEblaFr2HE
> 3: https://beam.apache.org/blog/splittable-do-fn/
> 4: https://github.com/apache/beam/pull/13006
>
>
> On Fri, Aug 28, 2020 at 1:45 AM Maximilian Michels  wrote:
>
>> Thanks Luke! I've had a pass.
>>
>> -Max
>>
>> On 28.08.20 01:22, Luke Cwik wrote:
>> > As an update.
>> >
>> > Direct and Twister2 are done.
>> > Samza: is ready for review[1].
>> > Flink: is almost ready for review. [2] lays all the groundwork for the
>> > migration and [3] finishes the migration (there is a timeout happening
>> > in FlinkSubmissionTest that I'm trying to figure out).
>> > No further updates on Spark[4] or Jet[5].
>> >
>> > @Maximilian Michels  or @t...@apache.org
>> > , can either of you take a look at the
>> > Flink PRs?
>> > @ke.wu...@icloud.com , Since Xinyu
>> delegated
>> > to you, can you take another look at the Samza PR?
>> >
>> > 1: https://github.com/apache/beam/pull/12617
>> > 2: https://github.com/apache/beam/pull/12706
>> > 3: https://github.com/apache/beam/pull/12708
>> > 4: https://github.com/apache/beam/pull/12603
>> > 5: https://github.com/apache/beam/pull/12616
>> >
>> > On Tue, Aug 18, 2020 at 11:42 AM Pulasthi Supun Wickramasinghe
>> > mailto:pulasthi...@gmail.com>> wrote:
>> >
>> > Hi Luke
>> >
>> > Will take a look at this as soon as possible and get back to you.
>> >
>> > Best Regards,
>> > Pulasthi
>> >
>> > On Tue, Aug 18, 2020 at 2:30 PM Luke Cwik > > > wrote:
>> >
>> > I have made some good progress here and have gotten to the
>> > following state for non-portable runners:
>> >
>> > DirectRunner[1]: Merged. Supports Read.Bounded and
>> Read.Unbounded.
>> > Twister2[2]: Ready for review. Supports Read.Bounded, the
>> > current runner doesn't support unbounded pipelines.
>> > Spark[3]: WIP. Supports Read.Bounded, Nexmark 

Re: [UPDATE] Beam 2.25.0 release progress update

[Robin Qiu]

One thing that could block my work soon is that I am still not added to
the beammaintainers team in Docker Hub. Pablo has helped me created a infra
ticket (https://issues.apache.org/jira/browse/INFRA-20900) but it has not
made progress. Could someone help with expediting this? Thanks!

On Mon, Oct 5, 2020 at 3:39 PM Robin Qiu  wrote:

> Update: the release blockers are all resolved and the fixes are
> cherry-picked into the release branch. I will work on creating a release
> candidate soon.
>
> On Fri, Sep 25, 2020 at 11:36 AM Robin Qiu  wrote:
>
>> Update: here is the list of release blockers to be resolved
>> https://issues.apache.org/jira/projects/BEAM/versions/12347147
>>
>> On Thu, Sep 24, 2020 at 5:00 PM Robin Qiu  wrote:
>>
>>> I see. Thanks for letting me know!
>>>
>>> On Thu, Sep 24, 2020 at 4:59 PM Robert Bradshaw 
>>> wrote:
>>>
 FYI, I'm waiting for Daniel's CLs to go in before starting my own.

 On Thu, Sep 24, 2020 at 9:29 AM Robert Bradshaw 
 wrote:

> OK, I'll get that started.
>
> On Thu, Sep 24, 2020 at 9:26 AM Robin Qiu  wrote:
>
>> No I haven't. Could you please do that? I wanted to sync with you
>> yesterday but I saw you were ooo.
>>
>> On Thu, Sep 24, 2020 at 9:03 AM Robert Bradshaw 
>> wrote:
>>
>>> [just you]
>>>
>>> Have you started an import at this commit, or should I?
>>>
>>> On Wed, Sep 23, 2020 at 5:51 PM Robin Qiu 
>>> wrote:
>>>
 Update: The 2.25.0 release branch has been cut.


 On Wed, Sep 23, 2020 at 3:34 PM Robin Qiu 
 wrote:

> Hello everyone,
>
> I am starting this email thread to share with you updates on Beam
> 2.25.0 release.
>
> Update: I plan to cut the release branch in a few hours, according
> to the release calendar [1].
>
> Thanks,
> Robin
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>
>
>


>>>
>>>

Re: [DISCUSS][BEAM-10670] Migrating BoundedSource/UnboundedSource to execute as a Splittable DoFn for non-portable Java runners

[Luke Cwik]

For the 2.25 release the Java Direct, Flink, Jet, Samza, Twister2 will use
SDF powered Read transforms. Users can opt-out
with --experiments=use_deprecated_read.

I was unable to get Spark done for 2.25 as I found out that Spark streaming
doesn't support watermark holds[1]. If someone knows more about the
watermark system in Spark I could use some guidance here as I believe I
have a version of unbounded SDF support written for Spark (I get all the
expected output from tests, just that watermarks aren't being held back so
PAssert fails).

Also, I started a doc[2] to produce an updated blog post since the original
SplittableDoFn blog from 2017 is out of date[3]. I was thinking of making
this a new blog post and having the old blog post point to it. We could
also remove the old blog post and or update it. Any thoughts?

Finally, I have a clean-up PR[4] that pushes the Read -> primitive Read
expansion into each of the runners instead of having it within Read
transform within beam-sdks-java-core.

1: https://github.com/apache/beam/pull/12603
2: http://doc/1kpn0RxqZaoacUPVSMYhhnfmlo8fGT-p50fEblaFr2HE
3: https://beam.apache.org/blog/splittable-do-fn/
4: https://github.com/apache/beam/pull/13006


On Fri, Aug 28, 2020 at 1:45 AM Maximilian Michels  wrote:

> Thanks Luke! I've had a pass.
>
> -Max
>
> On 28.08.20 01:22, Luke Cwik wrote:
> > As an update.
> >
> > Direct and Twister2 are done.
> > Samza: is ready for review[1].
> > Flink: is almost ready for review. [2] lays all the groundwork for the
> > migration and [3] finishes the migration (there is a timeout happening
> > in FlinkSubmissionTest that I'm trying to figure out).
> > No further updates on Spark[4] or Jet[5].
> >
> > @Maximilian Michels  or @t...@apache.org
> > , can either of you take a look at the
> > Flink PRs?
> > @ke.wu...@icloud.com , Since Xinyu
> delegated
> > to you, can you take another look at the Samza PR?
> >
> > 1: https://github.com/apache/beam/pull/12617
> > 2: https://github.com/apache/beam/pull/12706
> > 3: https://github.com/apache/beam/pull/12708
> > 4: https://github.com/apache/beam/pull/12603
> > 5: https://github.com/apache/beam/pull/12616
> >
> > On Tue, Aug 18, 2020 at 11:42 AM Pulasthi Supun Wickramasinghe
> > mailto:pulasthi...@gmail.com>> wrote:
> >
> > Hi Luke
> >
> > Will take a look at this as soon as possible and get back to you.
> >
> > Best Regards,
> > Pulasthi
> >
> > On Tue, Aug 18, 2020 at 2:30 PM Luke Cwik  > > wrote:
> >
> > I have made some good progress here and have gotten to the
> > following state for non-portable runners:
> >
> > DirectRunner[1]: Merged. Supports Read.Bounded and
> Read.Unbounded.
> > Twister2[2]: Ready for review. Supports Read.Bounded, the
> > current runner doesn't support unbounded pipelines.
> > Spark[3]: WIP. Supports Read.Bounded, Nexmark suite passes. Not
> > certain about level of unbounded pipeline support coverage since
> > Spark uses its own tiny suite of tests to get unbounded pipeline
> > coverage instead of the validates runner set.
> > Jet[4]: WIP. Supports Read.Bounded. Read.Unbounded definitely
> > needs additional work.
> > Sazma[5]: WIP. Supports Read.Bounded. Not certain about level of
> > unbounded pipeline support coverage since Spark uses its own
> > tiny suite of tests to get unbounded pipeline coverage instead
> > of the validates runner set.
> > Flink: Unstarted.
> >
> > @Pulasthi Supun Wickramasinghe  ,
> > can you help me with the Twister2 PR[2]?
> > @Ismaël Mejía , is PR[3] the expected
> > level of support for unbounded pipelines and hence ready for
> review?
> > @Jozsef Bartok , can you help me out
> > to get support for unbounded splittable DoFn's into Jet[4]?
> > @Xinyu Liu , is PR[5] the expected
> > level of support for unbounded pipelines and hence ready for
> review?
> >
> > 1: https://github.com/apache/beam/pull/12519
> > 2: https://github.com/apache/beam/pull/12594
> > 3: https://github.com/apache/beam/pull/12603
> > 4: https://github.com/apache/beam/pull/12616
> > 5: https://github.com/apache/beam/pull/12617
> >
> > On Tue, Aug 11, 2020 at 10:55 AM Luke Cwik  > > wrote:
> >
> > There shouldn't be any changes required since the wrapper
> > will smoothly transition the execution to be run as an SDF.
> > New IOs should strongly prefer to use SDF since it should be
> > simpler to write and will be more flexible but they can use
> > the "*Source"-based APIs. Eventually we'll 

Re: Self-checkpoint Support on Portable Flink

[Luke Cwik]

Thanks Boyuan, I left a few comments.

On Mon, Oct 5, 2020 at 11:12 AM Boyuan Zhang  wrote:

> Hi team,
>
> I'm looking at adding self-checkpoint support to portable Flink runner(
> BEAM-10940 ) for both
> batch and streaming. I summarized the problem that we want to solve and
> proposed 2 potential approaches in this doc
> 
> .
> I want to collect feedback on which approach is preferred and anything
> that I have not taken into consideration yet but I should.
> Many thanks to all your help!
>
> Boyuan
>
>

Re: Support streaming side-inputs in the Spark runner

[Ismaël Mejía]

The limitation of non being able to have side inputs in streaming has been
pending since a long time ago, and there is sadly not anyone to my
knowledge working on this.

One extra aspect to have in mind is that the support for streaming in the
Spark runner uses the Spark DStream API which does not have explicit
watermark handling so the runner has to implement this, this is where the
issues of lacking holds exist, this hopefully could be fixed in the future.

One important detail is that the native Spark Structured Streaming API has
the concept of Watermark but it is more limited than the definition of
Watermark on Beam so even the Spark runner that uses the new API would
require to implement Watermark handling too.

Notice that if you go with native Spark Streaming instead of Beam please
pay attention to the fact that the support for streaming is still limited
for example you cannot do two aggregations in sequence (this is a
limitation that affects the ongoing work on the Structured Streaming
runner) https://issues.apache.org/jira/browse/SPARK-26655 also less
important but still good to know you cannot do continuous processing on
Spark save for  map-like operations, so you should default to the
micro-batch based approach if you any aggregation.
https://issues.apache.org/jira/browse/SPARK-20928



On Fri, Oct 2, 2020 at 9:05 PM Luke Cwik  wrote:

> Support for watermark holds is missing for both Spark streaming
> implementations (DStream and structured streaming) so watermark based
> triggers don't produce the correct output.
>
> Excluding the direct runner, Flink is the OSS runner with the most people
> working on it adding features and fixing bugs in it.
> Spark batch is in a good state but streaming development is still ongoing
> and also has a small group of folks.
>
>
> On Fri, Oct 2, 2020 at 10:16 AM  wrote:
>
>> For clarification, is it just streaming side inputs that present an issue
>> for SparkRunner or are there other areas that need work?  We've started
>> work on a Beam-based project that includes both streaming and batch
>> oriented work and a Spark cluster was our choice due to the perception that
>> it could handle both types of applications.
>>
>> However, that would have to be reevaluated if SparkRunner isn't up for
>> streaming deployments.  And it seems that SparkStructuredStreamingRunner
>> still needs some time before it's a fully-featured solution.  I guess I'm
>> trying to get a sense of whether these runners are still being actively
>> developed or were they donated by a third-party and are now suffering from
>> bit-rot.
>>
>> Oct 1, 2020, 10:54 by lc...@google.com:
>>
>> I would suggest trying FlinkRunner as it is a much more complete
>> streaming implementation.
>> SparkRunner has several key things that are missing that won't allow your
>> pipeline to function correctly.
>> If you're really invested in getting SparkRunner working though feel free
>> to contribute the necessary implementations for watermark holds and
>> broadcast state necessary for side inputs.
>>
>> On Tue, Sep 29, 2020 at 9:06 AM Rajagopal, Viswanathan <
>> rajagop...@dnb.com> wrote:
>>
>> Hi Team,
>>
>>
>>
>> I have a streaming pipeline (built using Apache Beam with Spark
>> Runner)which consumes events tagged with timestamps from Unbounded source
>> (Kinesis Stream) and batch them into FixedWindows of 5 mins each and then,
>> write all events in a window into a single / multiple files based on shards.
>>
>> We are trying to achieve the following through Apache Beam constructs
>>
>> *1.   **Create a PCollectionView from unbounded source and pass it
>> as a side-input to our main pipeline.*
>>
>> *2.   **Have a hook method that invokes per window that enables us
>> to do some operational activities per window.*
>>
>> *3.   **Stop the stream processor (graceful stop) from external
>> system.*
>>
>>
>>
>> *Approaches that we tried for 1).*
>>
>> ·Creating a PCollectionView from unbounded source and pass it as
>> a side-input to our main pipeline.
>>
>> ·Input Pcollection goes through FixedWindow transform.
>>
>> ·Created custom CombineFn that takes combines all inputs for a
>> window and produce single value Pcollection.
>>
>> ·Output of Window transform it goes to CombineFn (custom fn) and
>> creates a PCollectionView from CombineFn (using
>> Combine.Globally().asSingletonView() as this output would be passed as a
>> side-input for our main pipeline.
>>
>> o   Getting the following exception (while running with streaming option
>> set to true)
>>
>> ·java.lang.IllegalStateException: No TransformEvaluator
>> registered for UNBOUNDED transform View.CreatePCollectionView
>>
>>- Noticed that SparkRunner doesn’t support the streaming side-inputs
>>   in the Spark runner
>>   -
>>  
>> https://www.javatips.net/api/beam-master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java
>>  

Beam Dependency Check Report (2020-10-05)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
chromedriver-binary
83.0.4103.39.0
86.0.4240.22.0
2020-07-08
2020-09-07BEAM-10426
google-cloud-bigquery
1.28.0
2.0.0
None
2020-10-05BEAM-5537
google-cloud-dlp
1.0.0
2.0.0
2020-06-29
2020-10-05BEAM-10344
google-cloud-pubsub
1.7.0
2.1.0
2020-07-20
2020-10-05BEAM-5539
google-cloud-vision
1.0.0
2.0.0
2020-03-24
2020-10-05BEAM-9581
mock
2.0.0
4.0.2
2019-05-20
2020-10-05BEAM-7369
mypy-protobuf
1.18
1.23
2020-03-24
2020-06-29BEAM-10346
nbconvert
5.6.1
6.0.7
None
2020-10-05BEAM-11007
pyarrow
0.17.1
1.0.1
2020-07-27
2020-08-24BEAM-10582
PyHamcrest
1.10.1
2.0.2
2020-01-20
2020-07-08BEAM-9155
pytest
4.6.11
6.1.1
2020-07-08
2020-10-05BEAM-8606
pytest-xdist
1.34.0
2.1.0
2020-08-17
2020-08-28BEAM-10713
tenacity
5.1.5
6.2.0
2019-11-11
2020-06-29BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.amazonaws:amazon-kinesis-producer
0.13.1
0.14.1
2019-07-31
2020-07-31BEAM-10628
com.azure:azure-storage-blob
12.1.0
12.9.0-beta.1
2019-12-05
2020-10-01BEAM-10800
com.datastax.cassandra:cassandra-driver-core
3.8.0
4.0.0
2019-10-29
2019-03-18BEAM-8674
com.esotericsoftware:kryo
4.0.2
5.0.0-RC9
2018-03-20
2020-08-14BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.29.0
0.33.0
2020-07-20
2020-09-14BEAM-6645
com.google.api.grpc:grpc-google-cloud-pubsub-v1
1.85.1
1.90.1
2020-03-09
2020-08-04BEAM-8677
com.google.api.grpc:grpc-google-cloud-pubsublite-v1
0.1.6
0.4.1
2020-05-28
2020-09-28BEAM-11008
com.google.api.grpc:grpc-google-common-protos
1.12.0
1.18.1
2018-06-29
2020-08-11BEAM-8633
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1
0.85.1
0.105.4
2020-01-08
2020-09-24BEAM-8678
com.google.api.grpc:proto-google-cloud-bigtable-v2
1.9.1
1.15.0
2020-01-10
2020-09-02BEAM-8679
com.google.api.grpc:proto-google-cloud-datastore-v1
0.85.0
0.88.0
2019-12-05
2020-09-17BEAM-8680
com.google.api.grpc:proto-google-cloud-pubsub-v1
1.85.1
1.90.1
2020-03-09
2020-08-04BEAM-8681
com.google.api.grpc:proto-google-cloud-pubsublite-v1
0.1.6
0.4.1
2020-05-28
2020-09-28BEAM-11009
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1
1.59.0
2.0.2
2020-07-16
2020-10-02BEAM-8682
com.google.apis:google-api-services-bigquery
v2-rev20200719-1.30.10
v2-rev20200925-1.30.10
2020-07-26
2020-10-01BEAM-8684
com.google.apis:google-api-services-clouddebugger
v2-rev20200501-1.30.10
v2-rev20200807-1.30.10
2020-07-14
2020-08-17BEAM-8750
com.google.apis:google-api-services-cloudresourcemanager
v1-rev20200720-1.30.10
v2-rev20200831-1.30.10
2020-07-25
2020-09-03BEAM-8751
com.google.apis:google-api-services-dataflow
v1b3-rev20200713-1.30.10
v1beta3-rev12-1.20.0
2020-07-25
2015-04-29BEAM-8752
com.google.apis:google-api-services-healthcare
v1beta1-rev20200713-1.30.10
v1-rev20200917-1.30.10
2020-07-24
2020-09-23BEAM-10349
com.google.apis:google-api-services-pubsub
v1-rev20200713-1.30.10
v1-rev20200909-1.30.10
2020-07-25
2020-09-18BEAM-8753
com.google.apis:google-api-services-storage
v1-rev20200611-1.30.10
v1-rev20200927-1.30.10
2020-07-10
2020-10-03BEAM-8754
com.google.auto.service:auto-service
1.0-rc6
1.0-rc7
2019-07-16