Re: [Question] Python Batch Pipeline On Flink - No Output

[Shankar Mane]

Please help..

On Fri, 4 Jun 2021, 19:58 Shankar Mane,  wrote:

> Team,
>
> I am new to Beam. And running a simple python pipeline on flink as runner.
> Pipelines get Successfully Finished But i am not getting expected output at
> the end.
>
> #
>
> Here is Pipeline code :
>
> #!/usr/bin/python3
>>
>> import argparse
>> import logging
>> import re
>> import typing
>>
>> import apache_beam as beam
>> from apache_beam.io import ReadFromText
>> from apache_beam.io import WriteToText
>> from apache_beam.io.kafka import ReadFromKafka, WriteToKafka
>>
>> from apache_beam.options.pipeline_options import PipelineOptions
>> from apache_beam.options.pipeline_options import SetupOptions
>> from decimal import Decimal
>>
>>
>> class AmoountExtractingDoFn(beam.DoFn):
>> def process(self, element):
>> # Returns an iterator over the words of this element.
>> try:
>> strArr = str(element).replace("\"", "").split(",")
>> print(strArr)
>> return [(strArr[1], float(strArr[2]))]
>> except Exception as e :
>> pass
>>
>> # Format the counts into a PCollection of strings.
>> def format_result(userid, amount):
>> try:
>> return '%s = %d' % (userid, amount)
>> except Exception as e:
>> pass
>>
>> def run(argv=None, save_main_session=True):
>>   """Main entry point; defines and runs the wordcount pipeline."""
>>   parser = argparse.ArgumentParser()
>>   parser.add_argument(
>>   '--input',
>>   dest='input',
>>   default='gs://dataflow-samples/shakespeare/kinglear.txt',
>>   help='Input file to process.')
>>   parser.add_argument(
>>   '--output',
>>   dest='output',
>>   required=True,
>>   help='Output file to write results to.')
>>   known_args, pipeline_args = parser.parse_known_args(argv)
>>
>>   # We use the save_main_session option because one or more DoFn's in this
>>   # workflow rely on global context (e.g., a module imported at module 
>> level).
>>   pipeline_options = PipelineOptions(pipeline_args)
>>   pipeline_options.view_as(SetupOptions).save_main_session = 
>> save_main_session
>>
>>   # The pipeline will be run on exiting the with block.
>>   with beam.Pipeline(options=pipeline_options) as p:
>> # Read the text file[pattern] into a PCollection.
>> lines = p | 'Read' >> ReadFromText(known_args.input)
>> counts = (
>> lines
>> | 'ExtractAmountPerUser' >> (beam.ParDo(AmoountExtractingDoFn()))
>> | 'GroupAndSum' >> beam.CombinePerKey(sum))
>>
>> output = counts | 'Format' >> beam.MapTuple(format_result)
>>
>> output | beam.Map(lambda x: (b'', 
>> x.encode('utf-8'))).with_output_types(typing.Tuple[bytes, bytes])\
>> | 'Write to Kafka' >> WriteToKafka(producer_config={'bootstrap.servers': 
>> '10.xxx.xxx.xxx:9092'}, topic='test2')
>>
>>
>
> #---
> -
>
> Here is CMDs to run above pipeline :
>
> CMD-1 : This pipeline is working fine on flink, finished successfully and
> output also gets generated at the end.
> CMD-2: This pipeline is working fine on flink, finished successfully and
> NO OUTPUT at the end.
>
>
>
>> # CMD-1 :
>> *WORKING - output gets generate*python3 batch.py \
>> --input beam-userbase.csv \
>> --output output/batch \
>> --runner=FlinkRunner \
>> --flink_submit_uber_jar \
>> --flink_master=localhost:8090 \
>> --environment_type=LOOPBACK
>>
>>
>
>> *# CMD-2:   **WORKING - NO OUTPUT*
>
> python3 batch.py \
>> --input beam-userbase.csv \
>> --output output/batch \
>> --runner=FlinkRunner \
>> --flink_submit_uber_jar \
>> --flink_master=localhost:8090 \
>> --environment_type=DOCKER \
>> --environment_config="apache/beam_python3.7_sdk:2.29.0"
>
>
>

Re: Introduction - Contribute

[Pablo Estrada]

Welcome Leo!
I've added you as a contributor on Jira.
Best
-P.

On Fri, Jun 18, 2021 at 12:36 PM Leonardo Campos <
campos.leona...@hotmail.com> wrote:

> Hi,
>
> my Apache Jira ID is leocampos
>
> Thanks
>
> --
> *De:* Alexey Romanenko 
> *Enviado:* segunda-feira, 14 de junho de 2021 09:00
> *Para:* dev@beam.apache.org 
> *Assunto:* Re: Introduction - Contribute
>
> Hi Leonardo,
>
> Great to hear that you are going to contribute to Beam!
>
> Feel free to take a look on Beam’s Contribution guide [1] in the first
> order. It should answer most of the potential questions but, if not, then
> don’t hesitate to send them to this mailing list.
>
> If you don’t have yet a precise idea from which area to start your
> contributions, please, check the starter tasks [2] - they should help you
> with your development setup and make comfortable with Beam codebase.
>
> Also, feel free to send your Apache Jira ID here and then you can assign
> Jira tasks to yourself.
>
> Welcome to Beam!
>
> —
> Alexey
>
> [1] https://beam.apache.org/contribute/
> [2] https://s.apache.org/beam-starter-tasks
>
> On 13 Jun 2021, at 09:21, Leonardo Campos 
> wrote:
>
> Hello,
>
> I would like to start contributing to Apache Beam in my free time.
>
> I'm currently working as a BigData Engineer at GameDuell, a German gaming
> company.
> I've been working with Java for many years now (since around 2004) and
> with Apache Beam since 2018 (I think).
>
> Github: https://github.com/leocampos
>
> Best regards,
> Leonardo Campos
>
>
>

Re: [PROPOSAL] Preparing for Beam 2.31.0 release

[Andrew Pilloud]

We had a few more release blockers show up this week, they are now all
resolved. I'll make another attempt at RC1 on Monday.

On Mon, Jun 14, 2021 at 5:22 PM Andrew Pilloud  wrote:

> All release blocking issues are now resolved. I will start building RC1
> tomorrow.
>
> On Mon, Jun 7, 2021 at 2:11 PM Andrew Pilloud  wrote:
>
>> The 2.31.0 release branch was cut based on the last commit on June 2nd
>> and is now available for cherry-picks[1]. There are currently 3 open
>> release blocking issues[2].
>>
>> Andrew
>>
>> [1] https://github.com/apache/beam/commits/release-2.31.0
>> [2]
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20is%20EMPTY%20AND%20fixVersion%20%3D%202.31.0
>>
>> On Wed, May 26, 2021 at 11:12 AM Ahmet Altay  wrote:
>>
>>> +1. Thank you Andrew.
>>>
>>> On Thu, May 20, 2021 at 1:15 PM Andrew Pilloud 
>>> wrote:
>>>
 Hi All,

 Beam 2.31.0 release is scheduled to be cut in two weeks, on June 2
 according to the release calendar [1].

 I'd like to volunteer myself to be the release manager for
 this release. I plan on cutting the release branch on the scheduled date.
 That week starts with a US holiday and I will be on vacation the entire
 week so I don't expect to make any progress on the release until June 7th.
 If there is a desire I could delay the release cut by a few days.

 Any comments or objections?

 Andrew

 [1]
 https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com&ctz=America/Los_Angeles

>>>

RE: Introduction - Contribute

[Leonardo Campos]

Hi,

my Apache Jira ID is leocampos

Thanks


De: Alexey Romanenko 
Enviado: segunda-feira, 14 de junho de 2021 09:00
Para: dev@beam.apache.org 
Assunto: Re: Introduction - Contribute

Hi Leonardo,

Great to hear that you are going to contribute to Beam!

Feel free to take a look on Beam’s Contribution guide [1] in the first order. 
It should answer most of the potential questions but, if not, then don’t 
hesitate to send them to this mailing list.

If you don’t have yet a precise idea from which area to start your 
contributions, please, check the starter tasks [2] - they should help you with 
your development setup and make comfortable with Beam codebase.

Also, feel free to send your Apache Jira ID here and then you can assign Jira 
tasks to yourself.

Welcome to Beam!

—
Alexey

[1] https://beam.apache.org/contribute/
[2] https://s.apache.org/beam-starter-tasks

On 13 Jun 2021, at 09:21, Leonardo Campos 
mailto:campos.leona...@hotmail.com>> wrote:

Hello,

I would like to start contributing to Apache Beam in my free time.

I'm currently working as a BigData Engineer at GameDuell, a German gaming 
company.
I've been working with Java for many years now (since around 2004) and with 
Apache Beam since 2018 (I think).

Github: https://github.com/leocampos

Best regards,
Leonardo Campos

Flaky test issue report (29)

[Beam Jira Bot]

This is your daily summary of Beam's current flaky tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20labels%20%3D%20flake)

These are P1 issues because they have a major negative impact on the community 
and make it hard to determine the quality of the software.

https://issues.apache.org/jira/browse/BEAM-12515: Python PreCommit flaking 
in PipelineOptionsTest.test_display_data (created 2021-06-18)
https://issues.apache.org/jira/browse/BEAM-12322: 
FnApiRunnerTestWithGrpcAndMultiWorkers flaky (py precommit) (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12309: 
PubSubIntegrationTest.test_streaming_data_only flake (created 2021-05-07)
https://issues.apache.org/jira/browse/BEAM-12307: 
PubSubBigQueryIT.test_file_loads flake (created 2021-05-07)
https://issues.apache.org/jira/browse/BEAM-12303: Flake in 
PubSubIntegrationTest.test_streaming_with_attributes (created 2021-05-06)
https://issues.apache.org/jira/browse/BEAM-12291: 
org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: 
false] is flaky (created 2021-05-05)
https://issues.apache.org/jira/browse/BEAM-12200: 
SamzaStoreStateInternalsTest is flaky (created 2021-04-20)
https://issues.apache.org/jira/browse/BEAM-12163: Python GHA PreCommits 
flake with grpc.FutureTimeoutError on SDK harness startup (created 2021-04-13)
https://issues.apache.org/jira/browse/BEAM-12061: beam_PostCommit_SQL 
failing on KafkaTableProviderIT.testFakeNested (created 2021-03-27)
https://issues.apache.org/jira/browse/BEAM-12019: 
apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
 is flaky (created 2021-03-18)
https://issues.apache.org/jira/browse/BEAM-11792: Python precommit failed 
(flaked?) installing package  (created 2021-02-10)
https://issues.apache.org/jira/browse/BEAM-11666: 
apache_beam.runners.interactive.recording_manager_test.RecordingManagerTest.test_basic_execution
 is flaky (created 2021-01-20)
https://issues.apache.org/jira/browse/BEAM-11661: hdfsIntegrationTest 
flake: network not found (py38 postcommit) (created 2021-01-19)
https://issues.apache.org/jira/browse/BEAM-11645: beam_PostCommit_XVR_Flink 
failing (created 2021-01-15)
https://issues.apache.org/jira/browse/BEAM-11541: 
testTeardownCalledAfterExceptionInProcessElement flakes on direct runner. 
(created 2020-12-30)
https://issues.apache.org/jira/browse/BEAM-10968: flaky test: 
org.apache.beam.sdk.metrics.MetricsTest$AttemptedMetricTests.testAttemptedDistributionMetrics
 (created 2020-09-25)
https://issues.apache.org/jira/browse/BEAM-10955: Flink Java Runner test 
flake: Could not find Flink job  (created 2020-09-23)
https://issues.apache.org/jira/browse/BEAM-10866: 
PortableRunnerTestWithSubprocesses.test_register_finalizations flaky on macOS 
(created 2020-09-09)
https://issues.apache.org/jira/browse/BEAM-10485: Failure / flake: 
ElasticsearchIOTest > testWriteWithIndexFn (created 2020-07-14)
https://issues.apache.org/jira/browse/BEAM-9649: 
beam_python_mongoio_load_test started failing due to mismatched results 
(created 2020-03-31)
https://issues.apache.org/jira/browse/BEAM-9232: 
BigQueryWriteIntegrationTests is flaky coercing to Unicode (created 2020-01-31)
https://issues.apache.org/jira/browse/BEAM-8101: Flakes in 
ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful for 
Direct, Spark, Flink (created 2019-08-27)
https://issues.apache.org/jira/browse/BEAM-8035: 
WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp 
order (sickbayed) (created 2019-08-22)
https://issues.apache.org/jira/browse/BEAM-7992: Unhandled type_constraint 
in 
apache_beam.io.gcp.bigquery_write_it_test.BigQueryWriteIntegrationTests.test_big_query_write_new_types
 (created 2019-08-16)
https://issues.apache.org/jira/browse/BEAM-7827: 
MetricsTest$AttemptedMetricTests.testAllAttemptedMetrics is flaky on 
DirectRunner (created 2019-07-26)
https://issues.apache.org/jira/browse/BEAM-7752: Java Validates 
DirectRunner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky 
(created 2019-07-16)
https://issues.apache.org/jira/browse/BEAM-6804: [beam_PostCommit_Java] 
[PubsubReadIT.testReadPublicData] Timeout waiting on Sub (created 2019-03-11)
https://issues.apache.org/jira/browse/BEAM-5286: 
[beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake]
 .sh script: text file busy. (created 2018-09-01)
https://issues.apache.org/jira/browse/BEAM-5172: 
org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky (created 
2018-08-20)

P1 issues report (43)

[Beam Jira Bot]

This is your daily summary of Beam's current P1 issues, not including flaky 
tests 
(https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20statusCategory%20!%3D%20Done%20AND%20priority%20%3D%20P1%20AND%20(labels%20is%20EMPTY%20OR%20labels%20!%3D%20flake).

See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the 
meaning and expectations around P1 issues.

https://issues.apache.org/jira/browse/BEAM-12508: Compiled 
gradle-wrapper.jar part of release (created 2021-06-17)
https://issues.apache.org/jira/browse/BEAM-12507: Remove website from 
release (created 2021-06-17)
https://issues.apache.org/jira/browse/BEAM-12500: Dataflow SocketException 
(SSLException) error while trying to send message from Cloud Pub/Sub to 
BigQuery (created 2021-06-16)
https://issues.apache.org/jira/browse/BEAM-12484: JdbcIO date conversion is 
sensitive to OS (created 2021-06-14)
https://issues.apache.org/jira/browse/BEAM-12467: 
java.io.InvalidClassException With Flink Kafka (created 2021-06-09)
https://issues.apache.org/jira/browse/BEAM-12436: 
[beam_PostCommit_Go_VR_flink| beam_PostCommit_Go_VR_spark] 
[:sdks:go:test:flinkValidatesRunner] Failure summary (created 2021-06-01)
https://issues.apache.org/jira/browse/BEAM-12422: Vendored gRPC 1.36.0 is 
using a log4j version with security issues (created 2021-05-28)
https://issues.apache.org/jira/browse/BEAM-12396: 
beam_PostCommit_XVR_Direct failed (flaked?) (created 2021-05-24)
https://issues.apache.org/jira/browse/BEAM-12389: 
beam_PostCommit_XVR_Dataflow flaky: Expand method not found (created 2021-05-21)
https://issues.apache.org/jira/browse/BEAM-12387: beam_PostCommit_Python* 
timing out (created 2021-05-21)
https://issues.apache.org/jira/browse/BEAM-12386: 
beam_PostCommit_Py_VR_Dataflow(_V2) failing metrics tests (created 2021-05-21)
https://issues.apache.org/jira/browse/BEAM-12380: Go SDK Kafka IO Transform 
implemented via XLang (created 2021-05-21)
https://issues.apache.org/jira/browse/BEAM-12374: Spark postcommit failing 
ResumeFromCheckpointStreamingTest (created 2021-05-20)
https://issues.apache.org/jira/browse/BEAM-12320: 
PubsubTableProviderIT.testSQLSelectsArrayAttributes[0] failing in SQL 
PostCommit (created 2021-05-10)
https://issues.apache.org/jira/browse/BEAM-12310: 
beam_PostCommit_Java_DataflowV2 failing (created 2021-05-07)
https://issues.apache.org/jira/browse/BEAM-12279: Implement 
destination-dependent sharding in FileIO.writeDynamic (created 2021-05-04)
https://issues.apache.org/jira/browse/BEAM-12256: 
PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some 
Avro logical types (created 2021-04-29)
https://issues.apache.org/jira/browse/BEAM-12076: Update Python 
cross-language Kafka source to read metadata (created 2021-03-31)
https://issues.apache.org/jira/browse/BEAM-11959: Python Beam SDK Harness 
hangs when installing pip packages (created 2021-03-11)
https://issues.apache.org/jira/browse/BEAM-11906: No trigger early 
repeatedly for session windows (created 2021-03-01)
https://issues.apache.org/jira/browse/BEAM-11875: XmlIO.Read does not 
handle XML encoding per spec (created 2021-02-26)
https://issues.apache.org/jira/browse/BEAM-11828: JmsIO is not 
acknowledging messages correctly (created 2021-02-17)
https://issues.apache.org/jira/browse/BEAM-11755: Cross-language 
consistency (RequiresStableInputs) is quietly broken (at least on portable 
flink runner) (created 2021-02-05)
https://issues.apache.org/jira/browse/BEAM-11578: `dataflow_metrics` 
(python) fails with TypeError (when int overflowing?) (created 2021-01-06)
https://issues.apache.org/jira/browse/BEAM-11434: Expose Spanner 
admin/batch clients in Spanner Accessor (created 2020-12-10)
https://issues.apache.org/jira/browse/BEAM-11148: Kafka 
commitOffsetsInFinalize OOM on Flink (created 2020-10-28)
https://issues.apache.org/jira/browse/BEAM-11017: Timer with dataflow 
runner can be set multiple times (dataflow runner) (created 2020-10-05)
https://issues.apache.org/jira/browse/BEAM-10670: Make non-portable 
Splittable DoFn the only option when executing Java "Read" transforms (created 
2020-08-10)
https://issues.apache.org/jira/browse/BEAM-10617: python 
CombineGlobally().with_fanout() cause duplicate combine results for sliding 
windows (created 2020-07-31)
https://issues.apache.org/jira/browse/BEAM-10569: SpannerIO tests don't 
actually assert anything. (created 2020-07-23)
https://issues.apache.org/jira/browse/BEAM-10529: Kafka XLang fails for 
?empty? key/values (created 2020-07-18)
https://issues.apache.org/jira/browse/BEAM-10288: Quickstart documents are 
out of date (created 2020-06-19)
https://issues.apache.org/jira/browse/BEAM-10244: Populate requirements 
cache fails on poetry-based packages (created 2020-06-11)
https://issues.apache.org/jira/browse/BEAM-10100: FileIO writeDynamic with 
AvroIO

Re: Aliasing Pub/Sub Lite IO in external repo

[Brian Hulette]

How will this be communicated to the user? The idea is that they will
discover PubsubLiteIO through their IDE as you described, but that will get
them to the Beam one that's subject to the long release cycle. Will it just
be documented somewhere that users should prefer
com.google.cloud.pubsublite.beam.PubsubLiteIO if there's a recent fix they
need?

I wonder if a similar result could be achieved just by making Beam's
PubsubLiteIO a stub with no implementation that directs users to the
com.google.cloud one somehow?

junit's matcher interface comes to mind as a precedent here. I have been
warned many times by
Matcher._dont_implement_Matcher___instead_extend_BaseMatcher_ [1].

[1]
https://junit.org/junit4/javadoc/4.13/org/hamcrest/Matcher.html#_dont_implement_Matcher___instead_extend_BaseMatcher_()

Brian

On Thu, Jun 17, 2021 at 3:56 PM Daniel Collins  wrote:

> > Question 1: How are you going to approach testing/CI?
> The pull requests in the java-pubsublite repo do not trigger Beam repo's
> CI. You want to deliver things to your customers after they are tested as
> much as possible.
>
> I'd like to run the integration tests in both locations. They would only
> be meaningful in the beam setup when we went to validate a version bump on
> the I/O.
>
> > Question2 : in the code below, what is the purpose of keeping the
> PubsubLiteIO in the Beam repo?
>
> Visibility and autocomplete. It means the core class will be in the beam
> javadoc and if you type `import org.apache.beam.sdk.io.gcp.pubsu` in an IDE
> you'll see pubsublite and PubsubLiteIO.
>
> On Thu, Jun 17, 2021 at 5:35 PM Tomo Suzuki  wrote:
>
>> Hi Daniel,
>> (You helped me apply some change to this strange setup a few months back.
>> Thank you for working on rectifying the situation.)
>>
>> I like that idea overall.
>>
>> Question 1: How are you going to approach testing/CI?
>> The pull requests in the java-pubsublite repo do not trigger Beam repo's
>> CI. You want to deliver things to your customers after they are tested as
>> much as possible.
>>
>>
>> Question2 : in the code below, what is the purpose of keeping the
>> PubsubLiteIO in the Beam repo?
>>
>> ```
>> class PubsubLiteIO extends com.google.cloud.pubsublite.beam.PubsubLiteIO
>> {}
>> 
>>
>> The backward compatibility came to my mind but I thought you may have
>> more reasons.
>>
>>
>> My memo:
>> java-pubsublite repsitory has:
>> https://github.com/googleapis/java-pubsublite/blob/master/pubsublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteIO.java
>> beam repo has:
>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java
>> (and other files in the same directory)
>> google-cloud-pubsublite is not part of the Libraries BOM (yet) because of
>> its pre-1.0 status.
>>
>>
>> On Thu, Jun 17, 2021 at 5:07 PM Daniel Collins 
>> wrote:
>>
>>> I don't know that the cycle would cause a problem- wouldn't it override
>>> and cause it to use beam-sdks-java-core:2.30.0 (at least until beam goes to
>>> 3.X.X)?
>>>
>>> Something we can do if this is an issue is mark pubsublite-beam-io's dep
>>> on beam-sdks-java-core as 'provided'. But I'd prefer to avoid this and just
>>> let overriding fix it if that works.
>>>
>>> On Thu, Jun 17, 2021 at 4:15 PM Andrew Pilloud 
>>> wrote:
>>>
 How do you plan to address the circular dependency? Won't this end up
 with Beam depending on older versions of itself?

 beam-sdks-java-io-google-cloud-platform:2.30.0 ->
 pubsublite-beam-io:0.16.0 -> beam-sdks-java-core:2.29.0

 On Thu, Jun 17, 2021 at 11:56 AM Daniel Collins 
 wrote:

> Hello beam developers,
>
> I'm the primary author of the Pub/Sub Lite I/O, and I'd like to get
> some feedback on a change to the model for hosting this I/O in beam. Our
> team has been frustrated by the fact that we have no way to release
> features or fixes for bugs to customers on time scales shorter than the 
> 1-2
> months of the beam release cycle, and that those fixes are necessarily
> coupled with a beam version upgrade. To work around this, I forked the I/O
> in beam to our own repo about 6 months ago and have been maintaining both
> copies in parallel.
>
> I'd like to retain our ability to quickly fix and improve the I/O
> while retaining end-user visibility within the beam repo. To do this, I'd
> like to remove all the implementation from the beam repo, and leave the 
> I/O
> there implemented as:
>
> ```
> class PubsubLiteIO extends
> com.google.cloud.pubsublite.beam.PubsubLiteIO {}
> 
> , and add a dependency on our beam artifact.
>
> This enables beam users who want to just use the
> beam-sdks-java-io-google-cloud-platform artifact to do so, but they can
> also track the canonical version separately in our repo to get fixes and
> improvements at a faster rate. Al

Re: Java precomit failing, (though no test are failing)

[Alexey Romanenko]

Can this fail https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3745/ 
be related ? Though, there is only one test that fails.

> On 17 Jun 2021, at 18:50, Alex Amato  wrote:
> 
> Hmm, perhaps it only happens sometimes. The other half of the time I "Run 
> Java Precommit" on this PR I hit this different failure:
> 
> The connection is not obvious to me, if its related to my PR. 
> https://github.com/apache/beam/pull/14804 
> 
> I only added some Precondition checks. But I don't see those failing anywhere.
> (Unless something indirect is causing it and stacktrace for that is not 
> printed, i.e. like in a subprocess).
> 
> Any ideas? Are these tests known to be failing right now?
> 
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3742/#showFailuresLink
>  
> 
> 
>  Test Result (32 failures / +32)
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteScriptedUpsert
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testReadWithMetadata
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithIndexFn
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testMaxParallelRequestsPerWindow
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteRetryValidRequest
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithMaxBatchSize
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteRetry
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testReadWithQueryString
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testSizes
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithMaxBatchSizeBytes
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithDocVersion
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithAllowableErrors
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithTypeFn
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteScriptedUpsert
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testReadWithQueryValueProvider
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testSplit
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteRetryValidRequest
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithDocVersion
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testSizes
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testMaxParallelRequestsPerWindow
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testReadWithQueryString
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWritePartialUpdate
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithMaxBatchSizeBytes
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testDefaultRetryPredicate
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithIndexFn
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithRouting
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteRetry
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testReadWithMetadata
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteFullAddressing
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithMaxBatchSize
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWriteWithIsDeleteFn
> org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest.testWrite
> 
> On Wed, Jun 16, 2021 at 5:24 PM Robert Burke  > wrote:
> Very odd as those paths do resolve now, redirecting to their pkg.go.dev 
>  paths. Very odd. This feels transient, but it's not 
> clear why that would return a 404 vs some other error.
> 
> On Wed, 16 Jun 2021 at 15:39, Kyle Weaver  > wrote:
> For tasks without structured JUnit output, we have to scroll up / ctrl-f / 
> grep for more logs. In this case it looks like it was probably a server-side 
> issue. These links work for me, so I'm assuming the problem has been resolved.
> 
> 11:31:04 > Task :release:go-licenses:java:dockerRun <>
> 11:31:04 package google.golang.org/protobuf/reflect/protoreflect 
> : unrecognized import 
> path "google.golang.org/protobuf/reflect/protoreflect 
> ": reading 
> https://google.golang.org/protobuf/reflect/protoreflect?go-get=1 
> : 404 Not 
> Found
> 11:31:04 package google.golang.org/protobuf/runtime/protoimpl 
> : unrecognized import 
> path "google.golang.org/protobuf/runtime/protoimpl 
> ": reading 
> https://google.golang.org/protobuf/runtime/protoi