[jira] [Assigned] (BEAM-4087) Gradle build does not allow to overwrite versions of provided dependencies
[ https://issues.apache.org/jira/browse/BEAM-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey reassigned BEAM-4087: Assignee: (was: Michael Luckey) > Gradle build does not allow to overwrite versions of provided dependencies > -- > > Key: BEAM-4087 > URL: https://issues.apache.org/jira/browse/BEAM-4087 > Project: Beam > Issue Type: Sub-task > Components: build-system >Affects Versions: 2.5.0 >Reporter: Ismaël Mejía >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > In order to test modules with provided dependencies in maven we can execute > for example for Kafka `mvn verify -Prelease -Dkafka.clients.version=0.9.0.1 > -pl 'sdks/java/io/kafka'` However we don't have an equivalent way to do this > with gradle because the version of the dependencies are defined locally and > not in the gradle.properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7752) Java Validates Direct Runner: testTeardownCalledAfterExceptionInFinishBundleStateful flaky?
[ https://issues.apache.org/jira/browse/BEAM-7752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889577#comment-16889577 ] Michael Luckey commented on BEAM-7752: -- [~ŁukaszG] These tests seems to be flaky on direct runner. Need to look into implementation there to see, whether those assumptions made by original test creator hold. iirc direct runner was problematic with timings on that tests. > Java Validates Direct Runner: > testTeardownCalledAfterExceptionInFinishBundleStateful flaky? > --- > > Key: BEAM-7752 > URL: https://issues.apache.org/jira/browse/BEAM-7752 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Lukasz Gajowy >Priority: Major > Labels: sickbay > > See: > [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Direct/663/] > another run on the same master state was successful. As I see the job's > history, the test fails from time to time. > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (BEAM-7307) Revert test-infra to new project layout
[ https://issues.apache.org/jira/browse/BEAM-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey reassigned BEAM-7307: Assignee: Luke Cwik (was: Michael Luckey) > Revert test-infra to new project layout > --- > > Key: BEAM-7307 > URL: https://issues.apache.org/jira/browse/BEAM-7307 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Michael Luckey >Assignee: Luke Cwik >Priority: Major > > To enable release we needed to revert test-infra changes done in BEAM-4046. > After release is done, we should revert changes from > https://github.com/apache/beam/pull/8581 > Note: Also fix typo for samba validatesRunner, see > https://github.com/apache/beam/pull/8581/files#diff-9591f0d06e82e711681fd77ed287578b -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7101) Remove usages of deprecated assertion self.assertEquals in Python codebase.
[ https://issues.apache.org/jira/browse/BEAM-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-7101. -- Resolution: Fixed Fix Version/s: Not applicable > Remove usages of deprecated assertion self.assertEquals in Python codebase. > --- > > Key: BEAM-7101 > URL: https://issues.apache.org/jira/browse/BEAM-7101 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Assignee: Elwin Arens >Priority: Trivial > Labels: beginner, easyfix, newbie, starter > Fix For: Not applicable > > Time Spent: 2.5h > Remaining Estimate: 0h > > https://github.com/apache/beam/search?q=self.assertEquals_q=self.assertEquals > These can be replaced with self.assertEqual. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7401) beam_PreCommit_CommunityMetrics_Cron consistently failing
[ https://issues.apache.org/jira/browse/BEAM-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846567#comment-16846567 ] Michael Luckey commented on BEAM-7401: -- cc [~ŁukaszG] > beam_PreCommit_CommunityMetrics_Cron consistently failing > - > > Key: BEAM-7401 > URL: https://issues.apache.org/jira/browse/BEAM-7401 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Michael Luckey >Priority: Major > > Since a while, community metrics Jenkins job [1] is broken. > [1] > https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7401) beam_PreCommit_CommunityMetrics_Cron consistently failing
Michael Luckey created BEAM-7401: Summary: beam_PreCommit_CommunityMetrics_Cron consistently failing Key: BEAM-7401 URL: https://issues.apache.org/jira/browse/BEAM-7401 Project: Beam Issue Type: Bug Components: testing Reporter: Michael Luckey Since a while, community metrics Jenkins job [1] is broken. [1] https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7367) kafka-clients.jar failure in :sdks:java:container:docker
[
https://issues.apache.org/jira/browse/BEAM-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843977#comment-16843977
]
Michael Luckey commented on BEAM-7367:
--
Might be caused by merge of https://issues.apache.org/jira/browse/BEAM-7349
cc: [~aromanenko]
> kafka-clients.jar failure in :sdks:java:container:docker
>
>
> Key: BEAM-7367
> URL: https://issues.apache.org/jira/browse/BEAM-7367
> Project: Beam
> Issue Type: Test
> Components: test-failures
>Reporter: Robert Bradshaw
>Priority: Major
>
> E.g.
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/8261/consoleText
> {noformat}
> > Task :sdks:java:container:docker FAILED
> ADD failed: stat
> /var/lib/docker/tmp/docker-builder421937979/target/kafka-clients.jar: no such
> file or directory
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-7275) ParDoLifecycleTest flaky on SparkValidatesRunner
[ https://issues.apache.org/jira/browse/BEAM-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-7275. > ParDoLifecycleTest flaky on SparkValidatesRunner > > > Key: BEAM-7275 > URL: https://issues.apache.org/jira/browse/BEAM-7275 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Fix For: Not applicable > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Fix of [BEAM-7197|https://issues.apache.org/jira/browse/BEAM-7197] seems to > have introduced some > [flakyness|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/buildTimeTrend] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7275) ParDoLifecycleTest flaky on SparkValidatesRunner
[ https://issues.apache.org/jira/browse/BEAM-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-7275. -- Resolution: Fixed Fix Version/s: Not applicable > ParDoLifecycleTest flaky on SparkValidatesRunner > > > Key: BEAM-7275 > URL: https://issues.apache.org/jira/browse/BEAM-7275 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Fix For: Not applicable > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Fix of [BEAM-7197|https://issues.apache.org/jira/browse/BEAM-7197] seems to > have introduced some > [flakyness|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/buildTimeTrend] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7302) Dependencies are broken in SNAPSHOT pom files
[
https://issues.apache.org/jira/browse/BEAM-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841377#comment-16841377
]
Michael Luckey commented on BEAM-7302:
--
FYI: Opened [PR 8594|https://github.com/apache/beam/pull/8594] to fix
generatePomFileForMavenJavaPublication
> Dependencies are broken in SNAPSHOT pom files
> -
>
> Key: BEAM-7302
> URL: https://issues.apache.org/jira/browse/BEAM-7302
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.14.0
>Reporter: Ismaël Mejía
>Assignee: Michael Luckey
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The generated pom in the SNAPSHOTS repository points to dependencies that
> don't have the correct name, for example [the beam-sdks-java-core
> pom|https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.14.0-SNAPSHOT/beam-sdks-java-core-2.14.0-20190514.072148-6.pom]
> points to
> {code}
>
> beam.model
> pipeline
> 2.14.0-SNAPSHOT
> compile
>
> {code}
> but such groupId and artifactId do not exist (and have not existed in the
> past).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7302) Dependencies are broken in SNAPSHOT pom files
[
https://issues.apache.org/jira/browse/BEAM-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840452#comment-16840452
]
Michael Luckey commented on BEAM-7302:
--
After short investigating, it seems neither components.java nor
shadow.component(publication) [1] will satisfy our requirements. So I suggest
we keep that manually build pom.xml for the time being.
Will only slightly modify to fix `generatePomFileForMavenJavaPublication` task,
which is currently broken as it does not configured dependent upon project,
i.e. archivesBaseName is unset.
[1]
https://github.com/johnrengelman/shadow/blob/master/src/main/groovy/com/github/jengelman/gradle/plugins/shadow/ShadowExtension.groovy#L19-L37
> Dependencies are broken in SNAPSHOT pom files
> -
>
> Key: BEAM-7302
> URL: https://issues.apache.org/jira/browse/BEAM-7302
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.14.0
>Reporter: Ismaël Mejía
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The generated pom in the SNAPSHOTS repository points to dependencies that
> don't have the correct name, for example [the beam-sdks-java-core
> pom|https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.14.0-SNAPSHOT/beam-sdks-java-core-2.14.0-20190514.072148-6.pom]
> points to
> {code}
>
> beam.model
> pipeline
> 2.14.0-SNAPSHOT
> compile
>
> {code}
> but such groupId and artifactId do not exist (and have not existed in the
> past).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7302) Dependencies are broken in SNAPSHOT pom files
[
https://issues.apache.org/jira/browse/BEAM-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839886#comment-16839886
]
Michael Luckey commented on BEAM-7302:
--
Unfortunately, as long as we use shadow plugin, we probably need to manually
configure our deps. See
https://imperceptiblethoughts.com/shadow/publishing/#publishing-with-maven-plugin
> Dependencies are broken in SNAPSHOT pom files
> -
>
> Key: BEAM-7302
> URL: https://issues.apache.org/jira/browse/BEAM-7302
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.14.0
>Reporter: Ismaël Mejía
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The generated pom in the SNAPSHOTS repository points to dependencies that
> don't have the correct name, for example [the beam-sdks-java-core
> pom|https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.14.0-SNAPSHOT/beam-sdks-java-core-2.14.0-20190514.072148-6.pom]
> points to
> {code}
>
> beam.model
> pipeline
> 2.14.0-SNAPSHOT
> compile
>
> {code}
> but such groupId and artifactId do not exist (and have not existed in the
> past).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7307) Revert test-infra to new project layout
Michael Luckey created BEAM-7307: Summary: Revert test-infra to new project layout Key: BEAM-7307 URL: https://issues.apache.org/jira/browse/BEAM-7307 Project: Beam Issue Type: Task Components: build-system Reporter: Michael Luckey Assignee: Michael Luckey To enable release we needed to revert test-infra changes done in BEAM-4046. After release is done, we should revert changes from https://github.com/apache/beam/pull/8581 Note: Also fix typo for samba validatesRunner, see https://github.com/apache/beam/pull/8581/files#diff-9591f0d06e82e711681fd77ed287578b -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7302) Dependencies are broken in SNAPSHOT pom files
[
https://issues.apache.org/jira/browse/BEAM-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839771#comment-16839771
]
Michael Luckey commented on BEAM-7302:
--
Thx, [~kenn] for jumping in. Agree on the cause. I am actually wondering, how
this slipped through. Going to merge this to fix the current issue and will
further look whether that rewriting is still required.
> Dependencies are broken in SNAPSHOT pom files
> -
>
> Key: BEAM-7302
> URL: https://issues.apache.org/jira/browse/BEAM-7302
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.14.0
>Reporter: Ismaël Mejía
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The generated pom in the SNAPSHOTS repository points to dependencies that
> don't have the correct name, for example [the beam-sdks-java-core
> pom|https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.14.0-SNAPSHOT/beam-sdks-java-core-2.14.0-20190514.072148-6.pom]
> points to
> {code}
>
> beam.model
> pipeline
> 2.14.0-SNAPSHOT
> compile
>
> {code}
> but such groupId and artifactId do not exist (and have not existed in the
> past).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-7302) Dependencies are broken in SNAPSHOT pom files
[
https://issues.apache.org/jira/browse/BEAM-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-7302 started by Michael Luckey.
> Dependencies are broken in SNAPSHOT pom files
> -
>
> Key: BEAM-7302
> URL: https://issues.apache.org/jira/browse/BEAM-7302
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.14.0
>Reporter: Ismaël Mejía
>Assignee: Michael Luckey
>Priority: Major
>
> The generated pom in the SNAPSHOTS repository points to dependencies that
> don't have the correct name, for example [the beam-sdks-java-core
> pom|https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.14.0-SNAPSHOT/beam-sdks-java-core-2.14.0-20190514.072148-6.pom]
> points to
> {code}
>
> beam.model
> pipeline
> 2.14.0-SNAPSHOT
> compile
>
> {code}
> but such groupId and artifactId do not exist (and have not existed in the
> past).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-7302) Dependencies are broken in SNAPSHOT pom files
[
https://issues.apache.org/jira/browse/BEAM-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey reassigned BEAM-7302:
Assignee: Michael Luckey
> Dependencies are broken in SNAPSHOT pom files
> -
>
> Key: BEAM-7302
> URL: https://issues.apache.org/jira/browse/BEAM-7302
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Affects Versions: 2.14.0
>Reporter: Ismaël Mejía
>Assignee: Michael Luckey
>Priority: Major
>
> The generated pom in the SNAPSHOTS repository points to dependencies that
> don't have the correct name, for example [the beam-sdks-java-core
> pom|https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.14.0-SNAPSHOT/beam-sdks-java-core-2.14.0-20190514.072148-6.pom]
> points to
> {code}
>
> beam.model
> pipeline
> 2.14.0-SNAPSHOT
> compile
>
> {code}
> but such groupId and artifactId do not exist (and have not existed in the
> past).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-7275) ParDoLifecycleTest flaky on SparkValidatesRunner
[ https://issues.apache.org/jira/browse/BEAM-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7275 started by Michael Luckey. > ParDoLifecycleTest flaky on SparkValidatesRunner > > > Key: BEAM-7275 > URL: https://issues.apache.org/jira/browse/BEAM-7275 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Fix of [BEAM-7197|https://issues.apache.org/jira/browse/BEAM-7197] seems to > have introduced some > [flakyness|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/buildTimeTrend] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7275) ParDoLifecycleTest flaky on SparkValidatesRunner
[ https://issues.apache.org/jira/browse/BEAM-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey updated BEAM-7275: - Status: Open (was: Triage Needed) > ParDoLifecycleTest flaky on SparkValidatesRunner > > > Key: BEAM-7275 > URL: https://issues.apache.org/jira/browse/BEAM-7275 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Fix of [BEAM-7197|https://issues.apache.org/jira/browse/BEAM-7197] seems to > have introduced some > [flakyness|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/buildTimeTrend] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7287) beam_PostCommit_Java11_ValidatesRunner_Direct broken
Michael Luckey created BEAM-7287: Summary: beam_PostCommit_Java11_ValidatesRunner_Direct broken Key: BEAM-7287 URL: https://issues.apache.org/jira/browse/BEAM-7287 Project: Beam Issue Type: Bug Components: test-failures Reporter: Michael Luckey Assignee: Michal Walenia Since we switched to the new Jenkins agents, beam_PostCommit_Java11_ValidatesRunner_Direct is consistently failing [1]. https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct/buildTimeTrend cc [~yifanzou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7276) Flink PVR Streaming tests failing
[
https://issues.apache.org/jira/browse/BEAM-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey updated BEAM-7276:
-
Description:
After adding [1]
{noformat}
Changes
Revert "Merge pull request #8228: [BEAM-7008] adding UTF8 String coder (commit:
cc8bc6c) (detail / githubweb)
Revert "Merge pull request #8545 from ihji/BEAM-7260" (commit: 466038d) (detail
/ githubweb)
{noformat}
Flinks streaming tests seem to be failing consistently [2]
[1] https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/1612/
[2] https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/
was:
After adding
{noformat}
Changes
Revert "Merge pull request #8228: [BEAM-7008] adding UTF8 String coder (commit:
cc8bc6c) (detail / githubweb)
Revert "Merge pull request #8545 from ihji/BEAM-7260" (commit: 466038d) (detail
/ githubweb)
{noformat}
Flinks streaming tests seem to be failing consistently [1]
[1] https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/
> Flink PVR Streaming tests failing
> -
>
> Key: BEAM-7276
> URL: https://issues.apache.org/jira/browse/BEAM-7276
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Michael Luckey
>Priority: Major
>
> After adding [1]
> {noformat}
> Changes
> Revert "Merge pull request #8228: [BEAM-7008] adding UTF8 String coder
> (commit: cc8bc6c) (detail / githubweb)
> Revert "Merge pull request #8545 from ihji/BEAM-7260" (commit: 466038d)
> (detail / githubweb)
> {noformat}
> Flinks streaming tests seem to be failing consistently [2]
> [1]
> https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/1612/
> [2] https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7276) Flink PVR Streaming tests failing
[
https://issues.apache.org/jira/browse/BEAM-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838390#comment-16838390
]
Michael Luckey commented on BEAM-7276:
--
cc: [~heejong]
cc: [~mxm]
> Flink PVR Streaming tests failing
> -
>
> Key: BEAM-7276
> URL: https://issues.apache.org/jira/browse/BEAM-7276
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
>Reporter: Michael Luckey
>Priority: Major
>
> After adding
> {noformat}
> Changes
> Revert "Merge pull request #8228: [BEAM-7008] adding UTF8 String coder
> (commit: cc8bc6c) (detail / githubweb)
> Revert "Merge pull request #8545 from ihji/BEAM-7260" (commit: 466038d)
> (detail / githubweb)
> {noformat}
> Flinks streaming tests seem to be failing consistently [1]
> [1] https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7276) Flink PVR Streaming tests failing
Michael Luckey created BEAM-7276:
Summary: Flink PVR Streaming tests failing
Key: BEAM-7276
URL: https://issues.apache.org/jira/browse/BEAM-7276
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Michael Luckey
After adding
{noformat}
Changes
Revert "Merge pull request #8228: [BEAM-7008] adding UTF8 String coder (commit:
cc8bc6c) (detail / githubweb)
Revert "Merge pull request #8545 from ihji/BEAM-7260" (commit: 466038d) (detail
/ githubweb)
{noformat}
Flinks streaming tests seem to be failing consistently [1]
[1] https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-7275) ParDoLifecycleTest flaky on SparkValidatesRunner
Michael Luckey created BEAM-7275: Summary: ParDoLifecycleTest flaky on SparkValidatesRunner Key: BEAM-7275 URL: https://issues.apache.org/jira/browse/BEAM-7275 Project: Beam Issue Type: Bug Components: test-failures Reporter: Michael Luckey Assignee: Michael Luckey Fix of [BEAM-7197|https://issues.apache.org/jira/browse/BEAM-7197] seems to have introduced some [flakyness|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/buildTimeTrend] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7197) ParDoLifecycleTest: exeption throwing tests broken
[
https://issues.apache.org/jira/browse/BEAM-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey updated BEAM-7197:
-
Status: Open (was: Triage Needed)
> ParDoLifecycleTest: exeption throwing tests broken
> --
>
> Key: BEAM-7197
> URL: https://issues.apache.org/jira/browse/BEAM-7197
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> ParDoLifecycleTest implements tests to assert that DoFn are tore down after
> another lifecycle method throw an exception.
>
> The implementation uses a static AtomicBoolean for assertions [1].
> Unfortunately, this is never reset which results in that boolean being true
> after the first test which happens to correctly call teardown on exception.
> Failures for tests executed after are essentially hidden.
> This can be seen e.g. by
> {code:bash}
> ./gradlew -p runners/spark/ validatesRunnerBatch --tests
> org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
>
> {code}
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java#L407-L412
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-7197) ParDoLifecycleTest: exeption throwing tests broken
[
https://issues.apache.org/jira/browse/BEAM-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-7197 started by Michael Luckey.
> ParDoLifecycleTest: exeption throwing tests broken
> --
>
> Key: BEAM-7197
> URL: https://issues.apache.org/jira/browse/BEAM-7197
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> ParDoLifecycleTest implements tests to assert that DoFn are tore down after
> another lifecycle method throw an exception.
>
> The implementation uses a static AtomicBoolean for assertions [1].
> Unfortunately, this is never reset which results in that boolean being true
> after the first test which happens to correctly call teardown on exception.
> Failures for tests executed after are essentially hidden.
> This can be seen e.g. by
> {code:bash}
> ./gradlew -p runners/spark/ validatesRunnerBatch --tests
> org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
>
> {code}
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java#L407-L412
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-7229) ParDoLifecycleTest: remove duplicated test methods
[ https://issues.apache.org/jira/browse/BEAM-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-7229. > ParDoLifecycleTest: remove duplicated test methods > -- > > Key: BEAM-7229 > URL: https://issues.apache.org/jira/browse/BEAM-7229 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > ParDoLifecycleTest contains following tests: > testTeardownCalledAfterExceptionInStartBundle > testTeardownCalledAfterExceptionInProcessElement > testTeardownCalledAfterExceptionInFinishBundle > testWithContextTeardownCalledAfterExceptionInSetup > testWithContextTeardownCalledAfterExceptionInStartBundle > testWithContextTeardownCalledAfterExceptionInProcessElement > testWithContextTeardownCalledAfterExceptionInFinishBundle > The actual implementation of testTeardownCalledAfterExceptionIn* and the > corresponding counterpart testWithContextTeardownCalledAfterExceptionIn* is > identical. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7229) ParDoLifecycleTest: remove duplicated test methods
[ https://issues.apache.org/jira/browse/BEAM-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-7229. -- Resolution: Fixed Fix Version/s: Not applicable > ParDoLifecycleTest: remove duplicated test methods > -- > > Key: BEAM-7229 > URL: https://issues.apache.org/jira/browse/BEAM-7229 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > ParDoLifecycleTest contains following tests: > testTeardownCalledAfterExceptionInStartBundle > testTeardownCalledAfterExceptionInProcessElement > testTeardownCalledAfterExceptionInFinishBundle > testWithContextTeardownCalledAfterExceptionInSetup > testWithContextTeardownCalledAfterExceptionInStartBundle > testWithContextTeardownCalledAfterExceptionInProcessElement > testWithContextTeardownCalledAfterExceptionInFinishBundle > The actual implementation of testTeardownCalledAfterExceptionIn* and the > corresponding counterpart testWithContextTeardownCalledAfterExceptionIn* is > identical. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7229) ParDoLifecycleTest: remove duplicated test methods
[ https://issues.apache.org/jira/browse/BEAM-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey updated BEAM-7229: - Status: Open (was: Triage Needed) > ParDoLifecycleTest: remove duplicated test methods > -- > > Key: BEAM-7229 > URL: https://issues.apache.org/jira/browse/BEAM-7229 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > > ParDoLifecycleTest contains following tests: > testTeardownCalledAfterExceptionInStartBundle > testTeardownCalledAfterExceptionInProcessElement > testTeardownCalledAfterExceptionInFinishBundle > testWithContextTeardownCalledAfterExceptionInSetup > testWithContextTeardownCalledAfterExceptionInStartBundle > testWithContextTeardownCalledAfterExceptionInProcessElement > testWithContextTeardownCalledAfterExceptionInFinishBundle > The actual implementation of testTeardownCalledAfterExceptionIn* and the > corresponding counterpart testWithContextTeardownCalledAfterExceptionIn* is > identical. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-7229) ParDoLifecycleTest: remove duplicated test methods
[ https://issues.apache.org/jira/browse/BEAM-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7229 started by Michael Luckey. > ParDoLifecycleTest: remove duplicated test methods > -- > > Key: BEAM-7229 > URL: https://issues.apache.org/jira/browse/BEAM-7229 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > > ParDoLifecycleTest contains following tests: > testTeardownCalledAfterExceptionInStartBundle > testTeardownCalledAfterExceptionInProcessElement > testTeardownCalledAfterExceptionInFinishBundle > testWithContextTeardownCalledAfterExceptionInSetup > testWithContextTeardownCalledAfterExceptionInStartBundle > testWithContextTeardownCalledAfterExceptionInProcessElement > testWithContextTeardownCalledAfterExceptionInFinishBundle > The actual implementation of testTeardownCalledAfterExceptionIn* and the > corresponding counterpart testWithContextTeardownCalledAfterExceptionIn* is > identical. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7229) ParDoLifecycleTest: remove duplicated test methods
Michael Luckey created BEAM-7229: Summary: ParDoLifecycleTest: remove duplicated test methods Key: BEAM-7229 URL: https://issues.apache.org/jira/browse/BEAM-7229 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Michael Luckey Assignee: Michael Luckey ParDoLifecycleTest contains following tests: testTeardownCalledAfterExceptionInStartBundle testTeardownCalledAfterExceptionInProcessElement testTeardownCalledAfterExceptionInFinishBundle testWithContextTeardownCalledAfterExceptionInSetup testWithContextTeardownCalledAfterExceptionInStartBundle testWithContextTeardownCalledAfterExceptionInProcessElement testWithContextTeardownCalledAfterExceptionInFinishBundle The actual implementation of testTeardownCalledAfterExceptionIn* and the corresponding counterpart testWithContextTeardownCalledAfterExceptionIn* is identical. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7193) ParDoLifecycleTest: duplicated inner class
[ https://issues.apache.org/jira/browse/BEAM-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-7193. -- Resolution: Fixed Fix Version/s: Not applicable > ParDoLifecycleTest: duplicated inner class > -- > > Key: BEAM-7193 > URL: https://issues.apache.org/jira/browse/BEAM-7193 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > > After migration from OldDoFn and later refactoring, ParDoLifecycleTest has 2 > nearly identical DoFn implementations. Code should be consolidated to use > only one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-7193) ParDoLifecycleTest: duplicated inner class
[ https://issues.apache.org/jira/browse/BEAM-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-7193. > ParDoLifecycleTest: duplicated inner class > -- > > Key: BEAM-7193 > URL: https://issues.apache.org/jira/browse/BEAM-7193 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > > After migration from OldDoFn and later refactoring, ParDoLifecycleTest has 2 > nearly identical DoFn implementations. Code should be consolidated to use > only one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7197) ParDoLifecycleTest: exeption throwing tests broken
Michael Luckey created BEAM-7197:
Summary: ParDoLifecycleTest: exeption throwing tests broken
Key: BEAM-7197
URL: https://issues.apache.org/jira/browse/BEAM-7197
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Michael Luckey
Assignee: Michael Luckey
ParDoLifecycleTest implements tests to assert that DoFn are tore down after
another lifecycle method throw an exception.
The implementation uses a static AtomicBoolean for assertions [1].
Unfortunately, this is never reset which results in that boolean being true
after the first test which happens to correctly call teardown on exception.
Failures for tests executed after are essentially hidden.
This can be seen e.g. by
{code:bash}
./gradlew -p runners/spark/ validatesRunnerBatch --tests
org.apache.beam.sdk.transforms.ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
{code}
[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoLifecycleTest.java#L407-L412
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830746#comment-16830746
]
Michael Luckey commented on BEAM-4046:
--
Well, I do not have the feeling these changes were approved. Unless I would
insist on some lazy consensus. I will bring that back to the mailing list and
ask for objections to merge.
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 20h 10m
> Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-7193) ParDoLifecycleTest: duplicated inner class
[ https://issues.apache.org/jira/browse/BEAM-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7193 started by Michael Luckey. > ParDoLifecycleTest: duplicated inner class > -- > > Key: BEAM-7193 > URL: https://issues.apache.org/jira/browse/BEAM-7193 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > After migration from OldDoFn and later refactoring, ParDoLifecycleTest has 2 > nearly identical DoFn implementations. Code should be consolidated to use > only one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7193) ParDoLifecycleTest: duplicated inner class
[ https://issues.apache.org/jira/browse/BEAM-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey updated BEAM-7193: - Status: Open (was: Triage Needed) > ParDoLifecycleTest: duplicated inner class > -- > > Key: BEAM-7193 > URL: https://issues.apache.org/jira/browse/BEAM-7193 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > After migration from OldDoFn and later refactoring, ParDoLifecycleTest has 2 > nearly identical DoFn implementations. Code should be consolidated to use > only one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-7193) ParDoLifecycleTest: duplicated inner class
[ https://issues.apache.org/jira/browse/BEAM-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey updated BEAM-7193: - Priority: Minor (was: Major) > ParDoLifecycleTest: duplicated inner class > -- > > Key: BEAM-7193 > URL: https://issues.apache.org/jira/browse/BEAM-7193 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Minor > > After migration from OldDoFn and later refactoring, ParDoLifecycleTest has 2 > nearly identical DoFn implementations. Code should be consolidated to use > only one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-7193) ParDoLifecycleTest: duplicated inner class
Michael Luckey created BEAM-7193: Summary: ParDoLifecycleTest: duplicated inner class Key: BEAM-7193 URL: https://issues.apache.org/jira/browse/BEAM-7193 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Michael Luckey Assignee: Michael Luckey After migration from OldDoFn and later refactoring, ParDoLifecycleTest has 2 nearly identical DoFn implementations. Code should be consolidated to use only one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-6859) Spark-runner: DoFn tearDown function will not be invoked if there is no data in the batch
[
https://issues.apache.org/jira/browse/BEAM-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey reassigned BEAM-6859:
Assignee: Michael Luckey
> Spark-runner: DoFn tearDown function will not be invoked if there is no data
> in the batch
> -
>
> Key: BEAM-6859
> URL: https://issues.apache.org/jira/browse/BEAM-6859
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
>Affects Versions: 2.8.0, 2.9.0, 2.10.0, 2.11.0
>Reporter: Ron Cai
>Assignee: Michael Luckey
>Priority: Critical
>
> In the implementation of
> [MultiDoFnFunction|https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java],
>
> {code:java}
> @Override
> public Iterator, WindowedValue>>
> call(Iterator> iter)
> throws Exception {
> if (!wasSetupCalled) {
> DoFnInvokers.invokerFor(doFn).invokeSetup();
> wasSetupCalled = true;
> }
> ...
>return new SparkProcessContext<>(
> doFn,
> doFnRunnerWithMetrics,
> outputManager,
> stateful ? new TimerDataIterator(timerInternals) :
> Collections.emptyIterator())
> .processPartition(iter)
> .iterator();
> }{code}
> It will call setup function of a DoFn every batch in spark streaming. And
> the tearDown function of DoFn will invoked by
> [SparkProcessContext|https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java]
> instance. But in the implementation of
> SparkProcessContext.processParition(), if the partition is empty, it will
> return an empty ArrayList instance directly. If there is no data in the
> batch, it means the tearDown function of DoFn will not be invoked for it is
> invoked in the ProcCtxtIterator instance which created only when there are
> data (parition.hasNext() == true).
> {code:java}
> Iterable processPartition(Iterator>
> partition) throws Exception {
>
> // skip if partition is empty.
> if (!partition.hasNext()) {
> return new ArrayList<>();
> }
>
> // process the partition; finishBundle() is called from within the
> output iterator.
> return this.getOutputIterable(partition, doFnRunner);
> }
> {code}
> If you want to reproduce the issue, just build a pipeline to read from
> KafkaIO.read and write by KafkaIO.write() to kafka and run as a spark
> streaming application, don't send any data to the kafka topic. Thread count
> of kafka producer will keep increasing and OOO at the end.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-6859) Spark-runner: DoFn tearDown function will not be invoked if there is no data in the batch
[
https://issues.apache.org/jira/browse/BEAM-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-6859 started by Michael Luckey.
> Spark-runner: DoFn tearDown function will not be invoked if there is no data
> in the batch
> -
>
> Key: BEAM-6859
> URL: https://issues.apache.org/jira/browse/BEAM-6859
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
>Affects Versions: 2.8.0, 2.9.0, 2.10.0, 2.11.0
>Reporter: Ron Cai
>Assignee: Michael Luckey
>Priority: Critical
>
> In the implementation of
> [MultiDoFnFunction|https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java],
>
> {code:java}
> @Override
> public Iterator, WindowedValue>>
> call(Iterator> iter)
> throws Exception {
> if (!wasSetupCalled) {
> DoFnInvokers.invokerFor(doFn).invokeSetup();
> wasSetupCalled = true;
> }
> ...
>return new SparkProcessContext<>(
> doFn,
> doFnRunnerWithMetrics,
> outputManager,
> stateful ? new TimerDataIterator(timerInternals) :
> Collections.emptyIterator())
> .processPartition(iter)
> .iterator();
> }{code}
> It will call setup function of a DoFn every batch in spark streaming. And
> the tearDown function of DoFn will invoked by
> [SparkProcessContext|https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java]
> instance. But in the implementation of
> SparkProcessContext.processParition(), if the partition is empty, it will
> return an empty ArrayList instance directly. If there is no data in the
> batch, it means the tearDown function of DoFn will not be invoked for it is
> invoked in the ProcCtxtIterator instance which created only when there are
> data (parition.hasNext() == true).
> {code:java}
> Iterable processPartition(Iterator>
> partition) throws Exception {
>
> // skip if partition is empty.
> if (!partition.hasNext()) {
> return new ArrayList<>();
> }
>
> // process the partition; finishBundle() is called from within the
> output iterator.
> return this.getOutputIterable(partition, doFnRunner);
> }
> {code}
> If you want to reproduce the issue, just build a pipeline to read from
> KafkaIO.read and write by KafkaIO.write() to kafka and run as a spark
> streaming application, don't send any data to the kafka topic. Thread count
> of kafka producer will keep increasing and OOO at the end.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829804#comment-16829804
]
Michael Luckey commented on BEAM-4046:
--
[~kenn] Ah, sorry for not clearly resolving the issue with duplicate project
names.
Yes, this is resolved by using gradles default project group/name, i.e. by
incorporating the projects path into group id we do not have any duplicates
anymore. See relevant changes in BeamModulePlugin [1]
[1]
https://github.com/apache/beam/pull/8194/commits/d8df4c87fc5d329bd694269fef509fa2e28da79a#diff-23833058cbf2c1172b90e7764032aa59
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 20h
> Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829263#comment-16829263
]
Michael Luckey edited comment on BEAM-4046 at 4/29/19 1:24 PM:
---
[~kenn], [~lcwik]
After keeping that open for a while, I would like to revive discussion on how
to proceed with this ticket.
Tests seem to be successful. So main point of discussion would be
- do we want to apply this change or just keep current state and drop this
ticket
- if we want to proceed, please take a look at the 'backwards compatibility'
hack introduced [1]. After looking into the requirements, it seemed easiest to
me to just rewrite gradle cli params. So I moved gradlew shell script to
gradlew_orig and just do some args rewriting. This seems to work (as Jenkins is
still able to execute all tests without seeding of changed jobs). We probably
should move to posix shell here and maybe put some love into that
implementation. Also we might need to implement similar for windows (not sure
though, whether that's really used)
So if we want to continue, I would do some final testing and would like to
merge after cut of next release branch. Currently it is difficult to keep pace
with the changes done to master.
[1]
https://github.com/apache/beam/pull/8194/commits/c8685a8ca7a4a0366f5f0aac63ea5e1e7d8e5bb8
was (Author: michel):
[~kenn], [~lcwik]
After keeping that open for a while, I would like to revive discussion on how
to proceed with this ticket.
Tests seem to be successful. So main point of discussion would be
- do we want to apply this change or just keep current state and drop this
ticket
- if we want to proceed, please take a look at the 'backwards compatibility'
hack introduced. After looking into the requirements, it seemed easiest to me
to just rewrite gradle cli params. So I moved gradlew shell script to
gradlew_orig and just do some args rewriting. This seems to work (as Jenkins is
still able to execute all tests without seeding of changed jobs). We probably
should move to posix shell here and maybe put some love into that
implementation. Also we might need to implement similar for windows (not sure
though, whether that's really used)
So if we want to continue, I would do some final testing and would like to
merge after cut of next release branch. Currently it is difficult to keep pace
with the changes done to master.
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 20h
> Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829263#comment-16829263
]
Michael Luckey commented on BEAM-4046:
--
[~kenn], [~lcwik]
After keeping that open for a while, I would like to revive discussion on how
to proceed with this ticket.
Tests seem to be successful. So main point of discussion would be
- do we want to apply this change or just keep current state and drop this
ticket
- if we want to proceed, please take a look at the 'backwards compatibility'
hack introduced. After looking into the requirements, it seemed easiest to me
to just rewrite gradle cli params. So I moved gradlew shell script to
gradlew_orig and just do some args rewriting. This seems to work (as Jenkins is
still able to execute all tests without seeding of changed jobs). We probably
should move to posix shell here and maybe put some love into that
implementation. Also we might need to implement similar for windows (not sure
though, whether that's really used)
So if we want to continue, I would do some final testing and would like to
merge after cut of next release branch. Currently it is difficult to keep pace
with the changes done to master.
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 20h
> Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7003) Test KafkaIO against different versions of Kafka
[
https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815508#comment-16815508
]
Michael Luckey commented on BEAM-7003:
--
Oh no, forgot about that workaround in BeamModulePlugin:
{noformat}
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 918df02c22..92962b91f2 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1145,7 +1145,7 @@ class BeamModulePlugin implements Plugin {
project.configurations.all { config ->
// The "errorprone" configuration controls the classpath used by
errorprone static analysis, which
// has different dependencies than our project.
-if (config.getName() != "errorprone") {
+if (config.getName() != "errorprone" &&
!config.getName().startsWith('variant')) {
config.resolutionStrategy {
force project.library.java.values()
}
{noformat}
That of course implies its not so easy to test as stated :(
> Test KafkaIO against different versions of Kafka
>
>
> Key: BEAM-7003
> URL: https://issues.apache.org/jira/browse/BEAM-7003
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Ismaël Mejía
>Priority: Minor
>
> KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our
> current test infrastructure only validates this against the version 1.0.0. We
> should find a mecanism to parametrize KafkaIO tests with different versions
> of Kafka to ensure that multi-version support works as expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-7003) Test KafkaIO against different versions of Kafka
[
https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815436#comment-16815436
]
Michael Luckey edited comment on BEAM-7003 at 4/11/19 2:02 PM:
---
Hi [~iemejia],
after creating a POC implementation [1], I wanted to discuss this first.
Implementation of BEAM-4087 enables adhoc replacement of the used Kafka client
version. With this we change version for the complete build. So this could be
considered a test, whether we would be able to compile with different version.
This implementation replaces the dependency during runtime. So this seems to
reflect an user exchanging her Kafka client jar for some other version. But as
Kafka client promises compatibility anyway - apart from versions lower some
0.10.2?), what do we intend to test here? Or, asked diffently, why would an
user exchange the Kafka client jar anyway?
One obvious problem seems to be, that KafkaIOTest relies on mocks provided by
the client jar. So I wonder whether we will mostly fail on some incompatible
mock change, which is probably not, what is intended. Especially executing
these tests below fail on the test code referencing non existing api in
versions < 0.11. Which not necessarily has to imply KafkaIO would not work with
lower versions (although KafkaIO is referencing non existing classes, so will
also not work with those versions).
On the other hand, there might be nevertheless a use case in providing such
functionality. E.g. I could imagine it being valuable to check HDFS filesystem
with different Hadoop versions.
Coming back to KafkaIO, I tend to believe, that the thing we require is some
test against a 'real' cluster. I.e. testing KafkaIO against some docker
containers running those versions we want to support?
Thoughts? You might just paste below snippet to build gradle and try yourself
with e.g.
{noformat}
./gradlew -p sdks/java/io/kafka/ clean check --continue --no-build-cache
{noformat}
[1]
{code}
// to be appended to sdks/java/io/kafka/build.gradle
ext.testVersions = [
"0.9.0.1",
"0.10.0.1",
"0.11.0.2",
"1.1.0",
"2.0.0",
"2.1.0",
"2.2.0",
]
configurations {
variant1_0_0
}
dependencies {
variant1_0_0 library.java.kafka_clients
}
def previous = ''
testVersions.each {
configurations.create("variant${it.replaceAll('\\.', '_')}")
dependencies.add "variant${it.replaceAll('\\.', '_')}",
"org.apache.kafka:kafka-clients:$it"
task "test$it"(type: Test) { task ->
description = "Runs variant tests for $it"
group = 'verification'
classpath = classpath - configurations.variant1_0_0 +
configurations."variant${it.replaceAll('\\.', '_')}"
// do we care about order here? they arent executed in parallel. Even if,
does it matter here?
shouldRunAfter "test$previous"
}
previous = it
check.dependsOn("test$it")
}
{code}
was (Author: michel):
Hi [~iemejia],
after creating a POC implementation [1], I wanted to discuss this first.
Implementation of BEAM-4087 enables adhoc replacement of the used Kafka client
version. With this we change version for the complete build. So this could be
considered a test, whether we would be able to compile with different version.
This implementation replaces the dependency during runtime. So this seems to
reflect an user exchanging her Kafka client jar for some other version. But as
Kafka client promises compatibility anyway - apart from versions lower some
0.10.2?), what do we intend to test here? Or, asked diffently, why would an
user exchange the Kafka client jar anyway?
One obvious problem seems to be, that KafkaIOTest relies on mocks provided by
the client jar. So I wonder whether we will mostly fail on some incompatible
mock change, which is probably not, what is intended. Especially executing
these tests below fail on the test code referencing non existing api in
versions < 0.11. Which not necessarily has to imply KafkaIO would not work with
lower versions (also KafkaIO is referencing non existing classes, so will also
not work with those versions).
On the other hand, there might be nevertheless a use case in providing such
functionality. E.g. I could imagine it being valuable to check HDFS filesystem
with different Hadoop versions.
Coming back to KafkaIO, I tend to believe, that the thing we require is some
test against a 'real' cluster. I.e. testing KafkaIO against some docker
containers running those versions we want to support?
Thoughts? You might just paste below snippet to build gradle and try yourself
with e.g.
{noformat}
./gradlew -p sdks/java/io/kafka/ clean check --continue --no-build-cache
{noformat}
[1]
{code}
// to be appended to sdks/java/io/kafka/build.gradle
ext.testVersions = [
"0.9.0.1",
"0.10.0.1",
"0.11.0.2",
"1.1.0",
"2.0.0",
"2.1.0",
"2.2.0",
]
configurations {
variant1_0_0
}
dependencies {
variant1_0_0
[jira] [Commented] (BEAM-7003) Test KafkaIO against different versions of Kafka
[
https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815436#comment-16815436
]
Michael Luckey commented on BEAM-7003:
--
Hi [~iemejia],
after creating a POC implementation [1], I wanted to discuss this first.
Implementation of BEAM-4087 enables adhoc replacement of the used Kafka client
version. With this we change version for the complete build. So this could be
considered a test, whether we would be able to compile with different version.
This implementation replaces the dependency during runtime. So this seems to
reflect an user exchanging her Kafka client jar for some other version. But as
Kafka client promises compatibility anyway - apart from versions lower some
0.10.2?), what do we intend to test here? Or, asked diffently, why would an
user exchange the Kafka client jar anyway?
One obvious problem seems to be, that KafkaIOTest relies on mocks provided by
the client jar. So I wonder whether we will mostly fail on some incompatible
mock change, which is probably not, what is intended. Especially executing
these tests below fail on the test code referencing non existing api in
versions < 0.11. Which not necessarily has to imply KafkaIO would not work with
lower versions (also KafkaIO is referencing non existing classes, so will also
not work with those versions).
On the other hand, there might be nevertheless a use case in providing such
functionality. E.g. I could imagine it being valuable to check HDFS filesystem
with different Hadoop versions.
Coming back to KafkaIO, I tend to believe, that the thing we require is some
test against a 'real' cluster. I.e. testing KafkaIO against some docker
containers running those versions we want to support?
Thoughts? You might just paste below snippet to build gradle and try yourself
with e.g.
{noformat}
./gradlew -p sdks/java/io/kafka/ clean check --continue --no-build-cache
{noformat}
[1]
{code}
// to be appended to sdks/java/io/kafka/build.gradle
ext.testVersions = [
"0.9.0.1",
"0.10.0.1",
"0.11.0.2",
"1.1.0",
"2.0.0",
"2.1.0",
"2.2.0",
]
configurations {
variant1_0_0
}
dependencies {
variant1_0_0 library.java.kafka_clients
}
def previous = ''
testVersions.each {
configurations.create("variant${it.replaceAll('\\.', '_')}")
dependencies.add "variant${it.replaceAll('\\.', '_')}",
"org.apache.kafka:kafka-clients:$it"
task "test$it"(type: Test) { task ->
description = "Runs variant tests for $it"
group = 'verification'
classpath = classpath - configurations.variant1_0_0 +
configurations."variant${it.replaceAll('\\.', '_')}"
// do we care about order here? they arent executed in parallel. Even if,
does it matter here?
shouldRunAfter "test$previous"
}
previous = it
check.dependsOn("test$it")
}
{code}
> Test KafkaIO against different versions of Kafka
>
>
> Key: BEAM-7003
> URL: https://issues.apache.org/jira/browse/BEAM-7003
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Ismaël Mejía
>Priority: Minor
>
> KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our
> current test infrastructure only validates this against the version 1.0.0. We
> should find a mecanism to parametrize KafkaIO tests with different versions
> of Kafka to ensure that multi-version support works as expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-7042) beam-sdks-java-core leaks antlr4 dependency
[ https://issues.apache.org/jira/browse/BEAM-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814923#comment-16814923 ] Michael Luckey commented on BEAM-7042: -- Yah... somehow that answer was expected :( So best we can do is properly retest to find the culprit. > beam-sdks-java-core leaks antlr4 dependency > --- > > Key: BEAM-7042 > URL: https://issues.apache.org/jira/browse/BEAM-7042 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Michael Luckey >Priority: Major > Fix For: 2.12.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > beam-sdks-java-core brings antlr4 and its transitive deps which is quite > unlikely and has a huge probability to conflict (antlt, jsonp in an outdated > version at least for the one breaking my apps). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7042) [regression] org.apache.beam:beam-sdks-java-core:jar:2.12.0 dependencies are way fatter and conflicting than before
[ https://issues.apache.org/jira/browse/BEAM-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814362#comment-16814362 ] Michael Luckey commented on BEAM-7042: -- Opened a PR. If it breaks, we might know, why it was added. https://github.com/apache/beam/pull/8268 > [regression] org.apache.beam:beam-sdks-java-core:jar:2.12.0 dependencies are > way fatter and conflicting than before > --- > > Key: BEAM-7042 > URL: https://issues.apache.org/jira/browse/BEAM-7042 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Reuven Lax >Priority: Major > Fix For: 2.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Hi guys, > seems current sdk brings antlr and its transitive deps which is quite > unlikely and has a huge probability to conflict (antlt, jsonp in an outdated > version at least for the one breaking my apps) > can it be cleaned up for the 2.12 to avoid to break application please? > Thanks, > Romain -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-7042) [regression] org.apache.beam:beam-sdks-java-core:jar:2.12.0 dependencies are way fatter and conflicting than before
[ https://issues.apache.org/jira/browse/BEAM-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814354#comment-16814354 ] Michael Luckey commented on BEAM-7042: -- I think this could be caused by [1] [~reuvenlax] do you remember, why this was added? Should we only require runtime? [1] https://github.com/apache/beam/pull/7635/files#diff-9390c20635aed5f42f83b97506a87333 > [regression] org.apache.beam:beam-sdks-java-core:jar:2.12.0 dependencies are > way fatter and conflicting than before > --- > > Key: BEAM-7042 > URL: https://issues.apache.org/jira/browse/BEAM-7042 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Reuven Lax >Priority: Major > Fix For: 2.13.0 > > > Hi guys, > seems current sdk brings antlr and its transitive deps which is quite > unlikely and has a huge probability to conflict (antlt, jsonp in an outdated > version at least for the one breaking my apps) > can it be cleaned up for the 2.12 to avoid to break application please? > Thanks, > Romain -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-4087) Gradle build does not allow to overwrite versions of provided dependencies
[ https://issues.apache.org/jira/browse/BEAM-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-4087 started by Michael Luckey. > Gradle build does not allow to overwrite versions of provided dependencies > -- > > Key: BEAM-4087 > URL: https://issues.apache.org/jira/browse/BEAM-4087 > Project: Beam > Issue Type: Sub-task > Components: build-system >Affects Versions: 2.5.0 >Reporter: Ismaël Mejía >Assignee: Michael Luckey >Priority: Major > > In order to test modules with provided dependencies in maven we can execute > for example for Kafka `mvn verify -Prelease -Dkafka.clients.version=0.9.0.1 > -pl 'sdks/java/io/kafka'` However we don't have an equivalent way to do this > with gradle because the version of the dependencies are defined locally and > not in the gradle.properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4087) Gradle build does not allow to overwrite versions of provided dependencies
[ https://issues.apache.org/jira/browse/BEAM-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey reassigned BEAM-4087: Assignee: Michael Luckey > Gradle build does not allow to overwrite versions of provided dependencies > -- > > Key: BEAM-4087 > URL: https://issues.apache.org/jira/browse/BEAM-4087 > Project: Beam > Issue Type: Sub-task > Components: build-system >Affects Versions: 2.5.0 >Reporter: Ismaël Mejía >Assignee: Michael Luckey >Priority: Major > > In order to test modules with provided dependencies in maven we can execute > for example for Kafka `mvn verify -Prelease -Dkafka.clients.version=0.9.0.1 > -pl 'sdks/java/io/kafka'` However we don't have an equivalent way to do this > with gradle because the version of the dependencies are defined locally and > not in the gradle.properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7003) Test KafkaIO against different versions of Kafka
[ https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey reassigned BEAM-7003: Assignee: (was: Michael Luckey) > Test KafkaIO against different versions of Kafka > > > Key: BEAM-7003 > URL: https://issues.apache.org/jira/browse/BEAM-7003 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Priority: Minor > > KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our > current test infrastructure only validates this against the version 1.0.0. We > should find a mecanism to parametrize KafkaIO tests with different versions > of Kafka to ensure that multi-version support works as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7003) Test KafkaIO against different versions of Kafka
[ https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey reassigned BEAM-7003: Assignee: Michael Luckey > Test KafkaIO against different versions of Kafka > > > Key: BEAM-7003 > URL: https://issues.apache.org/jira/browse/BEAM-7003 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Michael Luckey >Priority: Minor > > KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our > current test infrastructure only validates this against the version 1.0.0. We > should find a mecanism to parametrize KafkaIO tests with different versions > of Kafka to ensure that multi-version support works as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-7003) Test KafkaIO against different versions of Kafka
[ https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7003 started by Michael Luckey. > Test KafkaIO against different versions of Kafka > > > Key: BEAM-7003 > URL: https://issues.apache.org/jira/browse/BEAM-7003 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Michael Luckey >Priority: Minor > > KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our > current test infrastructure only validates this against the version 1.0.0. We > should find a mecanism to parametrize KafkaIO tests with different versions > of Kafka to ensure that multi-version support works as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work stopped] (BEAM-7003) Test KafkaIO against different versions of Kafka
[ https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7003 stopped by Michael Luckey. > Test KafkaIO against different versions of Kafka > > > Key: BEAM-7003 > URL: https://issues.apache.org/jira/browse/BEAM-7003 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ismaël Mejía >Assignee: Michael Luckey >Priority: Minor > > KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our > current test infrastructure only validates this against the version 1.0.0. We > should find a mecanism to parametrize KafkaIO tests with different versions > of Kafka to ensure that multi-version support works as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-7016) DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never initialized
[ https://issues.apache.org/jira/browse/BEAM-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-7016. > DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never > initialized > - > > Key: BEAM-7016 > URL: https://issues.apache.org/jira/browse/BEAM-7016 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Luke Cwik >Assignee: Michael Luckey >Priority: Minor > Labels: starter > Fix For: 2.13.0 > > Time Spent: 40m > Remaining Estimate: 0h > > DataflowWorkerLoggingInitializer should guard so that reset() can't run > unless initialize() is first run. We should also make initialize() not able > to be run unless it has never been run or reset() has been run. > Further details in ML thread: > https://lists.apache.org/thread.html/e0c04e1e5f1efb623256ace2393b5b9a899174e726e72f9903fedf19@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7016) DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never initialized
[ https://issues.apache.org/jira/browse/BEAM-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-7016. -- Resolution: Fixed Fix Version/s: 2.13.0 > DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never > initialized > - > > Key: BEAM-7016 > URL: https://issues.apache.org/jira/browse/BEAM-7016 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Luke Cwik >Assignee: Michael Luckey >Priority: Minor > Labels: starter > Fix For: 2.13.0 > > Time Spent: 40m > Remaining Estimate: 0h > > DataflowWorkerLoggingInitializer should guard so that reset() can't run > unless initialize() is first run. We should also make initialize() not able > to be run unless it has never been run or reset() has been run. > Further details in ML thread: > https://lists.apache.org/thread.html/e0c04e1e5f1efb623256ace2393b5b9a899174e726e72f9903fedf19@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-7016) DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never initialized
[ https://issues.apache.org/jira/browse/BEAM-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-7016 started by Michael Luckey. > DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never > initialized > - > > Key: BEAM-7016 > URL: https://issues.apache.org/jira/browse/BEAM-7016 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Luke Cwik >Assignee: Michael Luckey >Priority: Minor > Labels: starter > > DataflowWorkerLoggingInitializer should guard so that reset() can't run > unless initialize() is first run. We should also make initialize() not able > to be run unless it has never been run or reset() has been run. > Further details in ML thread: > https://lists.apache.org/thread.html/e0c04e1e5f1efb623256ace2393b5b9a899174e726e72f9903fedf19@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-7016) DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never initialized
[ https://issues.apache.org/jira/browse/BEAM-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey reassigned BEAM-7016: Assignee: Michael Luckey > DataflowWorkerLoggingInitializer.reset() sets System.out/err to null if never > initialized > - > > Key: BEAM-7016 > URL: https://issues.apache.org/jira/browse/BEAM-7016 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Luke Cwik >Assignee: Michael Luckey >Priority: Minor > Labels: starter > > DataflowWorkerLoggingInitializer should guard so that reset() can't run > unless initialize() is first run. We should also make initialize() not able > to be run unless it has never been run or reset() has been run. > Further details in ML thread: > https://lists.apache.org/thread.html/e0c04e1e5f1efb623256ace2393b5b9a899174e726e72f9903fedf19@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-7017) Improve Apache Rat failure message during build time
[
https://issues.apache.org/jira/browse/BEAM-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey resolved BEAM-7017.
--
Resolution: Fixed
Fix Version/s: Not applicable
> Improve Apache Rat failure message during build time
>
>
> Key: BEAM-7017
> URL: https://issues.apache.org/jira/browse/BEAM-7017
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> 0.4.0 of the plugin embeds the filenames with invalid licenses in the
> exception message of the failing task so it now appears at the bottom of the
> build.
> {code}
> > Task :rat FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':rat'.
> > A failure occurred while executing org.nosphere.apache.rat.RatWork
>> Apache Rat audit failure
>
> *
> Summary
> ---
> Generated at: 2019-04-05T14:39:11-07:00
>
> Notes: 5
> Binaries: 123
> Archives: 4
> Standards: 5105
>
> Apache Licensed: 5104
> Generated Documents: 0
>
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
>
> 1 Unknown Licenses
>
> *
>
> Files with unapproved licenses:
>
>
> /usr/local/google/home/lcwik/git/beam/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
>
> *
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-6981) :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
[
https://issues.apache.org/jira/browse/BEAM-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey closed BEAM-6981.
> :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
>
>
> Key: BEAM-6981
> URL: https://issues.apache.org/jira/browse/BEAM-6981
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Beams javadoc project does not apply beam module plugin.
> As rhe aggregateJavadoc tasks configuration does not set
> {noformat}
> options.encoding = 'UTF-8'
> {noformat}
> and the option is also not inherited from beam modules applyJavaNature
> current javadoc creation relies on systems default file encoding (or whatever
> happens to be active for the jvm)
> So we either need to set it explicitly, or apply our plugin. The latter is
> probably better, as it ensures all configuration globally set for java
> projects will be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-6981) :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
[
https://issues.apache.org/jira/browse/BEAM-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey resolved BEAM-6981.
--
Resolution: Fixed
Fix Version/s: Not applicable
> :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
>
>
> Key: BEAM-6981
> URL: https://issues.apache.org/jira/browse/BEAM-6981
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Beams javadoc project does not apply beam module plugin.
> As rhe aggregateJavadoc tasks configuration does not set
> {noformat}
> options.encoding = 'UTF-8'
> {noformat}
> and the option is also not inherited from beam modules applyJavaNature
> current javadoc creation relies on systems default file encoding (or whatever
> happens to be active for the jvm)
> So we either need to set it explicitly, or apply our plugin. The latter is
> probably better, as it ensures all configuration globally set for java
> projects will be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-6973) Quieten javadoc generation
[ https://issues.apache.org/jira/browse/BEAM-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-6973. -- Resolution: Fixed Fix Version/s: Not applicable > Quieten javadoc generation > -- > > Key: BEAM-6973 > URL: https://issues.apache.org/jira/browse/BEAM-6973 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 50m > Remaining Estimate: 0h > > javadoc linter complains about lots of missing 'param' and 'return' tags. > This clutters logs and might hide more important things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-6973) Quieten javadoc generation
[ https://issues.apache.org/jira/browse/BEAM-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-6973. > Quieten javadoc generation > -- > > Key: BEAM-6973 > URL: https://issues.apache.org/jira/browse/BEAM-6973 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Fix For: Not applicable > > Time Spent: 1h 50m > Remaining Estimate: 0h > > javadoc linter complains about lots of missing 'param' and 'return' tags. > This clutters logs and might hide more important things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-6178) Create BOM for Beam
[ https://issues.apache.org/jira/browse/BEAM-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-6178. > Create BOM for Beam > --- > > Key: BEAM-6178 > URL: https://issues.apache.org/jira/browse/BEAM-6178 > Project: Beam > Issue Type: Improvement > Components: build-system >Affects Versions: 2.10.0 >Reporter: Garrett Jones >Assignee: Garrett Jones >Priority: Critical > Labels: triaged > Fix For: Not applicable > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Add a module to Beam which generates a BOM for Beam modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6178) Create BOM for Beam
[ https://issues.apache.org/jira/browse/BEAM-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810355#comment-16810355 ] Michael Luckey commented on BEAM-6178: -- Thanks for your confirmation. Going to resolve now. > Create BOM for Beam > --- > > Key: BEAM-6178 > URL: https://issues.apache.org/jira/browse/BEAM-6178 > Project: Beam > Issue Type: Improvement > Components: build-system >Affects Versions: 2.10.0 >Reporter: Garrett Jones >Assignee: Garrett Jones >Priority: Critical > Labels: triaged > Time Spent: 4h 50m > Remaining Estimate: 0h > > Add a module to Beam which generates a BOM for Beam modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-6178) Create BOM for Beam
[ https://issues.apache.org/jira/browse/BEAM-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-6178. -- Resolution: Fixed Fix Version/s: Not applicable > Create BOM for Beam > --- > > Key: BEAM-6178 > URL: https://issues.apache.org/jira/browse/BEAM-6178 > Project: Beam > Issue Type: Improvement > Components: build-system >Affects Versions: 2.10.0 >Reporter: Garrett Jones >Assignee: Garrett Jones >Priority: Critical > Labels: triaged > Fix For: Not applicable > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Add a module to Beam which generates a BOM for Beam modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6178) Create BOM for Beam
[ https://issues.apache.org/jira/browse/BEAM-6178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809360#comment-16809360 ] Michael Luckey commented on BEAM-6178: -- [~garrettjonesgoogle] Is there any work left on this issue or may we close? > Create BOM for Beam > --- > > Key: BEAM-6178 > URL: https://issues.apache.org/jira/browse/BEAM-6178 > Project: Beam > Issue Type: Improvement > Components: build-system >Affects Versions: 2.10.0 >Reporter: Garrett Jones >Assignee: Garrett Jones >Priority: Critical > Labels: triaged > Time Spent: 4h 50m > Remaining Estimate: 0h > > Add a module to Beam which generates a BOM for Beam modules. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6519) java extension must not use org.apache.beam.sdk.util
[ https://issues.apache.org/jira/browse/BEAM-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809351#comment-16809351 ] Michael Luckey commented on BEAM-6519: -- [~romain.manni-bucau] Will this be fixed with https://github.com/apache/beam/pull/8217 ? Or did you encounter more violations? > java extension must not use org.apache.beam.sdk.util > > > Key: BEAM-6519 > URL: https://issues.apache.org/jira/browse/BEAM-6519 > Project: Beam > Issue Type: Bug > Components: build-system >Affects Versions: 2.9.0 >Reporter: Romain Manni-Bucau >Assignee: Romain Manni-Bucau >Priority: Blocker > Labels: triaged > > Some shades of beam reuse sdk java core packages (the util one in particular) > This is preventing to use beam in several environments (OSGi, classloader > isolation, java 11/jpms etc), please fix it asap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-6981) :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
[
https://issues.apache.org/jira/browse/BEAM-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-6981 started by Michael Luckey.
> :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
>
>
> Key: BEAM-6981
> URL: https://issues.apache.org/jira/browse/BEAM-6981
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
>
> Beams javadoc project does not apply beam module plugin.
> As rhe aggregateJavadoc tasks configuration does not set
> {noformat}
> options.encoding = 'UTF-8'
> {noformat}
> and the option is also not inherited from beam modules applyJavaNature
> current javadoc creation relies on systems default file encoding (or whatever
> happens to be active for the jvm)
> So we either need to set it explicitly, or apply our plugin. The latter is
> probably better, as it ensures all configuration globally set for java
> projects will be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6981) :beam-sdks-java-javadoc:aggregateJavadoc has no encoding set
Michael Luckey created BEAM-6981:
Summary: :beam-sdks-java-javadoc:aggregateJavadoc has no encoding
set
Key: BEAM-6981
URL: https://issues.apache.org/jira/browse/BEAM-6981
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Michael Luckey
Assignee: Michael Luckey
Beams javadoc project does not apply beam module plugin.
As rhe aggregateJavadoc tasks configuration does not set
{noformat}
options.encoding = 'UTF-8'
{noformat}
and the option is also not inherited from beam modules applyJavaNature current
javadoc creation relies on systems default file encoding (or whatever happens
to be active for the jvm)
So we either need to set it explicitly, or apply our plugin. The latter is
probably better, as it ensures all configuration globally set for java projects
will be used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-6971) Remove :beam-website:testWebsite from gradle build target
[ https://issues.apache.org/jira/browse/BEAM-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-6971. -- Resolution: Fixed Fix Version/s: Not applicable > Remove :beam-website:testWebsite from gradle build target > - > > Key: BEAM-6971 > URL: https://issues.apache.org/jira/browse/BEAM-6971 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > Rationale: > - the task seems to be very flaky. In fact, I always need to add '-x > :beam-website:testWebsite' to my build [1] > - task uses docker, which imho adds a (unnecessary) severe constraint on the > build task. E.g. A part time user is unable to execute these tests in a > docker environment > - these tests are accessing production environment. So myself hitting the > build several times an hour could be considered a DOS attack. > See also > https://lists.apache.org/thread.html/f5eeadadbfd976b4d6db05e4ffc91aeaaa8e35b950ff6505706f526e@%3Cdev.beam.apache.org%3E > [1] https://issues.apache.org/jira/browse/BEAM-6760 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-6971) Remove :beam-website:testWebsite from gradle build target
[ https://issues.apache.org/jira/browse/BEAM-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey closed BEAM-6971. > Remove :beam-website:testWebsite from gradle build target > - > > Key: BEAM-6971 > URL: https://issues.apache.org/jira/browse/BEAM-6971 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Fix For: Not applicable > > Time Spent: 40m > Remaining Estimate: 0h > > Rationale: > - the task seems to be very flaky. In fact, I always need to add '-x > :beam-website:testWebsite' to my build [1] > - task uses docker, which imho adds a (unnecessary) severe constraint on the > build task. E.g. A part time user is unable to execute these tests in a > docker environment > - these tests are accessing production environment. So myself hitting the > build several times an hour could be considered a DOS attack. > See also > https://lists.apache.org/thread.html/f5eeadadbfd976b4d6db05e4ffc91aeaaa8e35b950ff6505706f526e@%3Cdev.beam.apache.org%3E > [1] https://issues.apache.org/jira/browse/BEAM-6760 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-6973) Quieten javadoc generation
[ https://issues.apache.org/jira/browse/BEAM-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-6973 started by Michael Luckey. > Quieten javadoc generation > -- > > Key: BEAM-6973 > URL: https://issues.apache.org/jira/browse/BEAM-6973 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > javadoc linter complains about lots of missing 'param' and 'return' tags. > This clutters logs and might hide more important things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6973) Quieten javadoc generation
Michael Luckey created BEAM-6973: Summary: Quieten javadoc generation Key: BEAM-6973 URL: https://issues.apache.org/jira/browse/BEAM-6973 Project: Beam Issue Type: Improvement Components: build-system Reporter: Michael Luckey Assignee: Michael Luckey javadoc linter complains about lots of missing 'param' and 'return' tags. This clutters logs and might hide more important things. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (BEAM-6971) Remove :beam-website:testWebsite from gradle build target
[ https://issues.apache.org/jira/browse/BEAM-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-6971 started by Michael Luckey. > Remove :beam-website:testWebsite from gradle build target > - > > Key: BEAM-6971 > URL: https://issues.apache.org/jira/browse/BEAM-6971 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Rationale: > - the task seems to be very flaky. In fact, I always need to add '-x > :beam-website:testWebsite' to my build [1] > - task uses docker, which imho adds a (unnecessary) severe constraint on the > build task. E.g. A part time user is unable to execute these tests in a > docker environment > - these tests are accessing production environment. So myself hitting the > build several times an hour could be considered a DOS attack. > See also > https://lists.apache.org/thread.html/f5eeadadbfd976b4d6db05e4ffc91aeaaa8e35b950ff6505706f526e@%3Cdev.beam.apache.org%3E > [1] https://issues.apache.org/jira/browse/BEAM-6760 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6971) Remove :beam-website:testWebsite from gradle build target
[ https://issues.apache.org/jira/browse/BEAM-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey updated BEAM-6971: - Description: Rationale: - the task seems to be very flaky. In fact, I always need to add '-x :beam-website:testWebsite' to my build [1] - task uses docker, which imho adds a (unnecessary) severe constraint on the build task. E.g. A part time user is unable to execute these tests in a docker environment - these tests are accessing production environment. So myself hitting the build several times an hour could be considered a DOS attack. See also https://lists.apache.org/thread.html/f5eeadadbfd976b4d6db05e4ffc91aeaaa8e35b950ff6505706f526e@%3Cdev.beam.apache.org%3E [1] https://issues.apache.org/jira/browse/BEAM-6760 was: Rationale: - the task seems to be very flaky. In fact, I always need to add '-x :beam-website:testWebsite' to my build [1] - task uses docker, which imho adds a (unnecessary) severe constraint on the build task. E.g. A part time user is unable to execute these tests in a docker environment - these tests are accessing production environment. So myself hitting the build several times an hour could be considered a DOS attack. [1] https://issues.apache.org/jira/browse/BEAM-6760 > Remove :beam-website:testWebsite from gradle build target > - > > Key: BEAM-6971 > URL: https://issues.apache.org/jira/browse/BEAM-6971 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Rationale: > - the task seems to be very flaky. In fact, I always need to add '-x > :beam-website:testWebsite' to my build [1] > - task uses docker, which imho adds a (unnecessary) severe constraint on the > build task. E.g. A part time user is unable to execute these tests in a > docker environment > - these tests are accessing production environment. So myself hitting the > build several times an hour could be considered a DOS attack. > See also > https://lists.apache.org/thread.html/f5eeadadbfd976b4d6db05e4ffc91aeaaa8e35b950ff6505706f526e@%3Cdev.beam.apache.org%3E > [1] https://issues.apache.org/jira/browse/BEAM-6760 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6971) Remove :beam-website:testWebsite from gradle build target
Michael Luckey created BEAM-6971: Summary: Remove :beam-website:testWebsite from gradle build target Key: BEAM-6971 URL: https://issues.apache.org/jira/browse/BEAM-6971 Project: Beam Issue Type: Improvement Components: build-system Reporter: Michael Luckey Assignee: Michael Luckey Rationale: - the task seems to be very flaky. In fact, I always need to add '-x :beam-website:testWebsite' to my build [1] - task uses docker, which imho adds a (unnecessary) severe constraint on the build task. E.g. A part time user is unable to execute these tests in a docker environment - these tests are accessing production environment. So myself hitting the build several times an hour could be considered a DOS attack. [1] https://issues.apache.org/jira/browse/BEAM-6760 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807313#comment-16807313
]
Michael Luckey commented on BEAM-4046:
--
As Luke pointed out, we hit a confirmed gradle issue with duplicate project
names here, see https://github.com/gradle/gradle/issues/847
{noformat}
container
dataflow
examples
fn-execution
java
jdbc
job-server
job-server-container
py3
{noformat}
Maybe one of the proposed workarounds is viable.
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Priority: Major
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey reassigned BEAM-4046:
Assignee: Michael Luckey
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4046) Decouple gradle project names and maven artifact ids
[
https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807009#comment-16807009
]
Michael Luckey commented on BEAM-4046:
--
Thanks, Luke, for pointing that out. Indeed, I have not yet tested deeply, so
it might happen that those issues are difficult to solve.
Regarding the backwards compatibility, the 'best' I came up with, apart from
having own gradle distribution with hacked task resolution or rewriting
Gradle-wrapper.jar is to hook into parameter resolution of gradlew resp
gradlew.bat.
This would imply to (temporarily!) replace the current scripts by some enhanced
implementation which would map any param starting with ':beam-' to the
corresponding target project. This could possibly work, but feels *really*
hacky :(
> Decouple gradle project names and maven artifact ids
>
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Kenneth Knowles
>Priority: Major
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}.
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle,
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be
> mandatory then it needs to be documented inline everywhere - the
> settings.gradle should say why it is so bizarre, and each build.gradle should
> indicate what its project id is.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6949) Python Gcp tests stalling on machine without credentials
Michael Luckey created BEAM-6949:
Summary: Python Gcp tests stalling on machine without credentials
Key: BEAM-6949
URL: https://issues.apache.org/jira/browse/BEAM-6949
Project: Beam
Issue Type: Bug
Components: test-failures
Reporter: Michael Luckey
Assignee: Pablo Estrada
When executing
{noformat}
python setup.py nosetests --tests
apache_beam.io.gcp.bigquery_file_loads_test:TestBigQueryFileLoads.test_records_traverse_transform_with_mocks
{noformat}
the test get stuck in between.
It seems, this test requires proper GCP credentials available. If those are not
set, used libraries seem to fall back to some 'local mode' which results in
starting a custom webserver and waiting forever.
Impact:
- bad user experience
- creating kind of zombie processes. After killing the test, the web server is
still running 'forever' and blocking standard ports (8080, 8090) on dev
machine. Also gradle daemon 'never' stops but keeps running in background.
See also thread on mailing list [1]:
{noformat}
Running
python setup.py nosetests --tests
apache_beam.io.gcp.bigquery_file_loads_test:TestBigQueryFileLoads.test_records_traverse_transform_with_mocks
and hitting 'Ctrl-C' after it got stuck, results in following output:
'KeyboardInterrupt [while running
\'WriteToBigQuery/BigQueryBatchFileLoads/RemoveTempTables/Delete\']\n
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/v2/auth?scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery+https%3A%
If your browser is on a different machine then exit and re-run this
application with the command-line parameter
--noauth_local_webserver
Failed to find "code" in the query parameters of the redirect.
Invalid authorization: Try running with --noauth_local_webserver.
{noformat}
[1]
https://lists.apache.org/thread.html/80f3de15b04ffb93bac114dbf2fc8623fa5cd8994592072352362ba7@%3Cdev.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (BEAM-6946) rat task failing on .test-infra/tools/vendor/
[
https://issues.apache.org/jira/browse/BEAM-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey closed BEAM-6946.
> rat task failing on .test-infra/tools/vendor/
> -
>
> Key: BEAM-6946
> URL: https://issues.apache.org/jira/browse/BEAM-6946
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The folder .test-infra/tools/vendor/ is created during build, but not part of
> .gitignore.
> This causes rat task to fail, if reexecuted without explicit clean before.
> Steps to reproduce:
> {noformat}
> # ./gradlew -p .test-infra/tools/ build
> # git status --porcelain
> ?? .test-infra/tools/vendor/
> # ./gradlew rat
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-6946) rat task failing on .test-infra/tools/vendor/
[
https://issues.apache.org/jira/browse/BEAM-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey resolved BEAM-6946.
--
Resolution: Fixed
Fix Version/s: Not applicable
> rat task failing on .test-infra/tools/vendor/
> -
>
> Key: BEAM-6946
> URL: https://issues.apache.org/jira/browse/BEAM-6946
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The folder .test-infra/tools/vendor/ is created during build, but not part of
> .gitignore.
> This causes rat task to fail, if reexecuted without explicit clean before.
> Steps to reproduce:
> {noformat}
> # ./gradlew -p .test-infra/tools/ build
> # git status --porcelain
> ?? .test-infra/tools/vendor/
> # ./gradlew rat
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-6946) rat task failing on .test-infra/tools/vendor/
[
https://issues.apache.org/jira/browse/BEAM-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on BEAM-6946 started by Michael Luckey.
> rat task failing on .test-infra/tools/vendor/
> -
>
> Key: BEAM-6946
> URL: https://issues.apache.org/jira/browse/BEAM-6946
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The folder .test-infra/tools/vendor/ is created during build, but not part of
> .gitignore.
> This causes rat task to fail, if reexecuted without explicit clean before.
> Steps to reproduce:
> {noformat}
> # ./gradlew -p .test-infra/tools/ build
> # git status --porcelain
> ?? .test-infra/tools/vendor/
> # ./gradlew rat
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (BEAM-6948) Remove superseded ant.xml for javadoc creation
[ https://issues.apache.org/jira/browse/BEAM-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on BEAM-6948 started by Michael Luckey. > Remove superseded ant.xml for javadoc creation > -- > > Key: BEAM-6948 > URL: https://issues.apache.org/jira/browse/BEAM-6948 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Assignee: Michael Luckey >Priority: Major > > Javadoc previously was created by maven delegating to ant. After migration to > Gradle, the corresponding ant build file is superseded by a build.gradle [1], > [2] > The pom.xml was deleted later after gradle migration was accepted. ant.xml > seems to be forgotten. > [1] https://issues.apache.org/jira/browse/BEAM-4108 > [2] https://github.com/apache/beam/pull/5121 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-6948) Remove superseded ant.xml for javadoc creation
Michael Luckey created BEAM-6948: Summary: Remove superseded ant.xml for javadoc creation Key: BEAM-6948 URL: https://issues.apache.org/jira/browse/BEAM-6948 Project: Beam Issue Type: Improvement Components: build-system Reporter: Michael Luckey Assignee: Michael Luckey Javadoc previously was created by maven delegating to ant. After migration to Gradle, the corresponding ant build file is superseded by a build.gradle [1], [2] The pom.xml was deleted later after gradle migration was accepted. ant.xml seems to be forgotten. [1] https://issues.apache.org/jira/browse/BEAM-4108 [2] https://github.com/apache/beam/pull/5121 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6946) rat task failing on .test-infra/tools/vendor/
[
https://issues.apache.org/jira/browse/BEAM-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey updated BEAM-6946:
-
Description:
The folder .test-infra/tools/vendor/ is created during build, but not part of
.gitignore.
This causes rat task to fail, if reexecuted without explicit clean before.
Steps to reproduce:
{noformat}
# ./gradlew -p .test-infra/tools/ build
# git status --porcelain
?? .test-infra/tools/vendor/
# ./gradlew rat
{noformat}
was:
The folder .test-infra/tools/vendor/ is created during build, but not part of
.gitignore.
This causes rat task to fail, if reexecuted without explicit clean before.
> rat task failing on .test-infra/tools/vendor/
> -
>
> Key: BEAM-6946
> URL: https://issues.apache.org/jira/browse/BEAM-6946
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
>
> The folder .test-infra/tools/vendor/ is created during build, but not part of
> .gitignore.
> This causes rat task to fail, if reexecuted without explicit clean before.
> Steps to reproduce:
> {noformat}
> # ./gradlew -p .test-infra/tools/ build
> # git status --porcelain
> ?? .test-infra/tools/vendor/
> # ./gradlew rat
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6946) rat task failing on .test-infra/tools/vendor/
Michael Luckey created BEAM-6946: Summary: rat task failing on .test-infra/tools/vendor/ Key: BEAM-6946 URL: https://issues.apache.org/jira/browse/BEAM-6946 Project: Beam Issue Type: Bug Components: build-system Reporter: Michael Luckey Assignee: Michael Luckey The folder .test-infra/tools/vendor/ is created during build, but not part of .gitignore. This causes rat task to fail, if reexecuted without explicit clean before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6527) Parallel tox (unit) tests run on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799036#comment-16799036 ] Michael Luckey commented on BEAM-6527: -- [~markflyhigh] could this be closed now? > Parallel tox (unit) tests run on Jenkins > > > Key: BEAM-6527 > URL: https://issues.apache.org/jira/browse/BEAM-6527 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Mark Liu >Assignee: Mark Liu >Priority: Major > Labels: triaged > Time Spent: 11h 10m > Remaining Estimate: 0h > > Existing tox unit test suite (basic, gcp and cython) will be enabled in > multiple version of Python 3, which will significantly increase runtime of > Pre/PostCommit build. A parallel is wanted in tox or Gradle invocation to > control the time in a reasonable range (<30mins for PreCommit is desired). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6760) [beam_PreCommit_Website_Cron] TestWebsite task failing
[ https://issues.apache.org/jira/browse/BEAM-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793164#comment-16793164 ] Michael Luckey commented on BEAM-6760: -- Honestly, I did not follow the discussion on mailing list regarding this. And unfortunately do not have an understanding, what's the root case to this issue. Is it jira preventing some DOS attacks and therefore slowing down responses? Or is jira just overloaded? Or something completely different? > [beam_PreCommit_Website_Cron] TestWebsite task failing > -- > > Key: BEAM-6760 > URL: https://issues.apache.org/jira/browse/BEAM-6760 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Labels: currently-failing, triaged > Time Spent: 20m > Remaining Estimate: 0h > > _Use this form to file an issue for test failure:_ > * Jenkins Job: https://builds.apache.org/job/beam_PreCommit_Website_Cron/725/ > * Gradle Build Scan: https://scans.gradle.com/s/kvfj7ly76dme4 > * [Test source code|TODO] > Initial investigation: > There are broken-link failures for links that are not broken. Strange. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-6760) [beam_PreCommit_Website_Cron] TestWebsite task failing
[ https://issues.apache.org/jira/browse/BEAM-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792610#comment-16792610 ] Michael Luckey edited comment on BEAM-6760 at 3/14/19 12:21 PM: This seems still to be broken. See https://scans.gradle.com/s/37dapyjlibxnq/console-log?task=:beam-website:testWebsite or https://scans.gradle.com/s/52bux7l5lwjky/console-log?task=:beam-website:testWebsite was (Author: michel): This seems still to be broken. See https://scans.gradle.com/s/37dapyjlibxnq/console-log?task=:beam-website:testWebsite > [beam_PreCommit_Website_Cron] TestWebsite task failing > -- > > Key: BEAM-6760 > URL: https://issues.apache.org/jira/browse/BEAM-6760 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Labels: currently-failing, triaged > Time Spent: 20m > Remaining Estimate: 0h > > _Use this form to file an issue for test failure:_ > * Jenkins Job: https://builds.apache.org/job/beam_PreCommit_Website_Cron/725/ > * Gradle Build Scan: https://scans.gradle.com/s/kvfj7ly76dme4 > * [Test source code|TODO] > Initial investigation: > There are broken-link failures for links that are not broken. Strange. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6760) [beam_PreCommit_Website_Cron] TestWebsite task failing
[ https://issues.apache.org/jira/browse/BEAM-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792610#comment-16792610 ] Michael Luckey commented on BEAM-6760: -- This seems still to be broken. See https://scans.gradle.com/s/37dapyjlibxnq/console-log?task=:beam-website:testWebsite > [beam_PreCommit_Website_Cron] TestWebsite task failing > -- > > Key: BEAM-6760 > URL: https://issues.apache.org/jira/browse/BEAM-6760 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Pablo Estrada >Assignee: Pablo Estrada >Priority: Major > Labels: currently-failing, triaged > Time Spent: 20m > Remaining Estimate: 0h > > _Use this form to file an issue for test failure:_ > * Jenkins Job: https://builds.apache.org/job/beam_PreCommit_Website_Cron/725/ > * Gradle Build Scan: https://scans.gradle.com/s/kvfj7ly76dme4 > * [Test source code|TODO] > Initial investigation: > There are broken-link failures for links that are not broken. Strange. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6798) Reconsider usage of gradle release plugin
[
https://issues.apache.org/jira/browse/BEAM-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790318#comment-16790318
]
Michael Luckey commented on BEAM-6798:
--
Well, I ll take it. Did not start yet, though.
> Reconsider usage of gradle release plugin
> -
>
> Key: BEAM-6798
> URL: https://issues.apache.org/jira/browse/BEAM-6798
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
>
> Currently, we use the gradle release plugin in a way probably not matching
> plugins own expectations. Some of this was discussed in [1]
> After release branch was cut, we call [2]
> {noformat}
> ./gradlew release
> {noformat}
> Apart from doing some validations, this creates two commits changing version
> property
> # sets version in gradle.properties to '${RELEASE}-RC${RC_NUM}' (Commit_1)
> # sets version in gradle.properties to back to '${RELEASE}-SNAPSHOT'
> (Commit_2)
> Commit_1 will also be tagged as (tag: v${RELEASE}-RC${RC_NUM})
> Afterwards, we continue with 'Commit_2' in testing, bundling and publishing.
> I.e. looking into source distribution published, this is not the one tagged,
> but its successor. This is probably suboptimal.
> The release plugins expectations would probably more along the lines to
> actually increment next version (either patch, minor or even major) and
> release on that Commit_1.
> Based on my current understanding, it seems easier to either
> * drop usage of gradle release plugin and just fall back to a plain 'exec
> git tag'
> * use a beam-release task which depends on gradle release checks, but does
> no version changes nor commits
> The former has the drawback to also drop the checks done by release plugin,
> e.g.
> * checkCommitNeeded
> * checkUpdateNeeded
> * checkSnapshotDependencies
> * runBuildTasks
> * createReleaseTag
> which might be still valuable.
> [1]
> [https://lists.apache.org/thread.html/205472bdaf3c2c5876533750d417c19b0d1078131a3dc04916082ce8@%3Cdev.beam.apache.org%3E]
> [2]
> [https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L92-L94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-6798) Reconsider usage of gradle release plugin
[
https://issues.apache.org/jira/browse/BEAM-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Luckey reassigned BEAM-6798:
Assignee: Michael Luckey
> Reconsider usage of gradle release plugin
> -
>
> Key: BEAM-6798
> URL: https://issues.apache.org/jira/browse/BEAM-6798
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Michael Luckey
>Assignee: Michael Luckey
>Priority: Major
>
> Currently, we use the gradle release plugin in a way probably not matching
> plugins own expectations. Some of this was discussed in [1]
> After release branch was cut, we call [2]
> {noformat}
> ./gradlew release
> {noformat}
> Apart from doing some validations, this creates two commits changing version
> property
> # sets version in gradle.properties to '${RELEASE}-RC${RC_NUM}' (Commit_1)
> # sets version in gradle.properties to back to '${RELEASE}-SNAPSHOT'
> (Commit_2)
> Commit_1 will also be tagged as (tag: v${RELEASE}-RC${RC_NUM})
> Afterwards, we continue with 'Commit_2' in testing, bundling and publishing.
> I.e. looking into source distribution published, this is not the one tagged,
> but its successor. This is probably suboptimal.
> The release plugins expectations would probably more along the lines to
> actually increment next version (either patch, minor or even major) and
> release on that Commit_1.
> Based on my current understanding, it seems easier to either
> * drop usage of gradle release plugin and just fall back to a plain 'exec
> git tag'
> * use a beam-release task which depends on gradle release checks, but does
> no version changes nor commits
> The former has the drawback to also drop the checks done by release plugin,
> e.g.
> * checkCommitNeeded
> * checkUpdateNeeded
> * checkSnapshotDependencies
> * runBuildTasks
> * createReleaseTag
> which might be still valuable.
> [1]
> [https://lists.apache.org/thread.html/205472bdaf3c2c5876533750d417c19b0d1078131a3dc04916082ce8@%3Cdev.beam.apache.org%3E]
> [2]
> [https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L92-L94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (BEAM-6726) Gradle Publish fails with Gradle 5
[ https://issues.apache.org/jira/browse/BEAM-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey resolved BEAM-6726. -- Resolution: Fixed > Gradle Publish fails with Gradle 5 > -- > > Key: BEAM-6726 > URL: https://issues.apache.org/jira/browse/BEAM-6726 > Project: Beam > Issue Type: Bug > Components: build-system >Affects Versions: 2.11.0 >Reporter: Ahmet Altay >Assignee: Michael Luckey >Priority: Blocker > Fix For: 2.12.0 > > Time Spent: 4h > Remaining Estimate: 0h > > cc: [~alanmyrvold] [~kenn] > :beam-sdks-java-bom:signMavenJavaPublication task fails with an obscure > error: > (https://scans.gradle.com/s/mcbb4axlx6agy/failure?openFailures=WzBd=WzFd#top=0): > Duplicate key pom-default.xml.asc:xml.asc:asc:null (attempted merging values > Signature pom-default.xml.asc:xml.asc:asc:null and Signature > pom-default.xml.asc:xml.asc:asc:null) > Downgrading to Gradle 4 by reverting > https://github.com/apache/beam/commit/cadb6f7fabc6faedc6037104338306688f17652f > works. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6795) Improve Release Scripts
[ https://issues.apache.org/jira/browse/BEAM-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789950#comment-16789950 ] Michael Luckey commented on BEAM-6795: -- In discussion of PR [#8026|https://github.com/apache/beam/pull/8026] it was suggested to add some consistency validations - check that user input matches across scripts, especially the signing key Current script implementations do not support here. > Improve Release Scripts > --- > > Key: BEAM-6795 > URL: https://issues.apache.org/jira/browse/BEAM-6795 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ahmet Altay >Priority: Major > > - Scripts use sudo to install binaries. Could be improved by local > installations, or perhaps using a container for build the release. > - Scripts make changes to bashrc file (e.g. alias hub to git), these could be > avoided. Even though scripts attempt make a backup file, it is easy to > override them if the script is cancelled. > - There are too many yes/no questions, configuration questions for > validations. They are not set and forget requires attention. (Possible > solutions: use command line arguments) > - Once script fails at any step (e.g. invalid password at a step) it fails > without giving a second chance and requires re-running from the top. > (Posssible idea: use breadcrumbs to continue the script for its last known > location.) > - Signing with GPG is not friendly when used from a remote terminal. Has > modal dialogs and does not interact well with gradle. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-6799) Streamline project version usage
[ https://issues.apache.org/jira/browse/BEAM-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Luckey updated BEAM-6799: - Description: Currently we have project version set in gradle.properties [1]. On the other hand we overwrite immediately the project version in BeamModulePlugin [2], so that the usage in gradle.properties is superfluous. This is intermingled with isRelease flag, which toggles version as snapshot version. Revisit that procedure and make consistent. See also BEAM-6726 (comments about removing isRelease) and BEAM-6798 about usage of version in gradle.properties. was: Currently we have project version set in gradle.properties [1]. On the other hand we overwrite immediately the project version in BeamModulePlugin [2], so that the usage in gradle.properties superfluous. This is intermingled with isRelease flag, which toggles version as snapshot version. Revisit that procedure and make consistent. See also BEAM-6726 (comments about removing isRelease) and BEAM-6798 about usage of version in gradle.properties. > Streamline project version usage > > > Key: BEAM-6799 > URL: https://issues.apache.org/jira/browse/BEAM-6799 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Priority: Major > > Currently we have project version set in gradle.properties [1]. On the other > hand we overwrite immediately the project version in BeamModulePlugin [2], so > that the usage in gradle.properties is superfluous. > This is intermingled with isRelease flag, which toggles version as snapshot > version. Revisit that procedure and make consistent. > See also BEAM-6726 (comments about removing isRelease) and BEAM-6798 about > usage of version in gradle.properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-6798) Reconsider usage of gradle release plugin
[
https://issues.apache.org/jira/browse/BEAM-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789629#comment-16789629
]
Michael Luckey commented on BEAM-6798:
--
This was also my first thought.
But after rethinking, I am not so sure anymore. Checking for uncommitted and
unpublished changes will probably never be needed. But *if* they trigger, we
probably are really happy that we had them.
In any case, changes here are pretty straightforward, either by custom release
task disabling the commits or adding those few git commands checking repo state.
> Reconsider usage of gradle release plugin
> -
>
> Key: BEAM-6798
> URL: https://issues.apache.org/jira/browse/BEAM-6798
> Project: Beam
> Issue Type: Improvement
> Components: build-system
>Reporter: Michael Luckey
>Priority: Major
>
> Currently, we use the gradle release plugin in a way probably not matching
> plugins own expectations. Some of this was discussed in [1]
> After release branch was cut, we call [2]
> {noformat}
> ./gradlew release
> {noformat}
> Apart from doing some validations, this creates two commits changing version
> property
> # sets version in gradle.properties to '${RELEASE}-RC${RC_NUM}' (Commit_1)
> # sets version in gradle.properties to back to '${RELEASE}-SNAPSHOT'
> (Commit_2)
> Commit_1 will also be tagged as (tag: v${RELEASE}-RC${RC_NUM})
> Afterwards, we continue with 'Commit_2' in testing, bundling and publishing.
> I.e. looking into source distribution published, this is not the one tagged,
> but its successor. This is probably suboptimal.
> The release plugins expectations would probably more along the lines to
> actually increment next version (either patch, minor or even major) and
> release on that Commit_1.
> Based on my current understanding, it seems easier to either
> * drop usage of gradle release plugin and just fall back to a plain 'exec
> git tag'
> * use a beam-release task which depends on gradle release checks, but does
> no version changes nor commits
> The former has the drawback to also drop the checks done by release plugin,
> e.g.
> * checkCommitNeeded
> * checkUpdateNeeded
> * checkSnapshotDependencies
> * runBuildTasks
> * createReleaseTag
> which might be still valuable.
> [1]
> [https://lists.apache.org/thread.html/205472bdaf3c2c5876533750d417c19b0d1078131a3dc04916082ce8@%3Cdev.beam.apache.org%3E]
> [2]
> [https://github.com/apache/beam/blob/master/release/src/main/scripts/build_release_candidate.sh#L92-L94]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-6799) Streamline project version usage
Michael Luckey created BEAM-6799: Summary: Streamline project version usage Key: BEAM-6799 URL: https://issues.apache.org/jira/browse/BEAM-6799 Project: Beam Issue Type: Improvement Components: build-system Reporter: Michael Luckey Currently we have project version set in gradle.properties [1]. On the other hand we overwrite immediately the project version in BeamModulePlugin [2], so that the usage in gradle.properties superfluous. This is intermingled with isRelease flag, which toggles version as snapshot version. Revisit that procedure and make consistent. See also BEAM-6726 (comments about removing isRelease) and BEAM-6798 about usage of version in gradle.properties. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
