[jira] [Updated] (BEAM-5561) Beam Dependency Update Request: org.apache.hbase:hbase-hadoop-compat
[ https://issues.apache.org/jira/browse/BEAM-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5561: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.apache.hbase:hbase-hadoop-compat > > > Key: BEAM-5561 > URL: https://issues.apache.org/jira/browse/BEAM-5561 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:31:02.747809 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:18:16.666045 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:12:22.916827 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:12:12.366222 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:16:01.084255 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:13:56.235802 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:13:55.833381 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:14:31.554034 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:13:37.148190 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:14:00.614911 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-hadoop-compat. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide >
[jira] [Updated] (BEAM-5739) Contributor Story: "Submitting your first PR"
[ https://issues.apache.org/jira/browse/BEAM-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5739: Labels: stale-P2 (was: ) > Contributor Story: "Submitting your first PR" > - > > Key: BEAM-5739 > URL: https://issues.apache.org/jira/browse/BEAM-5739 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Priority: P2 > Labels: stale-P2 > > We should write the user story for "Submitting your first PR", with > prescriptive steps on getting started. It should include: > * Forking the repo and setting up the dev environment > * How to build/test > * Choosing an IDE > * language / SDK-specific tips + website > * "When will my changes go live?" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7522) Support customized configuration in KafkaTableProvider
[
https://issues.apache.org/jira/browse/BEAM-7522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122783#comment-17122783
]
Beam JIRA Bot commented on BEAM-7522:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Support customized configuration in KafkaTableProvider
> --
>
> Key: BEAM-7522
> URL: https://issues.apache.org/jira/browse/BEAM-7522
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Mingmin Xu
>Priority: P2
> Labels: stale-P2
>
> expand KafkaTableProvider to support
> {{BeamKafkaTable.updateConsumerProperties(...)}}, so users can add customized
> configurations in DDL.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-4913) Beam Dependency Update Request: org.elasticsearch.client
[ https://issues.apache.org/jira/browse/BEAM-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4913: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.elasticsearch.client > > > Key: BEAM-4913 > URL: https://issues.apache.org/jira/browse/BEAM-4913 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:24:54.146686 > Please review and upgrade the org.elasticsearch.client to the latest > version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5808) Beam Dependency Update Request: com.esotericsoftware
[ https://issues.apache.org/jira/browse/BEAM-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5808: Labels: stale-P2 (was: ) > Beam Dependency Update Request: com.esotericsoftware > > > Key: BEAM-5808 > URL: https://issues.apache.org/jira/browse/BEAM-5808 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-22 12:13:43.602265 > - > Please consider upgrading the dependency com.esotericsoftware. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4770) Standalone seed job didn't revert Jenkins seed job changes introduced by a PR
[
https://issues.apache.org/jira/browse/BEAM-4770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123264#comment-17123264
]
Beam JIRA Bot commented on BEAM-4770:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Standalone seed job didn't revert Jenkins seed job changes introduced by a PR
> -
>
> Key: BEAM-4770
> URL: https://issues.apache.org/jira/browse/BEAM-4770
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Lukasz Gajowy
>Priority: P2
> Labels: stale-P2
>
> +What was the setup?+
> I created Pull Request (let's call it "A") of a branch with some changes in
> job__00_seed.groovy. Those changes involved adding a new glob pattern to
> detect files not only with job_*.groovy naming pattern but also with
> /**/*Job.groovy.
> I created another PR (let's call it "B") which was "empty" - no changes added
> to the codebase. I used such PR to run job_seed_standalone (or job_00_seed)
> from master branch to revert changes introduced by A.
> So:
> - to introduce change I ran: "Run seed job" in pr A's comment
> - to revert changes I ran: "Run seed job" / "Run seed job standalone" in pr
> B's comment.
>
> +What did I expect?+
> I expected every change introduced in the seed job by A reverted by B's
> standalone seed job. I even expected that B's seed job will revert the
> changes.
> +What actually happened?+
> After running standalone seed job from "B" some changes were not reverted.
> Jenkins was still expecting files with *Job. glob pattern when I was running
> the seed job afterwards. It couldn't find them on master so it resulted in
> the following error:
> {code:java}
> Resetting working tree
> > git reset --hard # timeout=10
> > git clean -fdx # timeout=10
> ERROR: no Job DSL script(s) found at .test-infra/jenkins/**/*Job.groovy
> Not sending mail to unregistered user ccla...@bluewin.ch
> Sending e-mails to: d...@beam.apache.org pabl...@google.com
> Setting status of 4715449665ae235f96c7a67ce5f80b774d5474e5 to FAILURE with
> url https://builds.apache.org/job/beam_SeedJob/2191/ and message: 'FAILURE
> '
> Using context: Jenkins: Seed Job
> Finished: FAILURE
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5241) runMobileGamingJavaDirect FAILED SocketException: Connection reset
[ https://issues.apache.org/jira/browse/BEAM-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123163#comment-17123163 ] Beam JIRA Bot commented on BEAM-5241: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > runMobileGamingJavaDirect FAILED SocketException: Connection reset > -- > > Key: BEAM-5241 > URL: https://issues.apache.org/jira/browse/BEAM-5241 > Project: Beam > Issue Type: Bug > Components: examples-java >Reporter: Andrew Pilloud >Priority: P2 > Labels: stale-P2 > > [https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/349/] > [https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/347/] > *04:19:41* > Task :beam-runners-direct-java:runMobileGamingJavaDirect > FAILED*04:19:41* Aug 23, 2018 11:19:32 AM > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl > createTable*04:19:41* INFO: Trying to create BigQuery table: > apache-beam-testing:beam_postrelease_mobile_gaming.leaderboard_DirectRunner_user*04:19:41* > bq query SELECT table_id FROM > beam_postrelease_mobile_gaming.__TABLES_SUMMARY__*04:19:41* [ERROR] Failed to > execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on > project word-count-beam: An exception occured while executing the Java class. > java.net.SocketException: Connection reset -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5540) Beam Dependency Update Request: ply
[ https://issues.apache.org/jira/browse/BEAM-5540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123125#comment-17123125 ] Beam JIRA Bot commented on BEAM-5540: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: ply > --- > > Key: BEAM-5540 > URL: https://issues.apache.org/jira/browse/BEAM-5540 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:25:03.237506 > - > Please consider upgrading the dependency ply. > The current version is 3.8. The latest version is 3.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:10:52.450675 > - > Please consider upgrading the dependency ply. > The current version is 3.8. The latest version is 3.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4548) Long execution delay when using DirectRunner to read from BigQuery Table
[ https://issues.apache.org/jira/browse/BEAM-4548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4548: Labels: stale-P2 (was: ) > Long execution delay when using DirectRunner to read from BigQuery Table > > > Key: BEAM-4548 > URL: https://issues.apache.org/jira/browse/BEAM-4548 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-direct >Affects Versions: 2.4.0 >Reporter: Brian Foo >Priority: P2 > Labels: stale-P2 > > When using DirectRunner to execute a simple select query against a BigQuery > table that contains 100 rows, the pipeline stalls for over 3 minutes. The > BigQuery UI can run the same query in under 2 seconds. > A similar issue was reported here: > [https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read|https://www.google.com/url?q=https://stackoverflow.com/questions/46907735/beam-direct-runner-slow-bigquery-read=D=hangouts=1528912448506000=AFQjCNHp9JWHFJOnJlBJmLODU1cGBIeXtg] > I ran a thread dump using Visual M seems like the main thread was in a state > of backoff: > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at com.google.api.client.util.Sleeper$1.sleep(Sleeper.java:43) > at com.google.api.client.util.BackOffUtils.next(BackOffUtils.java:50) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.nextBackOff(BigQueryServicesImpl.java:870) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.access$500(BigQueryServicesImpl.java:79) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:273) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.pollJob(BigQueryServicesImpl.java:247) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.executeQuery(BigQueryQuerySource.java:191) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQueryQuerySource.getTableToExtract(BigQueryQuerySource.java:136) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.extractFiles(BigQuerySourceBase.java:103) > at > org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.split(BigQuerySourceBase.java:134) > at > org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:210) > at > org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:87) > at > org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:62) > at > org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:144) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:201) > at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4486) BigQuery: FILE_LOADS + CREATE_NEVER + field-based partitioning => missing schema exception
[
https://issues.apache.org/jira/browse/BEAM-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123322#comment-17123322
]
Beam JIRA Bot commented on BEAM-4486:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> BigQuery: FILE_LOADS + CREATE_NEVER + field-based partitioning => missing
> schema exception
> --
>
> Key: BEAM-4486
> URL: https://issues.apache.org/jira/browse/BEAM-4486
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Glenn Ammons
>Priority: P2
> Labels: stale-P2
>
> Our pipeline gets this error from BigQuery when using
> BigQueryIO.Write.Method.FILE_LOADS,
> BigQueryIO.Write.CreateDisposition.CREATE_NEVER, and field-based time
> partitioning (full exception at the bottom of this note):
> Table with field based partitioning must have a schema.
> We do supply a schema when we create the pipeline by calling
> BigQuery.Write.withSchema, but this schema is ignored because the
> processElement method here:
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java]
> always provides a null schema when using CREATE_NEVER.
> I would expect Beam to use the provided schema no matter what setting we are
> using for the CreateDisposition.
>
> Full exception:
> java.io.IOException: Unable to insert job:
> 078646f70a664daaa1ed96832b233036_19e873cd24cf1968559515e49b3d868d_1_0-0,
> aborting after 9 .
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:236)
>
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:204)
>
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:144)
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.load(WriteTables.java:259)
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$600(WriteTables.java:77)
>
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:155)
> Caused by:
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad
> Request \{ "code" : 400, "errors" : [ { "domain" : "global", "message" :
> "Table with field based partitioning must have a schema.", "reason" :
> "invalid" } ], "message" : "Table with field based partitioning must have a
> schema." }
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
>
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
>
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
>
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
>
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:218)
>
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:204)
>
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:144)
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.load(WriteTables.java:259)
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$600(WriteTables.java:77)
>
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:155)
>
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
>
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
>
>
[jira] [Updated] (BEAM-5465) Have the sdks/go gradle tasks clean up the vendor directories on clean
[
https://issues.apache.org/jira/browse/BEAM-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-5465:
Labels: stale-P2 (was: )
> Have the sdks/go gradle tasks clean up the vendor directories on clean
> --
>
> Key: BEAM-5465
> URL: https://issues.apache.org/jira/browse/BEAM-5465
> Project: Beam
> Issue Type: Bug
> Components: build-system, sdk-go
>Reporter: Robert Burke
>Priority: P2
> Labels: stale-P2
>
> The sdks/go/\{test,examples,container} directories depend on the main beam
> package, which cause gogradle to vendor the beam package in those directories.
>
> The gogradle plugin doesn't clean up the [vendor
> directories|https://golang.org/cmd/go/#hdr-Vendor_Directories] that it sets
> up on builds, when clean is invoked.
>
> This leads to stale copies of the Go SDK's beam package vendored in local
> directories, which can lead to build failures of other tasks that invoke the
> tests or similar, when the code in those directories uses a more recent
> version of beam than what is cached.
>
> This doesn't happen for users directly using the go tool, with their git repo
> nested under Go PATH, since the go tool will correctly use the local repo
> copy of beam.
>
> A workaround on a Unix machine or similar, invoked from the beam repo root,
> is to delete the vendor and gogradle directories, and retry the task.
>
> rm -rf sdks/go/\{vendor,.gogradle}
> sdks/go/\{test,examples,container}/\{vendor,.gogradle}
>
> This cause gogradle to fetch a more recent copy of beam for vendoring.
>
> Ideally we fix the clean tasks for the go directories to delete the vendor
> directories as well, which will resolve the issue more reliably for those
> using gradle to test their changes against the go sdk.
> Related: BEAM-5379 is for avoiding the vendoring & cleaning cycle all
> together and migrating to gomodules.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5431) StarMap transform for Python SDK
[
https://issues.apache.org/jira/browse/BEAM-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123139#comment-17123139
]
Beam JIRA Bot commented on BEAM-5431:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> StarMap transform for Python SDK
>
>
> Key: BEAM-5431
> URL: https://issues.apache.org/jira/browse/BEAM-5431
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: P2
> Labels: stale-P2
>
> I'd like to propose a new high-level transform "StarMap" for the Python SDK.
> The transform would be syntactic sugar for ParDo like Map, but would would
> automatically unpack arguments like
> [itertools.starmap|https://docs.python.org/3/library/itertools.html#itertools.starmap]
> from Python's standard library.
> The use-case is to handle applying functions to tuples of arguments, which is
> a common pattern when using Beam's combine and group-by transforms. Right
> now, it's common to write functions with manual unpacking, e.g.,
> {code:java}
> def my_func(inputs):
> key, value = inputs
> ...
> beam.Map(my_func) {code}
> StarMap offers a much more readable alternative:
> {code:java}
> def my_func(key, value):
> ...
> beam.StarMap(my_func){code}
>
> The need for StarMap is especially pressing with the advent of Python 3
> support and the eventual wind-down of Python 2. Currently, it's common to
> achieve this pattern using unpacking in a function definition, e.g.,
> beam.Map(lambda (k, v): my_func(k, v)), but this is invalid syntax in Python
> 3. My internal search of Google's codebase turns up quite a few matches for
> "beam\.Map(lambda\ (", none of which would work on Python 3.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5000) Beam Dependency Update Request: org.scala-lang
[ https://issues.apache.org/jira/browse/BEAM-5000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123202#comment-17123202 ] Beam JIRA Bot commented on BEAM-5000: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.scala-lang > -- > > Key: BEAM-5000 > URL: https://issues.apache.org/jira/browse/BEAM-5000 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:33:43.419619 > Please review and upgrade the org.scala-lang to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7582) TestPubsub.listSubscriptions is flaky
[ https://issues.apache.org/jira/browse/BEAM-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7582: Labels: stale-P2 (was: ) > TestPubsub.listSubscriptions is flaky > - > > Key: BEAM-7582 > URL: https://issues.apache.org/jira/browse/BEAM-7582 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > > o.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline expired before > operation could complete. > > at > > io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) > > at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) > > at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) > > at > > com.google.pubsub.v1.SubscriberGrpc$SubscriberBlockingStub.listSubscriptions(SubscriberGrpc.java:1734) > > at > > org.apache.beam.sdk.io.gcp.pubsub.PubsubGrpcClient.listSubscriptions(PubsubGrpcClient.java:373) > > at > > org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.listSubscriptions(TestPubsub.java:165) > https://builds.apache.org/job/beam_PostCommit_SQL/1843/ > https://builds.apache.org/job/beam_PostCommit_SQL/1842/ > https://builds.apache.org/job/beam_PostCommit_SQL/1841/ > https://builds.apache.org/job/beam_PostCommit_SQL/1840/ > https://builds.apache.org/job/beam_PostCommit_SQL/1839/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7142) Data Driven testing for BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-7142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122850#comment-17122850 ] Beam JIRA Bot commented on BEAM-7142: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Data Driven testing for BeamSQL > --- > > Key: BEAM-7142 > URL: https://issues.apache.org/jira/browse/BEAM-7142 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > > Current way to write BeamSQL test cases is too heavy: developers need to > initialize pipeline, deal with PCollection, and use PAssert to verify > pipeline results (sometime through INSERT INTO table and read data from table > for assertion). > Data driven testing, instead, should only ask developer to provide SQL query > and a expected result in the form of List (simulate rows from result > table). The test execution interface should just be a static function like > "List run(String query)", and returned rows can be compared with > expected result by checking equality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6090) Beam Dependency Update Request: org.elasticsearch:elasticsearch
[ https://issues.apache.org/jira/browse/BEAM-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6090: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.elasticsearch:elasticsearch > --- > > Key: BEAM-6090 > URL: https://issues.apache.org/jira/browse/BEAM-6090 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-11-19 12:12:57.791447 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:12:02.158689 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:12:21.276206 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-10 12:14:41.860759 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-17 12:15:13.638673 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-31 15:21:29.050590 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-07 12:24:36.273791 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-14 12:13:14.211572 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-21 12:19:43.014366 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-28 12:11:23.556912 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is
[jira] [Commented] (BEAM-6320) SpannerReadIT.testQuery flaky
[
https://issues.apache.org/jira/browse/BEAM-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122978#comment-17122978
]
Beam JIRA Bot commented on BEAM-6320:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> SpannerReadIT.testQuery flaky
> -
>
> Key: BEAM-6320
> URL: https://issues.apache.org/jira/browse/BEAM-6320
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Andrew Pilloud
>Priority: P2
> Labels: stale-P2
>
> https://builds.apache.org/job/beam_PostCommit_Java/2218/
> {code}
> WARNING: No terminal state was returned. State value UNKNOWN
> Dec 27, 2018 9:08:13 PM org.apache.beam.runners.dataflow.TestDataflowRunner
> checkForPAssertSuccess
> WARNING: Metrics not present for Dataflow job
> 2018-12-27_13_02_39-18037927821693074732.
> Dec 27, 2018 9:08:13 PM org.apache.beam.runners.dataflow.TestDataflowRunner
> run
> WARNING: Dataflow job 2018-12-27_13_02_39-18037927821693074732 did not output
> a success or failure metric.
> {code}
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-12-27_13_02_39-18037927821693074732?project=apache-beam-testing
> {code}
> com.google.cloud.spanner.SpannerException: NOT_FOUND:
> io.grpc.StatusRuntimeException: NOT_FOUND: Database not found:
> projects/apache-beam-testing/instances/beam-test/databases/beam-testdb-vkf2iqc72tevvroaop
> resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
> resource_name:
> "projects/apache-beam-testing/instances/beam-test/databases/beam-testdb-vkf2iqc72tevvroaop"
> description: "Database does not exist."
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
> at
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
> at
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.createSession(GrpcSpannerRpc.java:350)
> at com.google.cloud.spanner.SpannerImpl$2.call(SpannerImpl.java:258)
> at com.google.cloud.spanner.SpannerImpl$2.call(SpannerImpl.java:255)
> at
> com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
> at
> com.google.cloud.spanner.SpannerImpl.createSession(SpannerImpl.java:254)
> at
> com.google.cloud.spanner.BatchClientImpl.batchReadOnlyTransaction(BatchClientImpl.java:51)
> at
> org.apache.beam.sdk.io.gcp.spanner.CreateTransactionFn.processElement(CreateTransactionFn.java:47)
> Caused by: java.util.concurrent.ExecutionException:
> io.grpc.StatusRuntimeException: NOT_FOUND: Database not found:
> projects/apache-beam-testing/instances/beam-test/databases/beam-testdb-vkf2iqc72tevvroaop
> resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
> resource_name:
> "projects/apache-beam-testing/instances/beam-test/databases/beam-testdb-vkf2iqc72tevvroaop"
> description: "Database does not exist."
> at
> com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
> at
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:450)
> at
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.createSession(GrpcSpannerRpc.java:350)
> at com.google.cloud.spanner.SpannerImpl$2.call(SpannerImpl.java:258)
> at com.google.cloud.spanner.SpannerImpl$2.call(SpannerImpl.java:255)
> at
> com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
> at
> com.google.cloud.spanner.SpannerImpl.createSession(SpannerImpl.java:254)
> at
> com.google.cloud.spanner.BatchClientImpl.batchReadOnlyTransaction(BatchClientImpl.java:51)
> at
> org.apache.beam.sdk.io.gcp.spanner.CreateTransactionFn.processElement(CreateTransactionFn.java:47)
> at
> org.apache.beam.sdk.io.gcp.spanner.CreateTransactionFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
> at
>
[jira] [Commented] (BEAM-4980) Beam Dependency Update Request: io.netty
[ https://issues.apache.org/jira/browse/BEAM-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123211#comment-17123211 ] Beam JIRA Bot commented on BEAM-4980: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: io.netty > > > Key: BEAM-4980 > URL: https://issues.apache.org/jira/browse/BEAM-4980 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:31:32.157694 > Please review and upgrade the io.netty to the latest version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5564) Beam Dependency Update Request: org.apache.hbase:hbase-shaded-client
[ https://issues.apache.org/jira/browse/BEAM-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123117#comment-17123117 ] Beam JIRA Bot commented on BEAM-5564: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.hbase:hbase-shaded-client > > > Key: BEAM-5564 > URL: https://issues.apache.org/jira/browse/BEAM-5564 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:31:08.852642 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:18:33.239489 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:12:39.622960 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:12:29.052367 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:16:17.882048 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:14:13.599574 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:14:13.450722 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:14:49.275805 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:13:53.869056 > - > Please consider upgrading the dependency > org.apache.hbase:hbase-shaded-client. > The current version is 1.2.6. The latest version is 2.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The
[jira] [Commented] (BEAM-7291) Upgrade hadoop-common
[ https://issues.apache.org/jira/browse/BEAM-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122825#comment-17122825 ] Beam JIRA Bot commented on BEAM-7291: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Upgrade hadoop-common > - > > Key: BEAM-7291 > URL: https://issues.apache.org/jira/browse/BEAM-7291 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5777) Running ParDo in loop with DirectRunners raises RuntimeException
[
https://issues.apache.org/jira/browse/BEAM-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-5777:
Labels: stale-P2 (was: )
> Running ParDo in loop with DirectRunners raises RuntimeException
>
>
> Key: BEAM-5777
> URL: https://issues.apache.org/jira/browse/BEAM-5777
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Kasia Kucharczyk
>Priority: P2
> Labels: stale-P2
> Attachments: all_output.txt
>
>
> The Python [load test of ParDo operation for
> SyntheticSources|https://github.com/apache/beam/blob/faff82860c66e4050f0cfa5e874ffe6035ed0c1c/sdks/python/apache_beam/testing/load_tests/par_do_test.py#L133]
> that I created contains parametrized loop of ParDo with no operation inside
> besides metrics (this issue). With setting the number of iterations to >~200
> and running the test on DirectRunner I was encountering test failures. The
> test outputs whole (really long) pipeline logs. Some test runs raised the
> following exception:
>
> {code:java}
> Traceback (most recent call last):
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/testing/load_tests/par_do_test.py",
> line 144, in testParDo
> result = p.run()
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/testing/test_pipeline.py",
> line 104, in run
> result = super(TestPipeline, self).run(test_runner_api)
> File "/Users/kasia/Repos/beam/sdks/python/apache_beam/pipeline.py", line
> 403, in run
> self.to_runner_api(), self.runner, self._options).run(False)
> File "/Users/kasia/Repos/beam/sdks/python/apache_beam/pipeline.py", line
> 416, in run
> return self.runner.run_pipeline(self)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 139, in run_pipeline
> return runner.run_pipeline(pipeline)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 229, in run_pipeline
> return self.run_via_runner_api(pipeline.to_runner_api())
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 232, in run_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1015, in run_stages
> pcoll_buffers, safe_coders).process_bundle.metrics
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1132, in run_stage
> self._progress_frequency).process_bundle(data_input, data_output)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1388, in process_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1260, in push
> response = self.worker.do_instruction(request)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 212, in do_instruction
> request.instruction_id)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 231, in process_bundle
> self.data_channel_factory)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 343, in __init__
> self.ops = self.create_execution_tree(self.process_bundle_descriptor)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 385, in create_execution_tree
> descriptor.transforms, key=topological_height, reverse=True)])
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 320, in wrapper
> result = cache[args] = func(*args)
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 368, in get_operation
> in descriptor.transforms[transform_id].outputs.items()
> File
> "/Users/kasia/Repos/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 367, in
> for tag, pcoll_id
> ... (3 last lines repeated for long period)
>
> RuntimeError: maximum recursion depth exceeded
> {code}
>
>
> From my observation, I can say the problem appeared with various iteration
> number depending on computer resources. On my weaker computer started failing
> on ~150 iterations. The test succeeds on DataFlow with 1000 iterations (I
> didn't check higher number).
> I provide whole test output in Attachements.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-5034) Bumping com.google.cloud.bigdataoss:gcsio and :util to 1.9.2 causes tests to fail
[ https://issues.apache.org/jira/browse/BEAM-5034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5034: Labels: stale-P2 (was: ) > Bumping com.google.cloud.bigdataoss:gcsio and :util to 1.9.2 causes tests to > fail > - > > Key: BEAM-5034 > URL: https://issues.apache.org/jira/browse/BEAM-5034 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Garrett Jones >Priority: P2 > Labels: stale-P2 > Time Spent: 10m > Remaining Estimate: 0h > > Current version in Beam: 1.4.5 > Latest released version: 1.9.2 > Test failure: > ``` > java.lang.NullPointerException at > com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.createRequest(GoogleCloudStorageReadChannel.java:966) > at > com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.getMetadata(GoogleCloudStorageReadChannel.java:765) > at > com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.initEncodingAndSize(GoogleCloudStorageReadChannel.java:754) > at > com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.(GoogleCloudStorageReadChannel.java:210) > at > com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.(GoogleCloudStorageReadChannel.java:172) > at > org.apache.beam.sdk.util.GcsUtilTest.testGCSChannelCloseIdempotent(GcsUtilTest.java:774) > ``` > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8260) Run Nexmark Java based queries on Flink portable runner
[ https://issues.apache.org/jira/browse/BEAM-8260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122642#comment-17122642 ] Beam JIRA Bot commented on BEAM-8260: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Run Nexmark Java based queries on Flink portable runner > --- > > Key: BEAM-8260 > URL: https://issues.apache.org/jira/browse/BEAM-8260 > Project: Beam > Issue Type: Sub-task > Components: testing-nexmark >Reporter: Ismaël Mejía >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7612) Fix Java parquetio write transform used from Python SDK
[ https://issues.apache.org/jira/browse/BEAM-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122770#comment-17122770 ] Beam JIRA Bot commented on BEAM-7612: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Fix Java parquetio write transform used from Python SDK > --- > > Key: BEAM-7612 > URL: https://issues.apache.org/jira/browse/BEAM-7612 > Project: Beam > Issue Type: Bug > Components: java-fn-execution >Reporter: Heejong Lee >Priority: P2 > Labels: stale-P2 > > Java parquetio write transform doesn't work when it's used from Python SDK. > Possibly related to Reshuffle translation (BEAM-7421). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4885) Beam Dependency Update Request: com.gradle
[ https://issues.apache.org/jira/browse/BEAM-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4885: Labels: stale-P2 (was: ) > Beam Dependency Update Request: com.gradle > -- > > Key: BEAM-4885 > URL: https://issues.apache.org/jira/browse/BEAM-4885 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:22:18.615407 > Please review and upgrade the com.gradle to the latest version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6538) Disallow empty step names in ProcesBundleDescriptors
[ https://issues.apache.org/jira/browse/BEAM-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6538: Labels: stale-P2 (was: ) > Disallow empty step names in ProcesBundleDescriptors > > > Key: BEAM-6538 > URL: https://issues.apache.org/jira/browse/BEAM-6538 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > I discovered some tests were setting empty step names, when I added a > precondition check to MetricsContainerStep map when adding java SDK metrics. > Remove the code which returns the the default non-stepped unbounded > MetricsContainer after ensuring step names are never passed in empty or null > in tests or from a runner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4704) String operations yield incorrect results when executed through SQL shell
[
https://issues.apache.org/jira/browse/BEAM-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-4704:
Labels: stale-P2 (was: )
> String operations yield incorrect results when executed through SQL shell
> -
>
> Key: BEAM-4704
> URL: https://issues.apache.org/jira/browse/BEAM-4704
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: P2
> Labels: stale-P2
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{TRIM}} is defined to trim _all_ the characters in the first string from the
> string-to-be-trimmed. Calcite has an incorrect implementation of this. We use
> our own fixed implementation. But when executed through the SQL shell, the
> results do not match what we get from the PTransform path. Here two test
> cases that pass on {{master}} but are incorrect in the shell:
> {code:sql}
> BeamSQL> select TRIM(LEADING 'eh' FROM 'hehe__hehe');
> ++
> | EXPR$0 |
> ++
> | hehe__hehe |
> ++
> {code}
> {code:sql}
> BeamSQL> select TRIM(TRAILING 'eh' FROM 'hehe__hehe');
> ++
> | EXPR$0 |
> ++
> | hehe__heh |
> ++
> {code}
> {code:sql}
> BeamSQL> select TRIM(BOTH 'eh' FROM 'hehe__hehe');
> ++
> | EXPR$0 |
> ++
> | hehe__heh |
> ++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5740) Refactor permissions section into bullet-points
[ https://issues.apache.org/jira/browse/BEAM-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123075#comment-17123075 ] Beam JIRA Bot commented on BEAM-5740: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Refactor permissions section into bullet-points > --- > > Key: BEAM-5740 > URL: https://issues.apache.org/jira/browse/BEAM-5740 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Scott Wegner >Priority: P2 > Labels: stale-P2 > > The permissions section has good content, but it's not easily browseable if > you're looking for a specific thing (i.e. Slack permissions). We should > refactor it into bullet points. > For permissions that require reaching out via email/Slack, we should link to > some previous example. It lowers the barrier to entry if a new contributor > can copy/paste some existing template. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5170) Download pages must include sigs and hashes for archived releaseq
[ https://issues.apache.org/jira/browse/BEAM-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123174#comment-17123174 ] Beam JIRA Bot commented on BEAM-5170: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Download pages must include sigs and hashes for archived releaseq > - > > Key: BEAM-5170 > URL: https://issues.apache.org/jira/browse/BEAM-5170 > Project: Beam > Issue Type: Bug > Components: website > Environment: > https://beam.apache.org/get-started/downloads/#downloading-source-code >Reporter: Sebb >Priority: P2 > Labels: stale-P2 > > The download page currently links to several old source releases under > https://archive.apache.org/dist/. > That is fine, but each link must be accompanied by the relevant signature and > one or more hashes. These can be found alongside the release artifacts. > [For current releases, the links must use https://www.apache.org/dist/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5568) Beam Dependency Update Request: org.apache.hive:hive-exec
[ https://issues.apache.org/jira/browse/BEAM-5568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123114#comment-17123114 ] Beam JIRA Bot commented on BEAM-5568: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.hive:hive-exec > - > > Key: BEAM-5568 > URL: https://issues.apache.org/jira/browse/BEAM-5568 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:31:33.219748 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.0.3.0.1.0-187 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:19:03.542754 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.0.3.0.1.0-187 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:12:56.903248 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.0 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:13:00.251518 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.0.3.0.2.0-50 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:16:44.769555 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.0.3.0.2.0-50 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:14:32.876035 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:14:32.940629 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:15:09.414921 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:14:12.726076 > - > Please consider upgrading the dependency org.apache.hive:hive-exec. > The current version is 2.1.0. The latest version is 3.1.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:14:35.647822 > - >
[jira] [Updated] (BEAM-4684) Support @RequiresStableInput on Dataflow runner in Java SDK
[ https://issues.apache.org/jira/browse/BEAM-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4684: Labels: stale-P2 (was: ) > Support @RequiresStableInput on Dataflow runner in Java SDK > --- > > Key: BEAM-4684 > URL: https://issues.apache.org/jira/browse/BEAM-4684 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Reporter: Robin Qiu >Priority: P2 > Labels: stale-P2 > Time Spent: 4h 40m > Remaining Estimate: 0h > > https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8029) Using BigQueryIO.read with DIRECT_READ causes Illegal Mutation
[
https://issues.apache.org/jira/browse/BEAM-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-8029:
Labels: stale-P2 (was: )
> Using BigQueryIO.read with DIRECT_READ causes Illegal Mutation
> ---
>
> Key: BEAM-8029
> URL: https://issues.apache.org/jira/browse/BEAM-8029
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.14.0
>Reporter: Chris Larsen
>Priority: P2
> Labels: stale-P2
>
>
> Code to read from BigQuery that is causing the issue:
> {code:java}
> pipeline
> .apply(BigQueryIO
> .read(SchemaAndRecord::getRecord)
> .from(options.getTableRef())
> .withMethod(Method.DIRECT_READ)
> .withCoder(AvroCoder.of(schema)))
> {code}
> If we remove .withMethod(Method.DIRECT_READ) then there is no issue.
>
> The error is:
> {code:java}
> org.apache.beam.sdk.util.IllegalMutationException: PTransform
> BigQueryIO.TypedRead/Read(BigQueryStorageTableSource) mutated value
> {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f":
> 52.0, "sample_time": 1564412307969368, "humidity": 74.3} after it was output
> (new value was {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0,
> "temperature_f": 52.0, "sample_time": 1564412360458615, "humidity": 74.7}).
> Values must not be mutated in any way after being output.
> at
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit
> (ImmutabilityCheckingBundleFactory.java:134)
> at org.apache.beam.runners.direct.EvaluationContext.commitBundles
> (EvaluationContext.java:210)
> at org.apache.beam.runners.direct.EvaluationContext.handleResult
> (EvaluationContext.java:151)
> at
> org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult
> (QuiescenceDriver.java:262)
> at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle
> (DirectTransformExecutor.java:189)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run
> (DirectTransformExecutor.java:126)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748)
> Caused by: org.apache.beam.sdk.util.IllegalMutationException: Value
> {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f":
> 52.0, "sample_time": 1564412307969368, "humidity": 74.3} mutated illegally,
> new value was {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0,
> "temperature_f": 52.0, "sample_time": 1564412360458615, "humidity": 74.7}.
> Encoding was
> AiZycGktcnBpMC10aGVybW9zdGF0AgAAADRAAgAAAEpAArDVsP7jtMcFAjMzMzMzk1JA,
> now
> AiZycGktcnBpMC10aGVybW9zdGF0AgAAADRAAgAAAEpAAu6FuLDktMcFAs3MzMzMrFJA.
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation
> (MutationDetectors.java:153)
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions
> (MutationDetectors.java:148)
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified
> (MutationDetectors.java:123)
> at
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit
> (ImmutabilityCheckingBundleFactory.java:124)
> at org.apache.beam.runners.direct.EvaluationContext.commitBundles
> (EvaluationContext.java:210)
> at org.apache.beam.runners.direct.EvaluationContext.handleResult
> (EvaluationContext.java:151)
> at
> org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult
> (QuiescenceDriver.java:262)
> at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle
> (DirectTransformExecutor.java:189)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run
> (DirectTransformExecutor.java:126)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4729) Conditionally propagate local GCS credentials to locally spawned docker images.
[ https://issues.apache.org/jira/browse/BEAM-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123269#comment-17123269 ] Beam JIRA Bot commented on BEAM-4729: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Conditionally propagate local GCS credentials to locally spawned docker > images. > --- > > Key: BEAM-4729 > URL: https://issues.apache.org/jira/browse/BEAM-4729 > Project: Beam > Issue Type: Task > Components: sdk-java-harness >Reporter: Robert Bradshaw >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6441) cut_release_branch.sh should not push to master without verification and a PR
[ https://issues.apache.org/jira/browse/BEAM-6441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6441: Labels: stale-P2 (was: ) > cut_release_branch.sh should not push to master without verification and a PR > - > > Key: BEAM-6441 > URL: https://issues.apache.org/jira/browse/BEAM-6441 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently, the cut_release_branch.sh does many things: > - Edits files in place to update the version > - Makes a local commit > - Pushing the local commit to master > - Creates a new branch > - Edits files in place to update the version > - Pushes the release branch > I think all of this except the push to master are OK. It is possible that we > have something - website, examples, new places where the version is > hardcoded, etc, that get broken in this process. Moving from x-SNAPSHOT to > (x+1)-SNAPSHOT is easy to do in a pull request and safe. The release branch > creation does not need to be synchronized with this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7292) Upgrade hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/BEAM-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122824#comment-17122824 ] Beam JIRA Bot commented on BEAM-7292: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Upgrade hadoop-mapreduce-client-core > > > Key: BEAM-7292 > URL: https://issues.apache.org/jira/browse/BEAM-7292 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7570) withCustomGcsTempLocation should also be implemented for BigQueryIO.Read
[ https://issues.apache.org/jira/browse/BEAM-7570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7570: Labels: stale-P2 (was: ) > withCustomGcsTempLocation should also be implemented for BigQueryIO.Read > > > Key: BEAM-7570 > URL: https://issues.apache.org/jira/browse/BEAM-7570 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Aaron Liblong >Priority: P2 > Labels: stale-P2 > > A function in BigQueryIO.Write called withCustomGcsTempLocation allows > specification at template execution time of a GCS location used by BigQuery > to write temp files. BigQuery also needs to write temp files for _read_ > operations, and therefore this function should be available in > BigQueryIO.Read. > This issue blocks the ability to deploy a template with BigQuery read ops to > an environment where users (who will execute the template) have only read > access. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6015) Uber task for Portable Flink scalability
[ https://issues.apache.org/jira/browse/BEAM-6015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6015: Labels: stale-P2 (was: ) > Uber task for Portable Flink scalability > > > Key: BEAM-6015 > URL: https://issues.apache.org/jira/browse/BEAM-6015 > Project: Beam > Issue Type: Task > Components: java-fn-execution, runner-flink >Reporter: Ankur Goenka >Priority: P2 > Labels: stale-P2 > > Task to track scalability issues with portable flink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7957) Warn at job submit time if a step is named with a / or empty in DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7957: Labels: stale-P2 (was: ) > Warn at job submit time if a step is named with a / or empty in DataflowRunner > -- > > Key: BEAM-7957 > URL: https://issues.apache.org/jira/browse/BEAM-7957 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: David Yan >Priority: P2 > Labels: stale-P2 > > When a job with an empty step name or a step name that has a "/" in it, it > quietly breaks the job graph in the Dataflow UI. We should at least warn the > user at job submit time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4910) Beam Dependency Update Request: org.elasticsearch
[ https://issues.apache.org/jira/browse/BEAM-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4910: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.elasticsearch > - > > Key: BEAM-4910 > URL: https://issues.apache.org/jira/browse/BEAM-4910 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:24:36.721462 > Please review and upgrade the org.elasticsearch to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5735) Contributor Guide Improvements
[ https://issues.apache.org/jira/browse/BEAM-5735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123090#comment-17123090 ] Beam JIRA Bot commented on BEAM-5735: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Contributor Guide Improvements > -- > > Key: BEAM-5735 > URL: https://issues.apache.org/jira/browse/BEAM-5735 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Scott Wegner >Priority: P2 > Labels: stale-P2 > Time Spent: 6h 50m > Remaining Estimate: 0h > > This is a wish-list for improvements to the Beam contributor guide. > Many thanks to [~rohdesam] for the feedback which helped shape this list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7610) SELECT COALESCE(...) isn't inferred as non-nullable field
[
https://issues.apache.org/jira/browse/BEAM-7610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122771#comment-17122771
]
Beam JIRA Bot commented on BEAM-7610:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> SELECT COALESCE(...) isn't inferred as non-nullable field
> -
>
> Key: BEAM-7610
> URL: https://issues.apache.org/jira/browse/BEAM-7610
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.13.0
>Reporter: Gleb Kanterov
>Priority: P2
> Labels: stale-P2
>
> In Calcite, Coalesce is described as:
> {code}
> ReturnTypes.cascade(ReturnTypes.LEAST_RESTRICTIVE,
> SqlTypeTransforms.LEAST_NULLABLE)
> {code}
> However, giving non-null constant as an argument doesn't result in a
> non-nullable expression:
> {code}
> Schema inputSchema = Schema.of(
> Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true)));
> PCollection input = p.apply(Create.of(ImmutableList.of())
> .withCoder(SchemaCoder.of(inputSchema)));
> Schema outputSchema = input
> .apply(SqlTransform.query("SELECT COALESCE(name, 'unknown') as name
> FROM PCOLLECTION"))
> .getSchema();
> assertEquals(
> Schema.builder().addStringField("name").build(),
> outputSchema);
> {code}
> Not sure if it's a problem in Calcite or Beam SQL.
> There are no other functions that can be used to produce a non-nullable field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5751) Beam Dependency Update Request: org.eclipse.jetty
[ https://issues.apache.org/jira/browse/BEAM-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123066#comment-17123066 ] Beam JIRA Bot commented on BEAM-5751: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.eclipse.jetty > - > > Key: BEAM-5751 > URL: https://issues.apache.org/jira/browse/BEAM-5751 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-15 12:13:16.511863 > - > Please consider upgrading the dependency org.eclipse.jetty. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7858) Portable wordcount tests must verify that output was created and is correct.
[ https://issues.apache.org/jira/browse/BEAM-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7858: Labels: portability stale-P2 (was: portability) > Portable wordcount tests must verify that output was created and is correct. > - > > Key: BEAM-7858 > URL: https://issues.apache.org/jira/browse/BEAM-7858 > Project: Beam > Issue Type: Bug > Components: runner-flink, runner-spark, sdk-py-harness, testing >Reporter: Valentyn Tymofieiev >Priority: P2 > Labels: portability, stale-P2 > > Currently we have a TODO: > https://github.com/apache/beam/blob/8f6247ef2382c8fd321e23435bbaf401cdb05b50/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1920, > and it causes bugs to go unnoticed, see: > https://issues.apache.org/jira/browse/BEAM-7857 > cc: [~altay] [~angoenka] [~ibzib] [~robertwb] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6629) Make sure Java Apache Beam Logs properly include step names.
[ https://issues.apache.org/jira/browse/BEAM-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6629: Labels: stale-P2 (was: ) > Make sure Java Apache Beam Logs properly include step names. > > > Key: BEAM-6629 > URL: https://issues.apache.org/jira/browse/BEAM-6629 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > I've noticed that the DataflowWorkerLoggingHandler grabs the current > execution state and extracts the step name from it. > > However, in the Java SDK > ExecutionState does not implement step names, I left that logic only in > DataflowExecutionState and omitted it from the base class. Thus when the Java > SDK uses its SimpleExecutionState, it will not receive these step names and > log them properly. > @swegner, can you confirm that this is the behaviour we want, for Java Beam > SDKs to also include their step name in the logs, as used by the > DatafloWorkerLoggingHandler? Java SDK code will end up using this handler as > well, in portability? > [https://github.com/apache/beam/blob/52e73282223980fc0df9fbdbeddb2abb24d6600e/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java#L148] > > Also, it may matter for dataflow the specific type of step name being used. > (initial, optomized, user name ,etc.). The full name context will definetly > not be available in the beam java SDK, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4923) Beam Dependency Update Request: net.ltgt.gradle
[ https://issues.apache.org/jira/browse/BEAM-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123231#comment-17123231 ] Beam JIRA Bot commented on BEAM-4923: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: net.ltgt.gradle > --- > > Key: BEAM-4923 > URL: https://issues.apache.org/jira/browse/BEAM-4923 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:25:40.041774 > Please review and upgrade the net.ltgt.gradle to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7180) Consider having @Validation.Required check to see if a field is set or not in PipelineOptions
[ https://issues.apache.org/jira/browse/BEAM-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122839#comment-17122839 ] Beam JIRA Bot commented on BEAM-7180: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Consider having @Validation.Required check to see if a field is set or not in > PipelineOptions > - > > Key: BEAM-7180 > URL: https://issues.apache.org/jira/browse/BEAM-7180 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Priority: P2 > Labels: stale-P2 > > Consider checking to see if the field is unset and fail validation if it is > unset. > Original thread: > https://lists.apache.org/thread.html/ead403b079bcf9c234564e294b14484f7e37e2ace598e44cd79867e2@%3Cdev.beam.apache.org%3E > This could be considered a backwards incompatible change. > Related to BEAM-2261. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4812) Use Beam schema in ParquetIO
[ https://issues.apache.org/jira/browse/BEAM-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4812: Labels: stale-P2 (was: ) > Use Beam schema in ParquetIO > > > Key: BEAM-4812 > URL: https://issues.apache.org/jira/browse/BEAM-4812 > Project: Beam > Issue Type: Bug > Components: io-java-parquet >Reporter: Lukasz Gajowy >Priority: P2 > Labels: stale-P2 > > It would be better if we eliminate the need for avro.schema (infer it?/obtain > it from PCollection?) and use org.apache.beam.sdk.schemas.Schema instead. > Link to discussion on user@: > [https://lists.apache.org/thread.html/1d270884aa9e6d7952857|https://lists.apache.org/thread.html/1d270884aa9e6d7952857203522f67dea22195edca631df33f7c054d@%3Cuser.beam.apache.org%3E%C2%A0] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4581) Python SDK does not correctly set the serialized_fn of WindowInto when translating to the Dataflow pipeline description
[
https://issues.apache.org/jira/browse/BEAM-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123299#comment-17123299
]
Beam JIRA Bot commented on BEAM-4581:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Python SDK does not correctly set the serialized_fn of WindowInto when
> translating to the Dataflow pipeline description
> ---
>
> Key: BEAM-4581
> URL: https://issues.apache.org/jira/browse/BEAM-4581
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Luke Cwik
>Priority: P2
> Labels: portability, stale-P2
>
> serialzed_fn at the bottom should be reference to the pipeline proto
> ptransform id representing this function.
> {code:java}
> {
> "kind": "ParallelDo",
> "name": "s4",
> "properties": {
> "display_data": [
> {
> "key": "fn",
> "label": "Transform Function",
> "namespace": "apache_beam.transforms.core.WindowInto",
> "shortValue": "WindowIntoFn",
> "type": "STRING",
> "value": "apache_beam.transforms.core.WindowIntoFn"
> }
> ],
> "non_parallel_inputs": {},
> "output_info": [
> {
> "encoding": {
> "@type": "kind:windowed_value",
> "component_encodings": [
> {
> "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> "component_encodings": [
> {
> "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> "component_encodings": []
> },
> {
> "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> "component_encodings": []
> }
> ],
> "is_pair_like": true
> },
> {
> "@type": "kind:global_window"
> }
> ],
> "is_wrapper": true
> },
> "output_name": "out",
> "user_name": "assert_that/WindowInto(WindowIntoFn).out"
> }
> ],
> "parallel_input": {
> "@type": "OutputReference",
> "output_name": "out",
> "step_name": "s2"
> },
> "serialized_fn": "",
> "user_name": "assert_that/WindowInto(WindowIntoFn)"
> }
> },
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4863) Implement consistentWithEquals/structuralValue on FullWindowedValueCoder
[ https://issues.apache.org/jira/browse/BEAM-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123248#comment-17123248 ] Beam JIRA Bot commented on BEAM-4863: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Implement consistentWithEquals/structuralValue on FullWindowedValueCoder > > > Key: BEAM-4863 > URL: https://issues.apache.org/jira/browse/BEAM-4863 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Priority: P2 > Labels: stale-P2 > Time Spent: 1h 50m > Remaining Estimate: 0h > > Implementing *consistentWithEquals*/*structuralValue* boosts significantly > the performance of using these values in comparison operations since it > doesn't require encoding the values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5335) [SQL] Output schema is not set incorrectly
[
https://issues.apache.org/jira/browse/BEAM-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-5335:
Labels: stale-P2 (was: )
> [SQL] Output schema is not set incorrectly
> --
>
> Key: BEAM-5335
> URL: https://issues.apache.org/jira/browse/BEAM-5335
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Anton Kedin
>Priority: P2
> Labels: stale-P2
>
> *From:
> https://stackoverflow.com/questions/52181795/how-do-i-get-an-output-schema-for-an-apache-beam-sql-query
> :*
> I've been playing with the Beam SQL DSL and I'm unable to use the output from
> a query without providing a code that's aware of the output schema manually.
> Can I infer the output schema rather than hardcoding it?
> Neither the walkthrough or the examples actually use the output from a query.
> I'm using Scio rather than the plain Java API to keep the code relatively
> readable and concise, I don't think that makes a difference for this question.
> Here's an example of what I mean.
> Given an input schema inSchema and some data source that is mapped onto a Row
> as follows: (in this example, Avro-based, but again, I don't think that
> matters):
> {code}
> sc.avroFile[Foo](args("input"))
>.map(fooToRow)
>.setCoder(inSchema.getRowCoder)
>.applyTransform(SqlTransform.query("SELECT COUNT(1) FROM PCOLLECTION"))
>.saveAsTextFile(args("output"))
> {code}
> Running this pipeline results in a KryoException as follows:
> {code}
> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> Serialization trace:
> fieldIndices (org.apache.beam.sdk.schemas.Schema)
> schema (org.apache.beam.sdk.values.RowWithStorage)
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> {code}
> However, inserting a RowCoder matching the SQL output, in this case a single
> count int column:
> {code}
>...snip...
>.applyTransform(SqlTransform.query("SELECT COUNT(1) FROM PCOLLECTION"))
>.setCoder(Schema.builder().addInt64Field("count").build().getRowCoder)
>.saveAsTextFile(args("output"))
> {code}
> Now the pipeline runs just fine.
> Having to manually tell the pipeline how to encode the SQL output seems
> unnecessary, given that we specify the input schema/coder(s) and a query. It
> seems to me that we should be able to infer the output schema from that - but
> I can't see how, other than maybe using Calcite directly?
> Before raising a ticket on the Beam Jira, I thought I'd check I wasn't
> missing something obvious!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7614) Event-time timers seem to sometimes fire multiple times on dataflow + streaming engine
[
https://issues.apache.org/jira/browse/BEAM-7614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122769#comment-17122769
]
Beam JIRA Bot commented on BEAM-7614:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Event-time timers seem to sometimes fire multiple times on dataflow +
> streaming engine
> --
>
> Key: BEAM-7614
> URL: https://issues.apache.org/jira/browse/BEAM-7614
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Steve Niemitz
>Priority: P2
> Labels: stale-P2
>
> This is kind of hard to reproduce, but I've seen it happen a few times in the
> wild now.
> We have a DoFn that sets an event-time timer at window.maxTimestamp, the
> timer callback does something like:
> {code:java}
> def onWindowClose(
> @StateId(...) key: ValueState[K],
> @StateId(...) values: CombiningState[V],
> out: OutputReceiver[O],
> ...
> ) {
>
> val k = key.read()
> val values = values.read()
> out.output(KV.of(k, values)
> key.clear()
> values.clear()
> }{code}
> Essentially, keep track of the key, accumulate values seen in a window, and
> emit them at the end of the window.
> ProcessElement is pretty simple as well:
> {code:java}
> def processElement(
> ctx: ProcessContext,
> @StateId(...) key: ValueState[K],
> @StateId(...) values: CombiningState[V],
> ...
> ) {
> key.write(ctx.element().getKey())
> value.add(ctx.element().getValue())
> timer.set(window.maxTimestamp())
> }{code}
> However, *ONLY* when running on streaming engine (this doesn't happen
> otherwise), I'll see cases where the onWindowClose timer fires with a null
> key, and empty values.
> This can only happen if the timer fired twice, since it wouldn't have been
> set if no elements had arrived, and if late data had arrived, it would have
> set the key (and added to the combining state). Also, we never have late
> date in our pipeline.
> An interesting other thing I noticed is that these "phantom firings" seem to
> happen ~10-15 minutes _AFTER_ the window closes.
> Again, its pretty rate, we'll have millions of keys in a window, and I'll
> only see the error happen every few hours (with hourly windows).
> Let me know if I can clarify anything else!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7809) TextIO.Write silently fails in python when writing to GCS streaming.
[ https://issues.apache.org/jira/browse/BEAM-7809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122738#comment-17122738 ] Beam JIRA Bot commented on BEAM-7809: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > TextIO.Write silently fails in python when writing to GCS streaming. > > > Key: BEAM-7809 > URL: https://issues.apache.org/jira/browse/BEAM-7809 > Project: Beam > Issue Type: New Feature > Components: io-py-gcp >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > It would be better if it raised an exception, then the user would be aware of > the issue. Right now it silently drops the data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4538) readAll for BigQuery IO
[ https://issues.apache.org/jira/browse/BEAM-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4538: Labels: stale-P2 (was: ) > readAll for BigQuery IO > --- > > Key: BEAM-4538 > URL: https://issues.apache.org/jira/browse/BEAM-4538 > Project: Beam > Issue Type: Wish > Components: io-java-gcp >Reporter: Ahmet Altay >Priority: P2 > Labels: stale-P2 > > Customer reported: > """ > BigQueryIO.readTableRows() does not support reading partitions specified by > side inputs; the only way to select partitions is to know them ahead of time > and pass them in on the command line in PipelineOptions for selection in a > WHERE clause. > > Ideally we'd have something like a readAll() transform, like we have for > TextIO, JdbcIO etc. that allows the reading configuration to be dynamic in a > sense. > """ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6707) TextIO.Write appear success but request not sent to Google Cloud Storage
[
https://issues.apache.org/jira/browse/BEAM-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-6707:
Labels: stale-P2 (was: )
> TextIO.Write appear success but request not sent to Google Cloud Storage
>
>
> Key: BEAM-6707
> URL: https://issues.apache.org/jira/browse/BEAM-6707
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.8.0
> Environment: Google Cloud Dataflow and Google Cloud Storage
>Reporter: Wei Cheng A
>Priority: P2
> Labels: stale-P2
>
> Google Cloud Dataflow is being used to run Apache Beam job.
> from the Dataflow log, the file operation appear to be success.
> Will copy temporary file FileResult{tempFilename=gs://xx, shard=0,
> window=org.apache.beam.sdk.transforms.windowing.GlobalWindow@x,
> paneInfo=PaneInfo{isFirst=true, isLast=true, timing=ON_TIME, index=0,
> onTimeIndex=0}} to final location gs:///20190211.csv
> But when I checked GCS and its log, there was no put or post request during
> that time.
> This issue happened intermittently. The file is copied successfully
> sometimes, after retry.
> I have checked the relevant Beam source code
> https://github.com/apache/beam/blob/c96b096b77c324b886ab94aebcf320976002c0d4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L763
> and
> https://github.com/apache/beam/blob/c96b096b77c324b886ab94aebcf320976002c0d4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L304
> seems like in rename() method, there are multiple conditions that the method
> would return without Exception and appear as "success" in log.
> Is there any bug in these Beam methods or I should check for error in my code?
> {code:java}
> TextIO.write().withoutSharding().to(options.getOutFilePath()));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-5075) Please add OWASP Dependency Check to the build
[ https://issues.apache.org/jira/browse/BEAM-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5075: Labels: build easy-fix security stale-P2 (was: build easy-fix security) > Please add OWASP Dependency Check to the build > -- > > Key: BEAM-5075 > URL: https://issues.apache.org/jira/browse/BEAM-5075 > Project: Beam > Issue Type: New Feature > Components: build-system >Affects Versions: 3.0.0, 2.6.0, 2.7.0 > Environment: All development, build, test, environments. >Reporter: Albert Baker >Priority: P2 > Labels: build, easy-fix, security, stale-P2 > Original Estimate: 1h > Remaining Estimate: 1h > > Please add OWASP Dependency Check to the build (pom.xml). OWASP DC makes an > outbound REST call to MITRE Common Vulnerabilities & Exposures (CVE) to > perform a lookup for each dependant .jar to list any/all known > vulnerabilities for each jar. This step is needed because a manual MITRE CVE > lookup/check on the main component does not include checking for > vulnerabilities in components or in dependant libraries. > OWASP Dependency check : > https://www.owasp.org/index.php/OWASP_Dependency_Check has plug-ins for most > Java build/make types (ant, maven, ivy, gradle). > Also, add the appropriate command to the nightly build to generate a report > of all known vulnerabilities in any/all third party libraries/dependencies > that get pulled in. example : mvn -Powasp -Dtest=false -DfailIfNoTests=false > clean aggregate > Generating this report nightly/weekly will help inform the project's > development team if any dependant libraries have a reported known > vulnerailities. Project teams that keep up with removing vulnerabilities on > a weekly basis will help protect businesses that rely on these open source > componets. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6644) Beam Dependency Update Request: com.github.ben-manes.versions
[ https://issues.apache.org/jira/browse/BEAM-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122928#comment-17122928 ] Beam JIRA Bot commented on BEAM-6644: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.github.ben-manes.versions > - > > Key: BEAM-6644 > URL: https://issues.apache.org/jira/browse/BEAM-6644 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-02-11 12:12:16.464526 > - > Please consider upgrading the dependency > com.github.ben-manes.versions. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4506) PTransform input_types hint with potential empty input
[ https://issues.apache.org/jira/browse/BEAM-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123317#comment-17123317 ] Beam JIRA Bot commented on BEAM-4506: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > PTransform input_types hint with potential empty input > -- > > Key: BEAM-4506 > URL: https://issues.apache.org/jira/browse/BEAM-4506 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: P2 > Labels: stale-P2 > > User reported issue: > ... PTransform class that takes a pcollection of Dict[str, numpy.ndarray] as > input, and I added the annotation of > @beam.typehints.with_input_types(beam.typehints.Dict[str, numpy.ndarray]). > However, this causes error to empty dict input, and the error message is > """Input type hint violation at _TopKStatsGeneratorImpl: expected Dict[str, > ndarray], got Dict[Union[], Union[]] """ > From the user report it seems like our type hints could be more flexible in > what they accept for empty dictionaries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6449) Create PostCommit smoke test suite (besides phrase triggered one)
[ https://issues.apache.org/jira/browse/BEAM-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122960#comment-17122960 ] Beam JIRA Bot commented on BEAM-6449: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Create PostCommit smoke test suite (besides phrase triggered one) > - > > Key: BEAM-6449 > URL: https://issues.apache.org/jira/browse/BEAM-6449 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Lukasz Gajowy >Priority: P2 > Labels: stale-P2, triaged > > Load tests are very huge and consume lots of resources. In order to avoid > unnecessary runs on not failing code, we should have a small post-commit > variant that would validate if everything works well. If run post commit (on > every commit) such smoke tests will give us time to fix build/runtime/other > errors before actual load test suites are run. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4938) Beam Dependency Update Request: io.grpc
[ https://issues.apache.org/jira/browse/BEAM-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4938: Labels: stale-P2 (was: ) > Beam Dependency Update Request: io.grpc > --- > > Key: BEAM-4938 > URL: https://issues.apache.org/jira/browse/BEAM-4938 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > 2018-07-25 20:26:53.651877 > Please review and upgrade the io.grpc to the latest version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8655) Strengthen trigger transcript tests
[ https://issues.apache.org/jira/browse/BEAM-8655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8655: Labels: stale-P2 (was: ) > Strengthen trigger transcript tests > --- > > Key: BEAM-8655 > URL: https://issues.apache.org/jira/browse/BEAM-8655 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Robert Bradshaw >Priority: P2 > Labels: stale-P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > We should run them with multiple keys, using combiners as well as > group-by-key, and when possible in batch mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4915) Beam Dependency Update Request: com.google.errorprone
[ https://issues.apache.org/jira/browse/BEAM-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123235#comment-17123235 ] Beam JIRA Bot commented on BEAM-4915: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.google.errorprone > - > > Key: BEAM-4915 > URL: https://issues.apache.org/jira/browse/BEAM-4915 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > 2018-07-25 20:25:03.527143 > Please review and upgrade the com.google.errorprone to the latest > version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7963) Unnesting with large schema causes error
[
https://issues.apache.org/jira/browse/BEAM-7963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7963:
Labels: stale-P2 (was: )
> Unnesting with large schema causes error
>
>
> Key: BEAM-7963
> URL: https://issues.apache.org/jira/browse/BEAM-7963
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.16.0
>Reporter: Sahith Nallapareddy
>Priority: P2
> Labels: stale-P2
>
> query: select id from table a, UNNEST(a.nested_field) this causes the
> following error if table a has a relatively large schema, especially with
> either Row> or Array> (nested repeated records or
> records with repeated records)
>
>
> {noformat}
> Aug 13, 2019 10:17:01 AM
> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner convertToBeamRel
> INFO: SQL:
> SELECT `a`.`artist_gid`
> FROM `beam`.`endpoint1` AS `a`,
> UNNEST(`a`.`genre`.`genres`) AS `genres`
> Aug 13, 2019 10:17:01 AM
> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner convertToBeamRel
> INFO: SQLPlan>
> LogicalProject(artist_gid=[$0])
> LogicalCorrelate(correlation=[$cor0], joinType=[inner],
> requiredColumns=[{87}])
> LogicalProject(artist_gid=[$0], artist_uri=[$1], date=[$2], id=[$3.id],
> gid=[$3.gid], name=[$3.name], redirect=[$3.redirect],
> fuzzyname=[$3.fuzzyname], inserted=[$3.inserted], echo_nest_artists=[$4],
> id10=[$5.id], similars=[$5.similars], gid12=[$6.gid], version=[$6.version],
> vector=[$6.vector], value=[$7.value], gid16=[$7.gid], domains=[$7.domains],
> playlist_adds=[$8], uri=[$9.uri], gid20=[$9.gid], region=[$9.region],
> popularity_raw=[$9.popularity_raw],
> popularity_normalized=[$9.popularity_normalized], percentile=[$9.percentile],
> rank=[$9.rank], popularity_regional=[$10], gid27=[$11.gid],
> artist_name=[$11.artist_name], bios=[$11.bios],
> ancestor_artists=[$11.ancestor_artists],
> descendant_artists=[$11.descendant_artists],
> asserted_similars=[$11.asserted_similars], tags=[$11.tags],
> genres=[$11.genres], members=[$11.members], members_past=[$11.members_past],
> meanings=[$11.meanings], country=[$11.country],
> voted_descriptions=[$11.voted_descriptions], years_active=[$11.years_active],
> amazon_urls=[$11.amazon_urls], itunes_urls=[$11.itunes_urls],
> lastfm_urls=[$11.lastfm_urls], facebook_urls=[$11.facebook_urls],
> urbandictionary_urls=[$11.urbandictionary_urls],
> wikipedia_urls=[$11.wikipedia_urls],
> twitter_screennames=[$11.twitter_screennames], categories=[$11.categories],
> childrens=[$11.category.childrens], classical=[$11.category.classical],
> curated=[$11.category.curated], deceptive=[$11.category.deceptive],
> generic=[$11.category.generic], inactive=[$11.category.inactive],
> karaoke=[$11.category.karaoke], non_artist=[$11.category.non_artist],
> soundalike=[$11.category.soundalike], unpreferred=[$11.category.unpreferred],
> offensive=[$11.category.offensive],
> do_not_recommend=[$11.category.do_not_recommend],
> do_not_support=[$11.category.do_not_support],
> pass_on_programming=[$11.category.pass_on_programming],
> deceased=[$11.category.deceased], edited_terms=[$11.edited_terms],
> edited_text_terms=[$11.edited_text_terms],
> free_text_terms=[$11.free_text_terms], display_terms=[$11.display_terms],
> extra=[$11.extra], force_curated_sims=[$11.sims_curation.force_curated_sims],
> curated_sims_uris=[$11.sims_curation.curated_sims_uris],
> blacklisted_sims_uris=[$11.sims_curation.blacklisted_sims_uris],
> display_bios=[$11.display_bios], discogs_uri=[$11.discogs_uri],
> musicbrainz_uri=[$11.musicbrainz_uri], rovi_music_uri=[$11.rovi_music_uri],
> blocked_display_bio_providers=[$11.blocked_display_bio_providers],
> portrait=[$11.portrait], hidden_portraits=[$11.hidden_portraits],
> primary_portrait=[$11.primary_portrait], imdb_urls=[$11.imdb_urls],
> instagram_screennames=[$11.instagram_screennames],
> myspace_urls=[$11.myspace_urls], tumblr_screennames=[$11.tumblr_screennames],
> youtube_urls=[$11.youtube_urls], gid85=[$12.gid],
> acousticVector=[$12.acousticVector], gid87=[$13.gid], genres88=[$13.genres],
> extended_genres=[$13.extended_genres], currency=[$14])
> BeamIOSourceRel(table=[[beam, endpoint1]])
> Uncollect
> LogicalProject(genres=[$cor0.genres_88])
> LogicalValues(tuples=[[{ 0 }]])
> org.apache.beam.sdk.extensions.sql.impl.SqlConversionException: Unable to
> convert query select artist_gid from endpoint1 a, UNNEST(a.genre.genres) as
> genres
> at
> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:170)
> at
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103)
> at
> org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:124)
>
[jira] [Updated] (BEAM-5505) Disable Row flattening in Apache Calcite
[ https://issues.apache.org/jira/browse/BEAM-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5505: Labels: stale-P2 (was: ) > Disable Row flattening in Apache Calcite > > > Key: BEAM-5505 > URL: https://issues.apache.org/jira/browse/BEAM-5505 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6901) Beam Dependency Update Request: sqlline:sqlline
[ https://issues.apache.org/jira/browse/BEAM-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122877#comment-17122877 ] Beam JIRA Bot commented on BEAM-6901: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: sqlline:sqlline > --- > > Key: BEAM-6901 > URL: https://issues.apache.org/jira/browse/BEAM-6901 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-03-25 04:23:12.026437 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.7.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-04-01 12:15:11.859956 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.7.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-04-08 12:15:43.829522 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.7.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-04-15 12:35:59.407782 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.7.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-04-22 12:13:31.922817 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.7.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-12 22:55:03.099214 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.9.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-12 23:32:43.232674 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.9.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-15 19:48:15.961831 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.9.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-18 12:13:32.145836 > - > Please consider upgrading the dependency sqlline:sqlline. > The current version is 1.4.0. The latest version is 1.9.0 > cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. >
[jira] [Commented] (BEAM-6202) Gracefully handle exceptions when waiting for Dataflow job completion.
[
https://issues.apache.org/jira/browse/BEAM-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122994#comment-17122994
]
Beam JIRA Bot commented on BEAM-6202:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Gracefully handle exceptions when waiting for Dataflow job completion.
> --
>
> Key: BEAM-6202
> URL: https://issues.apache.org/jira/browse/BEAM-6202
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core, test-failures
>Reporter: Robert Bradshaw
>Priority: P2
> Labels: stale-P2
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> If there is an error when trying to contact the dataflow service in Python's
> Dataflow.poll_for_job_completion, we may exit the thread prematurely.
> A typical manifestation is: Dataflow Runner fails with:
> {noformat}
> AssertionError: Job did not reach to a terminal state after waiting
> indefinitely.
> {noformat}
> however job execution continues, and succeeds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4934) Beam Dependency Update Request: com.github.ben-manes
[ https://issues.apache.org/jira/browse/BEAM-4934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123225#comment-17123225 ] Beam JIRA Bot commented on BEAM-4934: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.github.ben-manes > > > Key: BEAM-4934 > URL: https://issues.apache.org/jira/browse/BEAM-4934 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > 2018-07-25 20:26:34.578768 > Please review and upgrade the com.github.ben-manes to the latest > version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4996) Beam Dependency Update Request: org.apache.qpid
[ https://issues.apache.org/jira/browse/BEAM-4996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4996: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.apache.qpid > --- > > Key: BEAM-4996 > URL: https://issues.apache.org/jira/browse/BEAM-4996 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:33:22.425496 > Please review and upgrade the org.apache.qpid to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5011) Beam Dependency Update Request: org.springframework
[ https://issues.apache.org/jira/browse/BEAM-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123203#comment-17123203 ] Beam JIRA Bot commented on BEAM-5011: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.springframework > --- > > Key: BEAM-5011 > URL: https://issues.apache.org/jira/browse/BEAM-5011 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > 2018-07-25 20:34:52.205280 > Please review and upgrade the org.springframework to the latest > version None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6313) runMobileGamingJavaDirect flaky FAILED_TO_LOAD_NATIVE_LIBRARY snappy-java
[
https://issues.apache.org/jira/browse/BEAM-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-6313:
Labels: stale-P2 (was: )
> runMobileGamingJavaDirect flaky FAILED_TO_LOAD_NATIVE_LIBRARY snappy-java
> -
>
> Key: BEAM-6313
> URL: https://issues.apache.org/jira/browse/BEAM-6313
> Project: Beam
> Issue Type: Bug
> Components: examples-java
>Reporter: Andrew Pilloud
>Priority: P2
> Labels: stale-P2
>
> {code}
> 03:06:09 java.io.FileNotFoundException:
> /tmp/groovy-generated-2817251679708289106-tmpdir/.m2/repository/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar
> (No such file or directory)
> 03:06:09 java.lang.NullPointerException
> 03:06:09 at
> org.xerial.snappy.SnappyLoader.extractLibraryFile(SnappyLoader.java:243)
> 03:06:09 at
> org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:355)
> 03:06:09 at
> org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:176)
> 03:06:09 at
> org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:154)
> 03:06:09 at org.xerial.snappy.Snappy.(Snappy.java:47)
> 03:06:09 at
> org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:97)
> 03:06:09 at
> org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:89)
> 03:06:09 at
> org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:79)
> 03:06:09 at
> org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:50)
> 03:06:09 at
> org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:100)
> 03:06:09 at
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:79)
> 03:06:09 at org.apache.beam.sdk.io.Read$Unbounded.(Read.java:129)
> 03:06:09 at org.apache.beam.sdk.io.Read$Unbounded.(Read.java:124)
> 03:06:09 at org.apache.beam.sdk.io.Read.from(Read.java:56)
> 03:06:09 at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource.expand(PubsubUnboundedSource.java:1285)
> 03:06:09 at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSource.expand(PubsubUnboundedSource.java:112)
> 03:06:09 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
> 03:06:09 at
> org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
> 03:06:09 at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
> 03:06:09 at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:715)
> 03:06:09 at
> org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Read.expand(PubsubIO.java:532)
> 03:06:09 at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
> 03:06:09 at
> org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
> 03:06:09 at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
> 03:06:09 at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
> 03:06:09 at
> org.apache.beam.examples.complete.game.LeaderBoard.main(LeaderBoard.java:205)
> 03:06:09 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 03:06:09 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 03:06:09 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 03:06:09 at java.lang.reflect.Method.invoke(Method.java:498)
> 03:06:09 at
> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
> 03:06:09 at java.lang.Thread.run(Thread.java:748)
> 03:06:09 [ERROR] Failed command
> 03:06:09 [ERROR] Failed to execute goal
> org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project
> word-count-beam: An exception occured while executing the Java class.
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null -> [Help 1]
> 03:06:09 [ERROR]
> 03:06:09 [ERROR] To see the full stack trace of the errors, re-run Maven with
> the -e switch.
> 03:06:09 [ERROR] Re-run Maven using the -X switch to enable full debug
> logging.
> 03:06:09 [ERROR]
> 03:06:09 [ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> 03:06:09 [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> 03:06:09 :beam-runners-direct-java:runMobileGamingJavaDirect (Thread[Task
> worker for ':' Thread 5,5,main]) completed. Took 5 mins 39.017 secs.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4702) After SQL GROUP BY the result should be globally windowed
[
https://issues.apache.org/jira/browse/BEAM-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123278#comment-17123278
]
Beam JIRA Bot commented on BEAM-4702:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> After SQL GROUP BY the result should be globally windowed
> -
>
> Key: BEAM-4702
> URL: https://issues.apache.org/jira/browse/BEAM-4702
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: P2
> Labels: stale-P2
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Beam SQL runs in two contexts:
> 1. As a PTransform in a pipeline. A PTransform operates on a PCollection,
> which is always implicitly windows and a PTransform should operate per-window
> so it automatically works on bounded and unbounded data. This only works if
> the query has no windowing operators, in which case the GROUP BY stuff> should operate per-window.
> 2. As a top-level shell that starts and ends with SQL. In the relational
> model there are no implicit windows. Calcite has some extensions for
> windowing, but they manifest (IMO correctly) as just items in the GROUP BY
> list. The output of the aggregation is "just rows" again. So it should be
> globally windowed.
> The problem is that this semantic fix makes it so we cannot join windowing
> stream subqueries. Because we don't have retractions, we only support
> GroupByKey-based equijoins over windowed streams, with the default trigger.
> _These joins implicitly also join windows_. For example:
> {code}
> JOIN(left.id = right.id)
> SELECT ... GROUP BY id, TUMBLE(1 hour)
> SELECT ... GROUP BY id, TUMBLE(1 hour)
> {code}
> Semantically, there may be a joined row for 1:00pm on the left and 10:00pm on
> the right. But by the time the right-hand row for 10:00pm shows up, the left
> one may be GC'd. So this is implicitly, but nondeterministically, joining on
> the window as well. Before this PR, we left the windowing strategies for left
> and right in place, and asserted that they matched.
> If we re-window into the global window always, there _are no windowed
> streams_ so you just can't do stream joins. The solution is probably to track
> which field of a stream is the window and allow joins which also explicitly
> express the equijoin over the window field.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8319) Errorprone 0.0.13 fails during JDK11 build
[
https://issues.apache.org/jira/browse/BEAM-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-8319:
Labels: stale-P2 (was: )
> Errorprone 0.0.13 fails during JDK11 build
> --
>
> Key: BEAM-8319
> URL: https://issues.apache.org/jira/browse/BEAM-8319
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Lukasz Gajowy
>Priority: P2
> Labels: stale-P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm using openjdk 1.11.02. After switching version to;
> {code:java}
> javaVersion = 11 {code}
> in BeamModule Plugin and running
> {code:java}
> ./gradlew clean build -p sdks/java/code -xtest {code}
> building fails. I was able to run errorprone after upgrading it but had
> problems with conflicting guava version. See more here:
> https://issues.apache.org/jira/browse/BEAM-5085
>
> Stacktrace:
> {code:java}
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task
> ':model:pipeline:compileJava'.
> at
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.accept(ExecuteActionsTaskExecuter.java:121)
> at
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.accept(ExecuteActionsTaskExecuter.java:117)
> at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:184)
> at
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:110)
> at
> org.gradle.api.internal.tasks.execution.ResolveIncrementalChangesTaskExecuter.execute(ResolveIncrementalChangesTaskExecuter.java:84)
> at
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:91)
> at
> org.gradle.api.internal.tasks.execution.FinishSnapshotTaskInputsBuildOperationTaskExecuter.execute(FinishSnapshotTaskInputsBuildOperationTaskExecuter.java:51)
> at
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:102)
> at
> org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionStateTaskExecuter.execute(ResolveBeforeExecutionStateTaskExecuter.java:74)
> at
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
> at
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:109)
> at
> org.gradle.api.internal.tasks.execution.ResolveBeforeExecutionOutputsTaskExecuter.execute(ResolveBeforeExecutionOutputsTaskExecuter.java:67)
> at
> org.gradle.api.internal.tasks.execution.StartSnapshotTaskInputsBuildOperationTaskExecuter.execute(StartSnapshotTaskInputsBuildOperationTaskExecuter.java:52)
> at
> org.gradle.api.internal.tasks.execution.ResolveAfterPreviousExecutionStateTaskExecuter.execute(ResolveAfterPreviousExecutionStateTaskExecuter.java:46)
> at
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:93)
> at
> org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:45)
> at
> org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:94)
> at
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
> at
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
> at
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
> at
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:63)
> at
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:49)
> at
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:46)
> at
> org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
> at
> org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
> at
> org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
> at
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
> at
>
[jira] [Updated] (BEAM-6564) GrpcWindmillServer has inconsistent synchronization
[ https://issues.apache.org/jira/browse/BEAM-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6564: Labels: stale-P2 (was: ) > GrpcWindmillServer has inconsistent synchronization > --- > > Key: BEAM-6564 > URL: https://issues.apache.org/jira/browse/BEAM-6564 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > > This bug filed because findbugs complained. There are many complaints about > inconsistent synchronization that deserve review, and suppression if they are > good to go. Grep for this issue in the codebase to make an attempt at fixing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4957) Beam Dependency Update Request: org.apache.hive
[ https://issues.apache.org/jira/browse/BEAM-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4957: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.apache.hive > --- > > Key: BEAM-4957 > URL: https://issues.apache.org/jira/browse/BEAM-4957 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > 2018-07-25 20:29:09.321877 > Please review and upgrade the org.apache.hive to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6450) ExpectedLogs with Maven
[
https://issues.apache.org/jira/browse/BEAM-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122959#comment-17122959
]
Beam JIRA Bot commented on BEAM-6450:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> ExpectedLogs with Maven
> ---
>
> Key: BEAM-6450
> URL: https://issues.apache.org/jira/browse/BEAM-6450
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Jose Fernandez
>Priority: P2
> Labels: stale-P2
>
> Hi!
> I'm using apache beam with Maven and in the pom.xml the dependency is
> {color:#e8bf6a}{color}{color:#e8bf6a}
> {color}org.apache.beam{color:#e8bf6a}{color}{color:#e8bf6a}
>
> {color}beam-sdks-java-core{color:#e8bf6a}{color}{color:#e8bf6a}
>
> {color}2.8.0{color:#e8bf6a}{color}{color:#e8bf6a}{color}
> I can use
> {color:#cc7832}import
> {color}org.apache.beam.sdk.testing.PAssert{color:#cc7832};{color}
> but i can not use
> import org.apache.beam.sdk.testing.ExpectedLogs;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-6442) Incomplete JobService API Semantics
[ https://issues.apache.org/jira/browse/BEAM-6442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6442: Labels: stale-P2 (was: ) > Incomplete JobService API Semantics > --- > > Key: BEAM-6442 > URL: https://issues.apache.org/jira/browse/BEAM-6442 > Project: Beam > Issue Type: Test > Components: beam-model >Affects Versions: 2.9.0 >Reporter: Sam Rohde >Priority: P2 > Labels: stale-P2 > > The JobService API (beam_job_api.proto) allows for the possibility of never > seeing messages or states with Get(State|Message)Stream. This is because the > Get(State|Message)Stream calls need to have the job id which can only be > obtained from the RunJobResponse. But in order to see all messages/states the > streams need to be opened before the job starts. > This is fine in Dataflow as the preparation_id == job_id, but this is not > true in Flink. > Fix is to modify the API to only keep a single id to be used between the > preparation/run APIs. > Consumers of the API will have to be modified to meet the new semantics. > Dev list thread > (https://lists.apache.org/thread.html/3ace7585278c0545185fa4bb8d6975283d5c48c097e1bb2c2e18b9a2@%3Cdev.beam.apache.org%3E) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5510) Records including datetime to be saved as DATETIME or TIMESTAMP in BigQuery
[
https://issues.apache.org/jira/browse/BEAM-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123132#comment-17123132
]
Beam JIRA Bot commented on BEAM-5510:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Records including datetime to be saved as DATETIME or TIMESTAMP in BigQuery
> ---
>
> Key: BEAM-5510
> URL: https://issues.apache.org/jira/browse/BEAM-5510
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Pascal Gula
>Priority: P2
> Labels: stale-P2
>
> When trying to write some row in BigQuery that include a python datetime
> object, the marshaling used to save a row in BigQuery is impossible.
> {code:java}
> File
> "/home/pascal/Wks/GitHub/PEAT-AI/Albatros/venv/local/lib/python2.7/site-packages/apache_beam/internal/gcp/json_value.py",
> line 124, in to_json_value
> raise TypeError('Cannot convert %s to a JSON value.' % repr(obj))
> TypeError: Cannot convert datetime.datetime(2018, 9, 25, 18, 57, 18, 108579)
> to a JSON value. [while running 'save/WriteToBigQuery']
> {code}
> However, this is something perfectly feasible, as `google-cloud-python`
> supports it since this issue has been solved:
> [https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2957]
> thanks to this pull request:
> [https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3426/files]
> As similar approach could be taken for the `json_value.py` helper.
> Is there any workaround that can be applied to solve this issue?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-5601) Dataflow runner should support custom windowfn for portability
[ https://issues.apache.org/jira/browse/BEAM-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5601: Labels: portability stale-P2 (was: portability) > Dataflow runner should support custom windowfn for portability > -- > > Key: BEAM-5601 > URL: https://issues.apache.org/jira/browse/BEAM-5601 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Reporter: Scott Wegner >Priority: P2 > Labels: portability, stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4906) Beam Dependency Update Request: org.apache.derby
[ https://issues.apache.org/jira/browse/BEAM-4906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123239#comment-17123239 ] Beam JIRA Bot commented on BEAM-4906: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.derby > > > Key: BEAM-4906 > URL: https://issues.apache.org/jira/browse/BEAM-4906 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Time Spent: 50m > Remaining Estimate: 0h > > 2018-07-25 20:24:06.236084 > Please review and upgrade the org.apache.derby to the latest version > None > > cc: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5426) Use both destination and TableDestination for BQ load job IDs
[ https://issues.apache.org/jira/browse/BEAM-5426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5426: Labels: stale-P2 (was: ) > Use both destination and TableDestination for BQ load job IDs > - > > Key: BEAM-5426 > URL: https://issues.apache.org/jira/browse/BEAM-5426 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: stale-P2 > > Currently we use TableDestination when creating a unique load job ID for a > destination: > [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryHelpers.java#L359] > > This can result in a data loss issue if a user returns the same > TableDestination for different destination IDs. I think we can prevent this > if we include both IDs in the BQ load job ID. > > CC: [~reuvenlax] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5574) Beam Dependency Update Request: com.esotericsoftware.kryo:kryo
[ https://issues.apache.org/jira/browse/BEAM-5574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123110#comment-17123110 ] Beam JIRA Bot commented on BEAM-5574: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.esotericsoftware.kryo:kryo > -- > > Key: BEAM-5574 > URL: https://issues.apache.org/jira/browse/BEAM-5574 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:31:47.316547 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:19:34.193424 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:13:27.165194 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:13:46.501041 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:18:21.444889 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:15:21.075777 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:15:21.110876 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:16:02.457341 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:15:03.643257 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is 2.24.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:15:31.902326 > - > Please consider upgrading the dependency > com.esotericsoftware.kryo:kryo. > The current version is 2.21. The latest version is
[jira] [Commented] (BEAM-6090) Beam Dependency Update Request: org.elasticsearch:elasticsearch
[ https://issues.apache.org/jira/browse/BEAM-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123010#comment-17123010 ] Beam JIRA Bot commented on BEAM-6090: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.elasticsearch:elasticsearch > --- > > Key: BEAM-6090 > URL: https://issues.apache.org/jira/browse/BEAM-6090 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-11-19 12:12:57.791447 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:12:02.158689 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:12:21.276206 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-10 12:14:41.860759 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-17 12:15:13.638673 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha1 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-31 15:21:29.050590 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-07 12:24:36.273791 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-14 12:13:14.211572 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-01-21 12:19:43.014366 > - > Please consider upgrading the dependency > org.elasticsearch:elasticsearch. > The current version is 6.4.0. The latest version is 7.0.0-alpha2 > cc: [~echauchot], [~timrobertson100], > Please
[jira] [Updated] (BEAM-6898) Beam Dependency Update Request: com.google.api.grpc
[ https://issues.apache.org/jira/browse/BEAM-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6898: Labels: stale-P2 (was: ) > Beam Dependency Update Request: com.google.api.grpc > --- > > Key: BEAM-6898 > URL: https://issues.apache.org/jira/browse/BEAM-6898 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-03-25 04:21:45.384683 > - > Please consider upgrading the dependency com.google.api.grpc. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5859) Improve Traceability of Pipeline translation
[
https://issues.apache.org/jira/browse/BEAM-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-5859:
Labels: portability portability-flink stale-P2 (was: portability
portability-flink)
> Improve Traceability of Pipeline translation
>
>
> Key: BEAM-5859
> URL: https://issues.apache.org/jira/browse/BEAM-5859
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Maximilian Michels
>Priority: P2
> Labels: portability, portability-flink, stale-P2
> Attachments: tfx.png, wordcount.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Users often ask how they can reason about the pipeline translation. The Flink
> UI display a confusingly large graph without any trace of the original Beam
> pipeline:
> WordCount:
> !wordcount.png!
> TFX:
> !tfx.png!
> Some aspects which make understanding these graphs hard:
> * Users don't know how the Runner maps Beam to Flink concepts
> * The UI is awfully slow / hangs when the pipeline is reasonable complex
> * The operator names seem to use {{transform.getUniqueName()}} which doesn't
> generate readable name
> * So called Chaining combines operators into a single operator which makes
> understanding which Beam concept belongs to which Flink concept even harder
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-6389) Sources using Metrics.counter fail with 'Can't pickle ... DelegatingCounter'
[
https://issues.apache.org/jira/browse/BEAM-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122968#comment-17122968
]
Beam JIRA Bot commented on BEAM-6389:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Sources using Metrics.counter fail with 'Can't pickle ... DelegatingCounter'
>
>
> Key: BEAM-6389
> URL: https://issues.apache.org/jira/browse/BEAM-6389
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.9.0
>Reporter: Matthew Willson
>Priority: P2
> Labels: stale-P2
>
> This fails under Python 2.7 but works fine under Python 3 – looks like an
> easy fix.
> I am reading from a beam.io.utils.CountingSource:
> {{beam.io.Read(beam_io_utils.CountingSource(NUM_ROWS))}}
> This fails under python 2.7 with:
> Python exception: Can't pickle 'apache_beam.metrics.metric.DelegatingCounter'>: attribute lookup
> apache_beam.metrics.metric.DelegatingCounter failed
> Looks like this is because DelegatingCounter is actually an inner class
> defined inside apache_beam.metrics.metric.Metric. Under Python 3 pickling can
> resolve it under its qualified name
> apache_beam.metrics.metric.Metric.DelegatingCounter, but not in Python 2. A
> fix would be to move it to a top-level class in that module.
> This also applies to other sources which use Metrics.counter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-4544) Add support for intellij to work with vendored artifacts generated as part of the Apache Beam build process
[ https://issues.apache.org/jira/browse/BEAM-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4544: Labels: stale-P2 (was: ) > Add support for intellij to work with vendored artifacts generated as part of > the Apache Beam build process > --- > > Key: BEAM-4544 > URL: https://issues.apache.org/jira/browse/BEAM-4544 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Luke Cwik >Priority: P2 > Labels: stale-P2 > > Intellij has difficulty resolving vendored artifacts such as > *org.apache.beam.vendor.protobuf.v3.com.google.protobuf.Message*. This > prevents the code editor for making reasonable substitutions, intellisense > from working, and tracing through source code during debugging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7587) Spark portable runner: Streaming mode
[ https://issues.apache.org/jira/browse/BEAM-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122773#comment-17122773 ] Beam JIRA Bot commented on BEAM-7587: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Spark portable runner: Streaming mode > - > > Key: BEAM-7587 > URL: https://issues.apache.org/jira/browse/BEAM-7587 > Project: Beam > Issue Type: Wish > Components: runner-spark >Reporter: Kyle Weaver >Priority: P2 > Labels: portability-spark, stale-P2 > > So far all work on the Spark portable runner has been in batch mode. This is > intended as an uber-issue for tracking progress on adding support for > streaming. > -It might be advantageous to wait for the structured streaming (non-portable) > runner to be completed (to some reasonable extent) before undertaking this, > rather than using the DStream API.- Since work on the structured streaming > runner is blocked by SPARK-26655, we should implement this using DStreams > instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7772) Stop using Perfkit Benchmarker tool in all tests
[ https://issues.apache.org/jira/browse/BEAM-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7772: Labels: stale-P2 (was: ) > Stop using Perfkit Benchmarker tool in all tests > > > Key: BEAM-7772 > URL: https://issues.apache.org/jira/browse/BEAM-7772 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Lukasz Gajowy >Priority: P2 > Labels: stale-P2 > Time Spent: 6h 50m > Remaining Estimate: 0h > > [Devlist thread > link|https://lists.apache.org/thread.html/dab1c093799248787e8b75e63b66d7389b594b649a4d9a4a5db1cfbb@%3Cdev.beam.apache.org%3E] > > Currently Python, IOIT and some Dataflow and Spark performance tests are > relying on Perfkit Benchmarker tool. Due to the reasons discussed on the > devlist it was decided to remove it from Beam's tests. > Problems that we face currently: > # Changes to Gradle tasks/build configuration in the Beam codebase have to > be reflected in Perfkit code. This required PRs to Perfkit which can last and > the tests break due to this sometimes (no change in Perfkit + change already > there in beam = incompatibility). This is what happened in PR 8919 (above), > # Can't run in Python3 (depends on python 2 only library like functools32), > # Black box testing which hard to collect pipeline related metrics, > # Measurement of run time is inaccurate, > # It offers relatively small elasticity in comparison with eg. Jenkins tasks > in terms of setting up the testing infrastructure (runners, databases). For > example, if we'd like to setup Flink runner, and reuse it in consequent tests > in one go, that would be impossible. We can easily do this in Jenkins. > Tests that use Perfkit: > # IO integration tests, > # Python performance tests, > # beam_PerformanceTests_Dataflow (disabled), > # beam_PerformanceTests_Spark (failing constantly - looks not maintained). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7614) Event-time timers seem to sometimes fire multiple times on dataflow + streaming engine
[
https://issues.apache.org/jira/browse/BEAM-7614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7614:
Labels: stale-P2 (was: )
> Event-time timers seem to sometimes fire multiple times on dataflow +
> streaming engine
> --
>
> Key: BEAM-7614
> URL: https://issues.apache.org/jira/browse/BEAM-7614
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Reporter: Steve Niemitz
>Priority: P2
> Labels: stale-P2
>
> This is kind of hard to reproduce, but I've seen it happen a few times in the
> wild now.
> We have a DoFn that sets an event-time timer at window.maxTimestamp, the
> timer callback does something like:
> {code:java}
> def onWindowClose(
> @StateId(...) key: ValueState[K],
> @StateId(...) values: CombiningState[V],
> out: OutputReceiver[O],
> ...
> ) {
>
> val k = key.read()
> val values = values.read()
> out.output(KV.of(k, values)
> key.clear()
> values.clear()
> }{code}
> Essentially, keep track of the key, accumulate values seen in a window, and
> emit them at the end of the window.
> ProcessElement is pretty simple as well:
> {code:java}
> def processElement(
> ctx: ProcessContext,
> @StateId(...) key: ValueState[K],
> @StateId(...) values: CombiningState[V],
> ...
> ) {
> key.write(ctx.element().getKey())
> value.add(ctx.element().getValue())
> timer.set(window.maxTimestamp())
> }{code}
> However, *ONLY* when running on streaming engine (this doesn't happen
> otherwise), I'll see cases where the onWindowClose timer fires with a null
> key, and empty values.
> This can only happen if the timer fired twice, since it wouldn't have been
> set if no elements had arrived, and if late data had arrived, it would have
> set the key (and added to the combining state). Also, we never have late
> date in our pipeline.
> An interesting other thing I noticed is that these "phantom firings" seem to
> happen ~10-15 minutes _AFTER_ the window closes.
> Again, its pretty rate, we'll have millions of keys in a window, and I'll
> only see the error happen every few hours (with hourly windows).
> Let me know if I can clarify anything else!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-4682) Integrate support for timers using the portability APIs into Dataflow
[ https://issues.apache.org/jira/browse/BEAM-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4682: Labels: portability stale-P2 (was: portability) > Integrate support for timers using the portability APIs into Dataflow > - > > Key: BEAM-4682 > URL: https://issues.apache.org/jira/browse/BEAM-4682 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Luke Cwik >Priority: P2 > Labels: portability, stale-P2 > > Consider using the code produced in BEAM-4658 to support timers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4719) Enhanced LIMIT support
[ https://issues.apache.org/jira/browse/BEAM-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4719: Labels: stale-P2 (was: ) > Enhanced LIMIT support > -- > > Key: BEAM-4719 > URL: https://issues.apache.org/jira/browse/BEAM-4719 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > > Currently, Beam SQL supports LIMIT in two ways: > 1. Within a query, the results are subject to LIMIT. This works. > 2. The shell knows to cancel a pipeline when the limit is reached, even if > there is unfinished unbounded data. > The canceling of a pipeline works via a basic pattern match against the query > execution plan, checking a few child nodes of the BeamEnumerableConverter for > a BeamSortRel without a collation. If it can figure out what the limit is for > the outermost query, then it will cancel the pipeline. > A more robust approach might be to use traits (or some other thorough > analysis) to see if there is a known size for the outermost query. This > would, for example, be unaffected by any number of layer of non-size-changing > transformations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6868) Flink runner supports Bundle Finalization
[ https://issues.apache.org/jira/browse/BEAM-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122887#comment-17122887 ] Beam JIRA Bot commented on BEAM-6868: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Flink runner supports Bundle Finalization > - > > Key: BEAM-6868 > URL: https://issues.apache.org/jira/browse/BEAM-6868 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: Boyuan Zhang >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6917) ClickHouseIOTest failing in Java Precommits
[
https://issues.apache.org/jira/browse/BEAM-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122872#comment-17122872
]
Beam JIRA Bot commented on BEAM-6917:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> ClickHouseIOTest failing in Java Precommits
> ---
>
> Key: BEAM-6917
> URL: https://issues.apache.org/jira/browse/BEAM-6917
> Project: Beam
> Issue Type: Improvement
> Components: io-java-clickhouse
>Reporter: Udi Meiri
>Priority: P2
> Labels: stale-P2
>
> ```
> Caused by: com.github.dockerjava.api.exception.NotFoundException:
> {"message":"No such image: zookeeper:3.4.13"}
> ```
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/4981/testReport/junit/org.apache.beam.sdk.io.clickhouse/ClickHouseIOTest/classMethod/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted
[ https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123252#comment-17123252 ] Beam JIRA Bot commented on BEAM-4847: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is > exhausted > --- > > Key: BEAM-4847 > URL: https://issues.apache.org/jira/browse/BEAM-4847 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Boyuan Zhang >Priority: P2 > Labels: stale-P2 > Time Spent: 3h 20m > Remaining Estimate: 0h > > Java PreCommit seems like flaky: > https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5825) Vendor kryo
[ https://issues.apache.org/jira/browse/BEAM-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5825: Labels: stale-P2 (was: ) > Vendor kryo > --- > > Key: BEAM-5825 > URL: https://issues.apache.org/jira/browse/BEAM-5825 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6226) Add a custom tool to allow custom logging to be added for failing calls made by the RetryHttpRequestInitalizer
[ https://issues.apache.org/jira/browse/BEAM-6226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6226: Labels: stale-P2 (was: ) > Add a custom tool to allow custom logging to be added for failing calls made > by the RetryHttpRequestInitalizer > -- > > Key: BEAM-6226 > URL: https://issues.apache.org/jira/browse/BEAM-6226 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4565) Hot key fanout should not distribute keys to all shards.
[ https://issues.apache.org/jira/browse/BEAM-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123306#comment-17123306 ] Beam JIRA Bot commented on BEAM-4565: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Hot key fanout should not distribute keys to all shards. > > > Key: BEAM-4565 > URL: https://issues.apache.org/jira/browse/BEAM-4565 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py-core >Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0 >Reporter: Robert Bradshaw >Priority: P2 > Labels: stale-P2 > Time Spent: 1h > Remaining Estimate: 0h > > The goal is to reduce the number of value sent to a single post-GBK worker. > If combiner lifting happens, each bundle will sends a single value per > sub-key, causing an N-fold blowup in shuffle data and N reducers with the > same amount of data to consume as the single reducer in the non-fanout case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6118) Maven dependency conflicts between google-cloud-pubsub and Apache Beam packages
[
https://issues.apache.org/jira/browse/BEAM-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-6118:
Labels: maven stale-P2 (was: maven)
> Maven dependency conflicts between google-cloud-pubsub and Apache Beam
> packages
> ---
>
> Key: BEAM-6118
> URL: https://issues.apache.org/jira/browse/BEAM-6118
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Chris
>Priority: P2
> Labels: maven, stale-P2
>
> (I'm using BDD terminology to phrase the issue)
>
> AS A DEVELOPER I want to be able to use the following Maven dependencies (or
> newer versions) in the same Java project inside Eclipse SO THAT I can
> subscribe to a Google pub/sub independently from the mechanism supplied by
> Apache Beam, and efficiently update a dataset linked to my Apache Beam
> dataflow.
> I want to update my linked dataset by accepting push data that reflects
> changes to its data, instead of having to poll for new data each time I
> process an item from an Apache Beam pipeline that subscribes to a different
> Google pub/sub dataset.
>
> {code:java}
>
> com.google.cloud.dataflow
> google-cloud-dataflow-java-sdk-all
> 2.5.0
>
>
> com.google.cloud
> google-cloud-pubsub
> 1.53.0
>
> {code}
>
> ACCEPTANCE CRITERIA
> No dependency conflict related to io.grpc:grpc-core:jar is caused when I run
> {code:java}
> mvn clean install{code}
> against the project.
> Please see here (I documented itas a GitHub issue against the
> google-cloud-pubsub library, however it's primarily a Beam Maven dependency
> issue):
>
> [https://github.com/googleapis/google-cloud-java/issues/4095]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-5750) Beam Dependency Update Request: javax.servlet:javax.servlet-api
[ https://issues.apache.org/jira/browse/BEAM-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5750: Labels: stale-P2 (was: ) > Beam Dependency Update Request: javax.servlet:javax.servlet-api > --- > > Key: BEAM-5750 > URL: https://issues.apache.org/jira/browse/BEAM-5750 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-15 12:13:14.253682 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:13:20.677351 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:17:03.954722 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:14:50.209888 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:14:50.401239 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:15:31.652290 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:14:29.396363 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:14:52.265627 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-10 12:17:21.331309 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-17 12:17:42.365110 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-31 15:24:01.296226 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The
[jira] [Commented] (BEAM-7756) Stack trace not useful in SQL dsl for Java
[
https://issues.apache.org/jira/browse/BEAM-7756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122750#comment-17122750
]
Beam JIRA Bot commented on BEAM-7756:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Stack trace not useful in SQL dsl for Java
> --
>
> Key: BEAM-7756
> URL: https://issues.apache.org/jira/browse/BEAM-7756
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Jules Testard
>Priority: P2
> Labels: stale-P2
>
> I am testing Apache Beam to be used in our company's environment. I could be
> missing something, but it seems the stack tracing is very limited, in
> particular when using the SQL dsl available for the Beam Java SDK. This makes
> troubleshooting more difficult.
>
> One minimal example shows it very well:
> [https://github.com/jtestard/beam-sql-stack-trace] (the pom.xml in this repo
> should detail all dependencies used, the JDK used is Java 8). This GitHub
> repo gives attempts perform two back-to-back SQL transforms, the first of
> which contains a mismatch between input and output schema. The stack trace,
> however, gives little indication of the problem and does not trace to which
> transform the problem originated in:
>
> {{org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.RuntimeException: CalcFn failed to evaluate: {}}
> {{ final org.apache.beam.sdk.values.Row current =
> (org.apache.beam.sdk.values.Row) c.element();}}
> {{
> c.output(org.apache.beam.sdk.values.Row.withSchema(outputSchema).addValue(current.getValue(0)).addValue(current.getValue(1)).build());}}
> {{ }}}{{at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)}}
> {{ at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)}}
> {{ at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)}}
> {{ at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)}}
> {{ at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)}}
> {{ at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)}}
> {{ at dummy.MainTest.testMain(MainTest.java:18)}}
> {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{ at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{ at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)}}
> {{ at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}}
> {{ at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)}}
> {{ at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}}
> {{ at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)}}
> {{ at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)}}
> {{ at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)}}
> {{ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)}}
> {{ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)}}
> {{ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)}}
> {{ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)}}
> {{ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)}}
> {{ at org.junit.runners.ParentRunner.run(ParentRunner.java:363)}}
> {{ at org.junit.runner.JUnitCore.run(JUnitCore.java:137)}}
> {{ at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)}}
> {{ at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)}}
> {{ at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)}}
> {{ at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)}}
> {{ Caused by: java.lang.RuntimeException: CalcFn failed to evaluate: {}}
> {{ final org.apache.beam.sdk.values.Row current =
> (org.apache.beam.sdk.values.Row) c.element();}}
> {{
> c.output(org.apache.beam.sdk.values.Row.withSchema(outputSchema).addValue(current.getValue(0)).addValue(current.getValue(1)).build());}}
> {{ }}}{{at
>
[jira] [Commented] (BEAM-5582) Beam Dependency Update Request: org.apache.qpid:proton-j
[ https://issues.apache.org/jira/browse/BEAM-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123105#comment-17123105 ] Beam JIRA Bot commented on BEAM-5582: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.qpid:proton-j > > > Key: BEAM-5582 > URL: https://issues.apache.org/jira/browse/BEAM-5582 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:32:14.262913 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.29.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:20:30.055051 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.29.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:13:56.194158 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.29.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:14:27.657051 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.29.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:19:25.474102 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.29.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:16:40.570665 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.29.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:16:09.883944 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.30.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:16:43.877239 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.30.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:15:42.678641 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.30.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:16:06.009332 > - > Please consider upgrading the dependency org.apache.qpid:proton-j. > The current version is 0.13.1. The latest version is 0.31.0 > cc: > Please refer to [Beam Dependency Guide >
