[jira] [Assigned] (BEAM-6095) PubsubIO Write to multiple topics
[ https://issues.apache.org/jira/browse/BEAM-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime CLEMENT reassigned BEAM-6095: Assignee: (was: Maxime CLEMENT) > PubsubIO Write to multiple topics > - > > Key: BEAM-6095 > URL: https://issues.apache.org/jira/browse/BEAM-6095 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Maxime CLEMENT >Priority: P2 > Labels: stale-assigned > > When using PubsubIO, it currently write all PubsubMessage to one single topic. > > It should be nice if we could use it with some KV in > order to dispatch PubsubMessage over severals topics. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9869) adding self-contained Kafka service jar for testing
[ https://issues.apache.org/jira/browse/BEAM-9869?focusedWorklogId=439896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439896 ] ASF GitHub Bot logged work on BEAM-9869: Author: ASF GitHub Bot Created on: 02/Jun/20 05:27 Start Date: 02/Jun/20 05:27 Worklog Time Spent: 10m Work Description: ihji commented on pull request #11846: URL: https://github.com/apache/beam/pull/11846#issuecomment-637285003 Fixed solr test failure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439896) Time Spent: 2h 40m (was: 2.5h) > adding self-contained Kafka service jar for testing > --- > > Key: BEAM-9869 > URL: https://issues.apache.org/jira/browse/BEAM-9869 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Labels: stale-assigned > Time Spent: 2h 40m > Remaining Estimate: 0h > > adding self-contained Kafka service jar for testing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439895=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439895
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 05:27
Start Date: 02/Jun/20 05:27
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433627687
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone
[jira] [Work logged] (BEAM-8647) Remove .mailmap from the sources
[ https://issues.apache.org/jira/browse/BEAM-8647?focusedWorklogId=439894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439894 ] ASF GitHub Bot logged work on BEAM-8647: Author: ASF GitHub Bot Created on: 02/Jun/20 05:20 Start Date: 02/Jun/20 05:20 Worklog Time Spent: 10m Work Description: rmannibucau opened a new pull request #11886: URL: https://github.com/apache/beam/pull/11886 side note: if you really need this file (like for the website) maybe use a private asf repo and ensure it is not in a public deliverable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439894) Remaining Estimate: 0h Time Spent: 10m > Remove .mailmap from the sources > > > Key: BEAM-8647 > URL: https://issues.apache.org/jira/browse/BEAM-8647 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Romain Manni-Bucau >Priority: P2 > Time Spent: 10m > Remaining Estimate: 0h > > Hi, > > .mailmap manipulates individuals data which are considered "personal" (name, > email etc) > AFAIK Apache/Beam is not allowed to do it straight, in particular for EU > citizens (_GDPR)._ > Can the file be removed since it is not used by the beam project (at least > apache/beam repo)? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8647) Remove .mailmap from the sources
[ https://issues.apache.org/jira/browse/BEAM-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-8647: - Labels: (was: stale-P2) > Remove .mailmap from the sources > > > Key: BEAM-8647 > URL: https://issues.apache.org/jira/browse/BEAM-8647 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Romain Manni-Bucau >Priority: P2 > > Hi, > > .mailmap manipulates individuals data which are considered "personal" (name, > email etc) > AFAIK Apache/Beam is not allowed to do it straight, in particular for EU > citizens (_GDPR)._ > Can the file be removed since it is not used by the beam project (at least > apache/beam repo)? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9198) BeamSQL aggregation analytics functionality
[
https://issues.apache.org/jira/browse/BEAM-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Wang updated BEAM-9198:
---
Labels: gsoc gsoc2020 mentor (was: gsoc gsoc2020 mentor stale-assigned)
> BeamSQL aggregation analytics functionality
>
>
> Key: BEAM-9198
> URL: https://issues.apache.org/jira/browse/BEAM-9198
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Rui Wang
>Assignee: John Mora
>Priority: P2
> Labels: gsoc, gsoc2020, mentor
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Mentor email: ruw...@google.com. Feel free to send emails for your questions.
> Project Information
> -
> BeamSQL has a long list of of aggregation/aggregation analytics
> functionalities to support.
> To begin with, you will need to support this syntax:
> {code:sql}
> analytic_function_name ( [ argument_list ] )
> OVER (
> [ PARTITION BY partition_expression_list ]
> [ ORDER BY expression [{ ASC | DESC }] [, ...] ]
> [ window_frame_clause ]
> )
> {code}
> As there is a long list of analytics functions, a good start point is support
> rank() first.
> This will requires touch core components of BeamSQL:
> 1. SQL parser to support the syntax above.
> 2. SQL core to implement physical relational operator.
> 3. Distributed algorithms to implement a list of functions in a distributed
> manner.
> 4. Enable in ZetaSQL dialect.
> To understand what SQL analytics functionality is, you could check this great
> explanation doc:
> https://cloud.google.com/bigquery/docs/reference/standard-sql/analytic-function-concepts.
> To know about Beam's programming model, check:
> https://beam.apache.org/documentation/programming-guide/#overview
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9894) Add batch SnowflakeIO.Write to Java SDK
[
https://issues.apache.org/jira/browse/BEAM-9894?focusedWorklogId=439891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439891
]
ASF GitHub Bot logged work on BEAM-9894:
Author: ASF GitHub Bot
Created on: 02/Jun/20 04:58
Start Date: 02/Jun/20 04:58
Worklog Time Spent: 10m
Work Description: purbanow commented on a change in pull request #11794:
URL: https://github.com/apache/beam/pull/11794#discussion_r433620166
##
File path:
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
##
@@ -447,6 +494,346 @@ public void populateDisplayData(DisplayData.Builder
builder) {
}
}
+ /** Implementation of {@link #write()}. */
+ @AutoValue
+ public abstract static class Write extends PTransform,
PDone> {
+@Nullable
+abstract SerializableFunction getDataSourceProviderFn();
+
+@Nullable
+abstract String getTable();
+
+@Nullable
+abstract String getQuery();
+
+@Nullable
+abstract Location getLocation();
+
+@Nullable
+abstract String getFileNameTemplate();
+
+@Nullable
+abstract WriteDisposition getWriteDisposition();
+
+@Nullable
+abstract UserDataMapper getUserDataMapper();
+
+@Nullable
+abstract SnowflakeService getSnowflakeService();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+ abstract Builder setDataSourceProviderFn(
+ SerializableFunction dataSourceProviderFn);
+
+ abstract Builder setTable(String table);
+
+ abstract Builder setQuery(String query);
+
+ abstract Builder setLocation(Location location);
+
+ abstract Builder setFileNameTemplate(String fileNameTemplate);
+
+ abstract Builder setUserDataMapper(UserDataMapper userDataMapper);
+
+ abstract Builder setWriteDisposition(WriteDisposition
writeDisposition);
+
+ abstract Builder setSnowflakeService(SnowflakeService
snowflakeService);
+
+ abstract Write build();
+}
+
+/**
+ * Setting information about Snowflake server.
+ *
+ * @param config - An instance of {@link DataSourceConfiguration}.
+ */
+public Write withDataSourceConfiguration(final DataSourceConfiguration
config) {
+ return withDataSourceProviderFn(new
DataSourceProviderFromDataSourceConfiguration(config));
+}
+
+/**
+ * Setting function that will provide {@link DataSourceConfiguration} in
runtime.
+ *
+ * @param dataSourceProviderFn a {@link SerializableFunction}.
+ */
+public Write withDataSourceProviderFn(
+SerializableFunction dataSourceProviderFn) {
+ return toBuilder().setDataSourceProviderFn(dataSourceProviderFn).build();
+}
+
+/**
+ * A table name to be written in Snowflake.
+ *
+ * @param table - String with the name of the table.
+ */
+public Write to(String table) {
+ return toBuilder().setTable(table).build();
+}
+
+/**
+ * A query to be executed in Snowflake.
+ *
+ * @param query - String with query.
+ */
+public Write withQueryTransformation(String query) {
+ return toBuilder().setQuery(query).build();
+}
+
+/**
+ * A location object which contains connection config between Snowflake
and GCP.
+ *
+ * @param location - an instance of {@link Location}.
+ */
+public Write via(Location location) {
+ return toBuilder().setLocation(location).build();
+}
+
+/**
+ * A template name for files saved to GCP.
+ *
+ * @param fileNameTemplate - String with template name for files.
+ */
+public Write withFileNameTemplate(String fileNameTemplate) {
+ return toBuilder().setFileNameTemplate(fileNameTemplate).build();
+}
+
+/**
+ * User-defined function mapping user data into CSV lines.
+ *
+ * @param userDataMapper - an instance of {@link UserDataMapper}.
+ */
+public Write withUserDataMapper(UserDataMapper userDataMapper) {
+ return toBuilder().setUserDataMapper(userDataMapper).build();
+}
+
+/**
+ * A disposition to be used during writing to table phase.
+ *
+ * @param writeDisposition - an instance of {@link WriteDisposition}.
+ */
+public Write withWriteDisposition(WriteDisposition writeDisposition) {
+ return toBuilder().setWriteDisposition(writeDisposition).build();
+}
+
+/**
+ * A snowflake service which is supposed to be used. Note: Currently we
have {@link
+ * SnowflakeServiceImpl} with corresponding {@link
FakeSnowflakeServiceImpl} used for testing.
+ *
+ * @param snowflakeService - an instance of {@link SnowflakeService}.
+ */
+public Write withSnowflakeService(SnowflakeService snowflakeService) {
+ return toBuilder().setSnowflakeService(snowflakeService).build();
+}
+
+@Override
+public PDone
[jira] [Assigned] (BEAM-7835) Support Triggered PCollectionView Join Unbounded windowed PCollection in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Patwari reassigned BEAM-7835: --- Assignee: (was: Rahul Patwari) > Support Triggered PCollectionView Join Unbounded windowed PCollection in > BeamSQL > > > Key: BEAM-7835 > URL: https://issues.apache.org/jira/browse/BEAM-7835 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-assigned > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7758) Table returns PCollectionView in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-7758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Patwari reassigned BEAM-7758: --- Assignee: (was: Rahul Patwari) > Table returns PCollectionView in BeamSQL > > > Key: BEAM-7758 > URL: https://issues.apache.org/jira/browse/BEAM-7758 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-assigned > > We might be able to define a table with properties that says this table > return a PCollectionView. By doing so we will have a trigger based > PCollectionView available in SQL rel nodes. > Relevant thread: > https://lists.apache.org/thread.html/602121ee49886590ce4975f66aa0d270b4a5d64575337fca8bef1232@%3Cdev.beam.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7831) :sdks:python:portableWordCountBatch out of memory in python precommit
[
https://issues.apache.org/jira/browse/BEAM-7831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122737#comment-17122737
]
Beam JIRA Bot commented on BEAM-7831:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> :sdks:python:portableWordCountBatch out of memory in python precommit
> -
>
> Key: BEAM-7831
> URL: https://issues.apache.org/jira/browse/BEAM-7831
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Udi Meiri
>Priority: P2
> Labels: stale-P2
>
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/4989/consoleFull
> {code}
> 11:46:20 OpenJDK 64-Bit Server VM warning: INFO:
> os::commit_memory(0x000735d8, 716701696, 0) failed; error='Cannot
> allocate memory' (errno=12)
> 11:46:20 #
> 11:46:20 # There is insufficient memory for the Java Runtime Environment to
> continue.
> 11:46:20 # Native memory allocation (mmap) failed to map 716701696 bytes for
> committing reserved memory.
> 11:46:20 # An error report file with more information is saved as:
> 11:46:20 # /opt/apache/beam/hs_err_pid6.log
> 11:46:20 Traceback (most recent call last):
> 11:46:20 File "/usr/lib/python2.7/runpy.py", line 174, in
> _run_module_as_main
> 11:46:20 "__main__", fname, loader, pkg_name)
> 11:46:20 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> 11:46:20 exec code in run_globals
> 11:46:20 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/sdks/python/apache_beam/examples/wordcount.py",
> line 135, in
> 11:46:20 run()
> 11:46:20 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/sdks/python/apache_beam/examples/wordcount.py",
> line 115, in run
> 11:46:20 result.wait_until_finish()
> 11:46:20 File "apache_beam/runners/portability/portable_runner.py", line
> 436, in wait_until_finish
> 11:46:20 for state_response in self._state_stream:
> 11:46:20 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/192237/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 367, in next
> 11:46:20 return self._next()
> 11:46:20 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/192237/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 358, in _next
> 11:46:20 raise self
> 11:46:20 grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
> 11:46:20 status = StatusCode.UNAVAILABLE
> 11:46:20 details = "Socket closed"
> 11:46:20 debug_error_string =
> "{"created":"@1564166775.203261226","description":"Error received from peer
> ipv4:127.0.0.1:35209","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Socket
> closed","grpc_status":14}"
> 11:46:20 >
> 11:46:20 Exception in thread wait_until_finish_read:
> 11:46:20 Traceback (most recent call last):
> 11:46:20 File "/usr/lib/python2.7/threading.py", line 801, in
> __bootstrap_inner
> 11:46:20 self.run()
> 11:46:20 File "/usr/lib/python2.7/threading.py", line 754, in run
> 11:46:20 self.__target(*self.__args, **self.__kwargs)
> 11:46:20 File "apache_beam/runners/portability/portable_runner.py", line
> 418, in read_messages
> 11:46:20 for message in self._message_stream:
> 11:46:20 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/192237/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 367, in next
> 11:46:20 return self._next()
> 11:46:20 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Portable_Python_Commit/src/build/gradleenv/192237/local/lib/python2.7/site-packages/grpc/_channel.py",
> line 358, in _next
> 11:46:20 raise self
> 11:46:20 _Rendezvous: <_Rendezvous of RPC that terminated with:
> 11:46:20 status = StatusCode.UNAVAILABLE
> 11:46:20 details = "Socket closed"
> 11:46:20 debug_error_string =
> "{"created":"@1564166775.203284918","description":"Error received from peer
> ipv4:127.0.0.1:35209","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Socket
> closed","grpc_status":14}"
> 11:46:20 >
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-6826) All Python counters should use the same code path
[ https://issues.apache.org/jira/browse/BEAM-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122894#comment-17122894 ] Beam JIRA Bot commented on BEAM-6826: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > All Python counters should use the same code path > - > > Key: BEAM-6826 > URL: https://issues.apache.org/jira/browse/BEAM-6826 > Project: Beam > Issue Type: New Feature > Components: sdk-py-harness >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > Right now we have Metrics.counter, Metrics.distribution counters which use > MetricContainers to store and accumulate metrics. > CounterSet counters which use accumulators to store and accumulate metrics. > This code path is optimized using cython. > See cy_combiners.py > > Ideally we can keep the user interface (this should not change) for creating > metrics with Metrics.counter(), .etc. But use the underlying optimized > CounterSets. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8601) Options passed to TestStream should augment, not replace, options specified by --test-pipeline-options.
[ https://issues.apache.org/jira/browse/BEAM-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8601: Labels: stale-P2 (was: ) > Options passed to TestStream should augment, not replace, options specified > by --test-pipeline-options. > --- > > Key: BEAM-8601 > URL: https://issues.apache.org/jira/browse/BEAM-8601 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8789) Document InfluxDB/Kapacitor deployment on Kubernetes in project's wiki
[ https://issues.apache.org/jira/browse/BEAM-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8789: Labels: stale-P2 (was: ) > Document InfluxDB/Kapacitor deployment on Kubernetes in project's wiki > -- > > Key: BEAM-8789 > URL: https://issues.apache.org/jira/browse/BEAM-8789 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kamil Wasilewski >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6910) Beam does not consider BigQuery's processing location when getting query results
[
https://issues.apache.org/jira/browse/BEAM-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122875#comment-17122875
]
Beam JIRA Bot commented on BEAM-6910:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Beam does not consider BigQuery's processing location when getting query
> results
>
>
> Key: BEAM-6910
> URL: https://issues.apache.org/jira/browse/BEAM-6910
> Project: Beam
> Issue Type: Bug
> Components: dependencies, runner-dataflow, sdk-py-core
>Affects Versions: 2.11.0
> Environment: Python
>Reporter: Graham Polley
>Priority: P2
> Labels: stale-P2
>
> When using the BigQuery source with a SQL query in a pipeline, the
> "processing location" is not taken into consideration and the pipeline fails.
> For example, consider the following which uses {{BigQuerySource}} to read
> from BigQuery using some SQL. The BigQuery dataset and tables are located in
> {{australia-southeast1}}. The query is submitted successfully ([Beam works
> out the processing location by examining the first table referenced in the
> query and sets it
> accordingly|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L221]),
> but when Beam attempts to poll for the job status after it has been
> submitted, it fails because it doesn't set the {{location}} to be
> {{australia-southeast1}}, which is required by BigQuery:
>
> {code:java}
> p | 'read' >> beam.io.Read(beam.io.BigQuerySource(use_standard_sql=True,
> query='SELECT * from
> `a_project_id.dataset_in_australia.table_in_australia`'){code}
>
> {code:java}
> HttpNotFoundError: HttpError accessing
> :
> response: <{'status': '404', 'content-length': '328', 'x-xss-protection':
> '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF',
> '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Tue, 26 Mar
> 2019 03:11:32 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443";
> ma=2592000; v="46,44,43,39"', 'content-type': 'application/json;
> charset=UTF-8'}>, content <{
> "error": {
> "code": 404,
> "message": "Not found: Job a_project_id:5ad9cc803baa432290b6cd0203f556d9",
> "errors": [
> {
> "message": "Not found: Job
> a_project_id:5ad9cc803baa432290b6cd0203f556d9",
> "domain": "global",
> "reason": "notFound"
> }
> ],
> "status": "NOT_FOUND"
> }
> }
> {code}
>
> The problem can be seen/found here:
> [https://github.com/apache/beam/blob/v2.11.0/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L571]
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L357]
> The location of the job (in this case {{australia-southeast1}}) needs to
> set/inferred (or exposed via the API), otherwise its fails.
> For reference, Airflow had the same bug/problem:
> [https://github.com/apache/airflow/pull/4695]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7425) Reading BigQuery Table Data into Java Classes(Pojo) Directly
[
https://issues.apache.org/jira/browse/BEAM-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122795#comment-17122795
]
Beam JIRA Bot commented on BEAM-7425:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Reading BigQuery Table Data into Java Classes(Pojo) Directly
>
>
> Key: BEAM-7425
> URL: https://issues.apache.org/jira/browse/BEAM-7425
> Project: Beam
> Issue Type: New Feature
> Components: io-java-avro, io-java-gcp
>Affects Versions: 2.12.0
> Environment: Dataflow
>Reporter: Kishan Kumar
>Priority: P2
> Labels: stale-P2
>
> While Developing my code I used the below code snippet to read the table data
> from BigQuery.
>
> {code:java}
> PCollection gpseEftReasonCodes = input
> .apply("Reading xxyyzz",
> BigQueryIO
> .read(new ReadTable(ReasonCode.class))
> .withoutValidation()
> .withTemplateCompatibility()
> .fromQuery("Select * from dataset.xxyyzz")
> .usingStandardSql()
> .withCoder(SerializableCoder.of(xxyyzz.class))
> {code}
> Read Table Class:
> {code:java}
> @DefaultSchema(JavaBeanSchema.class)
> public class ReadTable implements SerializableFunction
> {
> private static final long serialVersionUID = 1L;
> private static Gson gson = new Gson();
> public static final Logger LOG = LoggerFactory.getLogger(ReadTable.class);
> private final Counter countingRecords =
> Metrics.counter(ReadTable.class, "Reading Records EFT Report");
> private Class class1;
>
> public ReadTable(Class class1) { this.class1 = class1; }
>
> public T apply(SchemaAndRecord schemaAndRecord) {
> Map mapping = new HashMap<>();
> int counter = 0;
> try {
> GenericRecord s = schemaAndRecord.getRecord();
> org.apache.avro.Schema s1 = s.getSchema();
> for (Field f : s1.getFields()) {
> counter++;
> mapping.put(f.name(), null==s.get(f.name()) ? null :
> String.valueOf(s.get(counter)));
> }
> countingRecords.inc();
> JsonElement jsonElement = gson.toJsonTree(mapping);
> return gson.fromJson(jsonElement, class1);
> } catch (Exception mp) {
> LOG.error("Found Wrong Mapping for the Record: "+mapping);
> mp.printStackTrace(); return null; }
> }
> }
> {code}
> So After Reading the data from Bigquery I was mapping data from
> SchemaAndRecord to pojo I was getting value for columns whose Data type is
> Numeric mention below.
> {code}
> last_update_amount=java.nio.HeapByteBuffer[pos=0 lim=16 cap=16]
> {code}
> My Expectation was I will get exact value but getting the HyperByte Buffer
> the version I am using is Apache beam 2.12.0. If any more information is
> needed then please let me know.
> Way 2 Tried:
> {code:java}
> GenericRecord s = schemaAndRecord.getRecord();
> org.apache.avro.Schema s1 = s.getSchema();
> for (Field f : s1.getFields()) {
> counter++;
> mapping.put(f.name(), null==s.get(f.name()) ? null :
> String.valueOf(s.get(counter)));
> if(f.name().equalsIgnoreCase("reason_code_id")) {
> BigDecimal numericValue = new Conversions.DecimalConversion()
>.fromBytes((ByteBuffer) s.get(f.name()), Schema.create(s1.getType()),
> s1.getLogicalType());
>System.out.println("Numeric Con"+numericValue);
> } else {
> System.out.println("Else Condition "+f.name());
> }
> {code}
> Facing Issue:
> {code}
> 2019-05-24 (14:10:37) org.apache.avro.AvroRuntimeException: Can't create a:
> RECORD
> {code}
>
> It would be Great if we have a method which maps all the BigQuery Data with
> Pojo Schema which Means if I have 10 Columns in BQ and in my Pojo I need only
> 5 Column then, in that case, BigQueryIO should map only that 5 Data values
> into Java Class and Rest will be Rejected As I am Doing After So much Effort.
> Numeric Data Type must be Deserialize by itself while fetching data like
> TableRow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-6821) FileBasedSink is not creating file paths according to target filesystem
[
https://issues.apache.org/jira/browse/BEAM-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122896#comment-17122896
]
Beam JIRA Bot commented on BEAM-6821:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> FileBasedSink is not creating file paths according to target filesystem
> ---
>
> Key: BEAM-6821
> URL: https://issues.apache.org/jira/browse/BEAM-6821
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: Windows 10
>Reporter: Gregory Kovelman
>Priority: P2
> Labels: stale-P2
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> File path generated in _open_writer_ method is not according to target
> filesystem, because
> os.path.join is used and not FileSystems.join.
> apache_beam\io\filebasedsink.py extract:
>
> {code:java}
> def _create_temp_dir(self, file_path_prefix):
> base_path, last_component = FileSystems.split(file_path_prefix)
> if not last_component:
># Trying to re-split the base_path to check if it's a root.
>new_base_path, _ = FileSystems.split(base_path)
>if base_path == new_base_path:
> raise ValueError('Cannot create a temporary directory for root path '
> 'prefix %s. Please specify a file path prefix with '
> 'at least two components.' % file_path_prefix)
> path_components = [base_path,
> 'beam-temp-' + last_component + '-' + uuid.uuid1().hex]
> return FileSystems.join(*path_components)
> @check_accessible(['file_path_prefix', 'file_name_suffix'])
> def open_writer(self, init_result, uid):
> # A proper suffix is needed for AUTO compression detection.
> # We also ensure there will be no collisions with uid and a
> # (possibly unsharded) file_path_prefix and a (possibly empty)
> # file_name_suffix.
> file_path_prefix = self.file_path_prefix.get()
> file_name_suffix = self.file_name_suffix.get()
> suffix = (
> '.' + os.path.basename(file_path_prefix) + file_name_suffix)
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> {code}
>
>
> This created incompatibilities between, for example, Windows and GCS.
> Expected: gs://bucket/beam-temp-result-uuid\\uid.result
> Actual: gs://bucket/beam-temp-result-uuid/uid.result
> Replacing os.path.join with FileSystems.join fixes the issue
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8714) Beam Dependency Update Request: org.apache.avro
[ https://issues.apache.org/jira/browse/BEAM-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8714: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.apache.avro > --- > > Key: BEAM-8714 > URL: https://issues.apache.org/jira/browse/BEAM-8714 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:43:32.227446 > - > Please consider upgrading the dependency org.apache.avro. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9159) Beam Dependency Update Request: com.google.zetasql:zetasql-types
[ https://issues.apache.org/jira/browse/BEAM-9159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122402#comment-17122402 ] Beam JIRA Bot commented on BEAM-9159: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.google.zetasql:zetasql-types > > > Key: BEAM-9159 > URL: https://issues.apache.org/jira/browse/BEAM-9159 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2020-01-20 12:09:39.384328 > - > Please consider upgrading the dependency > com.google.zetasql:zetasql-types. > The current version is 2019.12.1. The latest version is 2020.01.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8838) Apache Beam Metrics Counter giving incorrect count using SparkRunner
[ https://issues.apache.org/jira/browse/BEAM-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8838: Labels: stale-P2 (was: ) > Apache Beam Metrics Counter giving incorrect count using SparkRunner > > > Key: BEAM-8838 > URL: https://issues.apache.org/jira/browse/BEAM-8838 > Project: Beam > Issue Type: Bug > Components: runner-spark >Affects Versions: 2.13.0, 2.14.0, 2.16.0 > Environment: Cloudera Express 6.2.0 > Java Version: 1.8.0_181 > Spark 2.4.0-cdh6.2.0 > 1 Master Node and 3 Data node(64 cores, 128GB RAM) > --driver-memory "2g" --num-executors "6" --executor-cores "3" >Reporter: kunal >Priority: P2 > Labels: stale-P2 > > I am having source and target csv files with 10 million records and 250 > columns. I am running an apache beam pipeline which joins all columns from > source and target file. When I run this on spark cluster the pipeline > executes correctly with no exceptions but, The join beam metrics counter > returns double count when the following spark property is used. -- > executor-memory "2g" But, When I increase the excutor-memory to 11g then it > returns the correct count. > Count doubles only when I dump the results to file but if I don't dump then > counts are correct. > Note : > [https://stackoverflow.com/questions/59032734/apache-beam-metrics-counter-giving-incorrect-count-using-sparkrunner?noredirect=1#comment104344657_59032734] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8505) Too many variations of FnApiRunnerTest are running
[ https://issues.apache.org/jira/browse/BEAM-8505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8505: Labels: stale-P2 (was: ) > Too many variations of FnApiRunnerTest are running > -- > > Key: BEAM-8505 > URL: https://issues.apache.org/jira/browse/BEAM-8505 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Pablo Estrada >Priority: P2 > Labels: stale-P2 > Time Spent: 1h 20m > Remaining Estimate: 0h > > These variations add up to make the Python Precommit suite very slow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8707) Beam Dependency Update Request: org.antlr:antlr4
[ https://issues.apache.org/jira/browse/BEAM-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122543#comment-17122543 ] Beam JIRA Bot commented on BEAM-8707: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.antlr:antlr4 > > > Key: BEAM-8707 > URL: https://issues.apache.org/jira/browse/BEAM-8707 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:43:04.915244 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:09:50.321535 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:15:55.434512 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:15:10.823839 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:15:33.163546 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:10:41.033494 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:14:19.881205 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:13:41.685550 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.7.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:11:09.316818 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.8-1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:12:02.684503 > - > Please consider upgrading the dependency org.antlr:antlr4. > The current version is 4.7. The latest version is 4.8-1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03
[jira] [Updated] (BEAM-8222) Consider making insertId optional in BigQuery.insertAll
[ https://issues.apache.org/jira/browse/BEAM-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8222: Labels: stale-P2 (was: ) > Consider making insertId optional in BigQuery.insertAll > --- > > Key: BEAM-8222 > URL: https://issues.apache.org/jira/browse/BEAM-8222 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Boyuan Zhang >Priority: P2 > Labels: stale-P2 > > Current implementation of > StreamingWriteFn(https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteFn.java#L102) > sets insertId from input element, which is added an uniqueId by > https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java#L53. > Users report that if leaving insertId as empty, writing will be extremely > speeded up. Can we add an bqOption like, nonInsertId and emit empty id based > on this option? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6799) Streamline project version usage
[ https://issues.apache.org/jira/browse/BEAM-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122902#comment-17122902 ] Beam JIRA Bot commented on BEAM-6799: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Streamline project version usage > > > Key: BEAM-6799 > URL: https://issues.apache.org/jira/browse/BEAM-6799 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Michael Luckey >Priority: P2 > Labels: stale-P2 > > Currently we have project version set in gradle.properties [1]. On the other > hand we overwrite immediately the project version in BeamModulePlugin [2], so > that the usage in gradle.properties is superfluous. > This is intermingled with isRelease flag, which toggles version as snapshot > version. Revisit that procedure and make consistent. > See also BEAM-6726 (comments about removing isRelease) and BEAM-6798 about > usage of version in gradle.properties. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9145) Integrate Google Cloud AI functionalities
[ https://issues.apache.org/jira/browse/BEAM-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9145: Labels: stale-P2 (was: ) > Integrate Google Cloud AI functionalities > - > > Key: BEAM-9145 > URL: https://issues.apache.org/jira/browse/BEAM-9145 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp, io-py-gcp >Reporter: Kamil Wasilewski >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7659) Create IO tests for synthetic sources
[ https://issues.apache.org/jira/browse/BEAM-7659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7659: Labels: stale-P2 (was: ) > Create IO tests for synthetic sources > - > > Key: BEAM-7659 > URL: https://issues.apache.org/jira/browse/BEAM-7659 > Project: Beam > Issue Type: Wish > Components: testing >Reporter: Lukasz Gajowy >Priority: P2 > Labels: stale-P2 > > Synthetic sources are not tested thoroughly. It would be good to test > generated data consistency (IOIT tests?) in different configurations (eg. > multiple splits). > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8112) Support passing stateBackend through pipeline options in python sdks
[
https://issues.apache.org/jira/browse/BEAM-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-8112:
Labels: stale-P2 (was: )
> Support passing stateBackend through pipeline options in python sdks
>
>
> Key: BEAM-8112
> URL: https://issues.apache.org/jira/browse/BEAM-8112
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: Catlyn Kong
>Priority: P2
> Labels: stale-P2
>
> Currently the only way for python sdks to instruct flink to use a
> StateBackend different than the default (MemoryStateBackend) would be to
> specify state.backend in flink-conf.yaml, which creates the limitation of
> using the same statebackend for every job running on the same flink cluster.
> Ideally we should be able to pass it in to flink runner through
> PipelineOptions. Here's the error it spits out when I flag
> --state_backend=RocksDBStateBackend:
>
> {code:java}
> RuntimeError: Pipeline failed in state FAILED:
> com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot
> construct instance of `org.apache.flink.runtime.state.StateBackend` (no
> Creators, like default construct, exist): abstract types either need to be
> mapped to concrete types, have custom deserializer, or contain additional
> type information
> at [Source: (String)""RocksDBStateBackend""; line: 1, column: 1]
> {code}
> Acceptance Criteria:
> Flink StateBackend is configurable via command line options from python.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8779) Beam Dependency Update Request: software.amazon.awssdk:sns
[ https://issues.apache.org/jira/browse/BEAM-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122493#comment-17122493 ] Beam JIRA Bot commented on BEAM-8779: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: software.amazon.awssdk:sns > -- > > Key: BEAM-8779 > URL: https://issues.apache.org/jira/browse/BEAM-8779 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-19 21:16:49.149055 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.19 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:24:08.775701 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.26 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:23:46.101446 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.30 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:24:33.501651 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.40 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:19:09.662186 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.41 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:22:40.041350 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.42 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:22:30.059400 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.47 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:19:59.904063 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.52 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:21:52.719302 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.56 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:22:26.523336 > - > Please consider upgrading the dependency software.amazon.awssdk:sns. > The current version is 2.5.71. The latest version is 2.10.56 > cc: > Please refer to [Beam
[jira] [Commented] (BEAM-8060) Support DATE type
[ https://issues.apache.org/jira/browse/BEAM-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122685#comment-17122685 ] Beam JIRA Bot commented on BEAM-8060: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Support DATE type > - > > Key: BEAM-8060 > URL: https://issues.apache.org/jira/browse/BEAM-8060 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9107) Beam Dependency Update Request: org.eclipse.jetty.npn
[ https://issues.apache.org/jira/browse/BEAM-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122418#comment-17122418 ] Beam JIRA Bot commented on BEAM-9107: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.eclipse.jetty.npn > - > > Key: BEAM-9107 > URL: https://issues.apache.org/jira/browse/BEAM-9107 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2020-01-13 12:19:59.012236 > - > Please consider upgrading the dependency org.eclipse.jetty.npn. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7681) Support Fanout in Apache BEAM SQL extension
[ https://issues.apache.org/jira/browse/BEAM-7681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122760#comment-17122760 ] Beam JIRA Bot commented on BEAM-7681: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Support Fanout in Apache BEAM SQL extension > --- > > Key: BEAM-7681 > URL: https://issues.apache.org/jira/browse/BEAM-7681 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.13.0 >Reporter: Brachi Packter >Priority: P2 > Labels: stale-P2 > > I want to use Fanout improvements: > [https://www.waitingforcode.com/apache-beam/fanouts-apache-beam-combine-transform/read] > How can I use it when I'm doing my join and aggregation via BeamSQL? > Can you add API that I can configure the fanout? > For example: > `SqlTransform.query(query).withFanOut()` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7818) Create Presto I/O connector
[ https://issues.apache.org/jira/browse/BEAM-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122739#comment-17122739 ] Beam JIRA Bot commented on BEAM-7818: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Create Presto I/O connector > --- > > Key: BEAM-7818 > URL: https://issues.apache.org/jira/browse/BEAM-7818 > Project: Beam > Issue Type: New Feature > Components: io-ideas >Reporter: Jun Hee Yoo >Priority: P2 > Labels: stale-P2 > > New I/O connector for [Presto|https://prestodb.github.io/]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8634) Beam Dependency Update Request: com.google.protobuf:protoc
[ https://issues.apache.org/jira/browse/BEAM-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122584#comment-17122584 ] Beam JIRA Bot commented on BEAM-8634: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.google.protobuf:protoc > -- > > Key: BEAM-8634 > URL: https://issues.apache.org/jira/browse/BEAM-8634 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-12 22:48:07.434506 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.10.1 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:45.554723 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.10.1 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:42.171821 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.11.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:54.645988 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.11.1 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:12:00.561308 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:07:23.806223 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:58.322710 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:10:29.470332 > - > Please consider upgrading the dependency com.google.protobuf:protoc. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8677) Beam Dependency Update Request: com.google.api.grpc:grpc-google-cloud-pubsub-v1
[ https://issues.apache.org/jira/browse/BEAM-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8677: Labels: stale-P2 (was: ) > Beam Dependency Update Request: > com.google.api.grpc:grpc-google-cloud-pubsub-v1 > --- > > Key: BEAM-8677 > URL: https://issues.apache.org/jira/browse/BEAM-8677 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:38:35.896728 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.83.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:03:31.644132 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.83.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:08:28.427468 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.83.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:07:34.017888 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:07:43.055541 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:03:11.275415 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:06:47.027977 > - > Please consider upgrading the dependency > com.google.api.grpc:grpc-google-cloud-pubsub-v1. > The current version is 1.43.0. The latest version is 1.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9017) Beam Dependency Update Request: cachetools
[ https://issues.apache.org/jira/browse/BEAM-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9017: Labels: stale-P2 (was: ) > Beam Dependency Update Request: cachetools > -- > > Key: BEAM-9017 > URL: https://issues.apache.org/jira/browse/BEAM-9017 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-12-23 12:04:17.483984 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:00:14.523312 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:03:38.328424 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:04:04.593478 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:03:36.859072 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:04:22.622190 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:06:51.686822 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-10 12:03:51.875766 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-17 12:04:37.088086 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-24 12:04:37.014614 > - > Please consider upgrading the dependency cachetools. > The current version is 3.1.1. The latest version is 4.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9107) Beam Dependency Update Request: org.eclipse.jetty.npn
[ https://issues.apache.org/jira/browse/BEAM-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9107: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.eclipse.jetty.npn > - > > Key: BEAM-9107 > URL: https://issues.apache.org/jira/browse/BEAM-9107 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2020-01-13 12:19:59.012236 > - > Please consider upgrading the dependency org.eclipse.jetty.npn. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7853) PubsubIO and FixedWindows
[
https://issues.apache.org/jira/browse/BEAM-7853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7853:
Labels: stale-P2 (was: )
> PubsubIO and FixedWindows
> -
>
> Key: BEAM-7853
> URL: https://issues.apache.org/jira/browse/BEAM-7853
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Gregory Parsons
>Priority: P2
> Labels: stale-P2
>
> Hi all,
> I am having a potential issue with Windowing on cloud PubsubIO.
> I am finding that FixedWindows do not trigger on either DirectRunner or
> DataflowRunner after running a GroupBy transform.
> A basic pipeline with my use case would look like:
>
> {code:java}
> Pipeline p = Pipeline.create();
> PubsubIO.Read read = PubsubIO
> .readMessagesWithAttributes()
> .withTimestampAttribute("time")
> .fromTopic("test-topic");
> Window window = Window.into(
> FixedWindows.of(Duration.standardSeconds(10L))
> )
> .triggering(AfterWatermark.pastEndOfWindow())
> .withAllowedLateness(Duration.standardSeconds(10L))
> .discardingFiredPanes();
> PCollection>> windowedMessages = p
> .apply("Read Events", read)
> .apply("Apply Window", window)
> .apply("Convert to KV", ParDo.of(new ConvertToMapOnKey()))
> .apply("Group by key", GroupByKey.create())
> .apply("Log Pairs", ParDo.of(new LogGroupedEvents()));{code}
>
> LogGroupedEvents would log the key as a string, and the array of
> PubsubMessages in the grouped array. But this function never runs correctly.
> For simplicity I have simplified the pipeline to demonstrate the issue and
> have removed the actual use case of the pipeline. Therefore it may seem odd
> that I am grouping and logging simple messages but that is actually not what
> I am doing.
>
> If I swap the windowing function for one with triggers it works correctly.
> {code:java}
> Window getDefaultWindow(Long duration) {
> return Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> AfterProcessingTime
> .pastFirstElementInPane()
> .plusDelayOf(Duration.standardSeconds(duration)
> )
> ))
> .withAllowedLateness(Duration.standardSeconds(10L))
> .discardingFiredPanes()
> ;
> }
> {code}
>
> This could be due to me not understanding windowing and triggers but
> according the documentation and many examples online all that people use is a
> simple FixedWindow because it needs to automatically run a trigger at the end
> of the window per the beam docs:
>
> [https://beam.apache.org/documentation/programming-guide/#setting-your-pcollections-windowing-function]
> On example 7.3.1.
>
> I have been researching as much as I can about how windowing works
> internally. We arrived to our solution with triggering by looking at source
> code.
>
> Let me know if there is any other information you need from me to help look
> into this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-7692) Create a transform to assign an index when reading data from built in I/O transforms
[ https://issues.apache.org/jira/browse/BEAM-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123339#comment-17123339 ] SAILOKESH DIVI edited comment on BEAM-7692 at 6/2/20, 4:37 AM: --- This feature will be very helpful to assign index when reading data so that we can validate them for statistics was (Author: saidivi): This feature will be very helpful to assign index when reading data > Create a transform to assign an index when reading data from built in I/O > transforms > > > Key: BEAM-7692 > URL: https://issues.apache.org/jira/browse/BEAM-7692 > Project: Beam > Issue Type: New Feature > Components: io-ideas, io-java-text >Affects Versions: 2.13.0 >Reporter: SAILOKESH DIVI >Priority: P2 > Labels: beam > > As a beam user > when using any of the existing beams I/O transforms > I would like to add an index to each line read as part of the transform. > > As spark has zipWithIndex to assign an index when reading files with the beam > being abstraction layer for may runners. I would expect this feature should > be added to beam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8607) Beam Dependency Update Request: tenacity
[ https://issues.apache.org/jira/browse/BEAM-8607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8607: Labels: stale-P2 (was: ) > Beam Dependency Update Request: tenacity > > > Key: BEAM-8607 > URL: https://issues.apache.org/jira/browse/BEAM-8607 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-11 12:03:53.472802 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:02:03.620137 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:05:25.038093 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:05:53.236773 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:06:04.397652 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:01:30.790967 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:05:05.348777 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:05:25.917980 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:05:12.020217 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:05:49.204988 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:08:15.453029 > - > Please consider upgrading the dependency tenacity. > The current version is 5.1.5. The latest version is 6.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-10 12:05:31.222988 > - > Please consider
[jira] [Commented] (BEAM-6950) Beam Dependency Update Request: com.github.spotbugs
[ https://issues.apache.org/jira/browse/BEAM-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122871#comment-17122871 ] Beam JIRA Bot commented on BEAM-6950: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.github.spotbugs > --- > > Key: BEAM-6950 > URL: https://issues.apache.org/jira/browse/BEAM-6950 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Time Spent: 50m > Remaining Estimate: 0h > > - 2019-04-01 12:15:04.215434 > - > Please consider upgrading the dependency com.github.spotbugs. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8005) beam_PostCommit_Python37 timing out
[
https://issues.apache.org/jira/browse/BEAM-8005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122703#comment-17122703
]
Beam JIRA Bot commented on BEAM-8005:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> beam_PostCommit_Python37 timing out
> ---
>
> Key: BEAM-8005
> URL: https://issues.apache.org/jira/browse/BEAM-8005
> Project: Beam
> Issue Type: Bug
> Components: test-failures, testing
>Reporter: Udi Meiri
>Priority: P2
> Labels: stale-P2
>
> Seems to get stuck in :sdks:python:test-suites:dataflow:py37:postCommitIT
> {code}
> 10:03:30 Build timed out (after 100 minutes). Marking the build as aborted.
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/consoleFull
> possible culprits listed in changes for these PRs:
> https://builds.apache.org/job/beam_PostCommit_Python37/173/
> https://builds.apache.org/job/beam_PostCommit_Python37/174/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8704) Beam Dependency Update Request: javax.xml.bind
[ https://issues.apache.org/jira/browse/BEAM-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122547#comment-17122547 ] Beam JIRA Bot commented on BEAM-8704: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: javax.xml.bind > -- > > Key: BEAM-8704 > URL: https://issues.apache.org/jira/browse/BEAM-8704 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:42:30.637543 > - > Please consider upgrading the dependency javax.xml.bind. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7334) Create a better Schema builder
[
https://issues.apache.org/jira/browse/BEAM-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122810#comment-17122810
]
Beam JIRA Bot commented on BEAM-7334:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Create a better Schema builder
> --
>
> Key: BEAM-7334
> URL: https://issues.apache.org/jira/browse/BEAM-7334
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Reporter: Reuven Lax
>Priority: P2
> Labels: stale-P2
>
> Right now the Schema builder class is a bit verbose to use. The AddFields
> transform already contains a nicer way of building schemas, but it's coupled
> to that transform. We should refactor that code and make it available in the
> Schema builder. That would allow coder of the form
> Schema schema = Schema.builder()
> .addField("a", INT32)
> .addField("a.b", STRING)
> .addField("a.c", INT32).build();
>
> Which is much more concise for nested fields.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8282) Update Apache Qpid Proton deps version
[ https://issues.apache.org/jira/browse/BEAM-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122638#comment-17122638 ] Beam JIRA Bot commented on BEAM-8282: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Update Apache Qpid Proton deps version > -- > > Key: BEAM-8282 > URL: https://issues.apache.org/jira/browse/BEAM-8282 > Project: Beam > Issue Type: Improvement > Components: io-java-amqp >Reporter: Alexey Romanenko >Priority: P2 > Labels: stale-P2 > > Current version of Apache Qpid Proton used in AmqpIO is *0.13.1*, though, the > last released version is *0.33.2*. > Moving to new version will require to stop using Messenger API since it was > deprecated and removed a while ago. We need to consider alternatives for that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8712) Beam Dependency Update Request: org.apache.apex
[ https://issues.apache.org/jira/browse/BEAM-8712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122539#comment-17122539 ] Beam JIRA Bot commented on BEAM-8712: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.apex > --- > > Key: BEAM-8712 > URL: https://issues.apache.org/jira/browse/BEAM-8712 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:43:27.662529 > - > Please consider upgrading the dependency org.apache.apex. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5750) Beam Dependency Update Request: javax.servlet:javax.servlet-api
[ https://issues.apache.org/jira/browse/BEAM-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123070#comment-17123070 ] Beam JIRA Bot commented on BEAM-5750: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: javax.servlet:javax.servlet-api > --- > > Key: BEAM-5750 > URL: https://issues.apache.org/jira/browse/BEAM-5750 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-15 12:13:14.253682 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:13:20.677351 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-29 12:17:03.954722 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-05 12:14:50.209888 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-12 12:14:50.401239 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-19 12:15:31.652290 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-11-26 12:14:29.396363 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-03 12:14:52.265627 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-10 12:17:21.331309 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest version is 4.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-12-17 12:17:42.365110 > - > Please consider upgrading the dependency > javax.servlet:javax.servlet-api. > The current version is 3.1.0. The latest
[jira] [Commented] (BEAM-8067) Throw exception when truncating nano/micro to millis when creating time literals
[ https://issues.apache.org/jira/browse/BEAM-8067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122679#comment-17122679 ] Beam JIRA Bot commented on BEAM-8067: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Throw exception when truncating nano/micro to millis when creating time > literals > - > > Key: BEAM-8067 > URL: https://issues.apache.org/jira/browse/BEAM-8067 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > > time values in googlesql is encoded in a special form. Need a function to > extract sub-millis from time values and decide if rejection is needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6646) Beam Dependency Update Request: com.gradle.build-scan
[ https://issues.apache.org/jira/browse/BEAM-6646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122927#comment-17122927 ] Beam JIRA Bot commented on BEAM-6646: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: com.gradle.build-scan > - > > Key: BEAM-6646 > URL: https://issues.apache.org/jira/browse/BEAM-6646 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-02-11 12:12:25.062529 > - > Please consider upgrading the dependency com.gradle.build-scan. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7292) Upgrade hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/BEAM-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7292: Labels: stale-P2 (was: ) > Upgrade hadoop-mapreduce-client-core > > > Key: BEAM-7292 > URL: https://issues.apache.org/jira/browse/BEAM-7292 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Etienne Chauchot >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8638) Beam Dependency Update Request: org.apache.kafka
[ https://issues.apache.org/jira/browse/BEAM-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122581#comment-17122581 ] Beam JIRA Bot commented on BEAM-8638: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.kafka > > > Key: BEAM-8638 > URL: https://issues.apache.org/jira/browse/BEAM-8638 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-12 22:52:09.576977 > - > Please consider upgrading the dependency org.apache.kafka. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7165) FileIOTest.testMatchWatchForNewFiles flakey in java presubmit
[ https://issues.apache.org/jira/browse/BEAM-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122842#comment-17122842 ] Beam JIRA Bot commented on BEAM-7165: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > FileIOTest.testMatchWatchForNewFiles flakey in java presubmit > - > > Key: BEAM-7165 > URL: https://issues.apache.org/jira/browse/BEAM-7165 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > [https://builds.apache.org/job/beam_PreCommit_Java_Commit/5634/testReport/junit/org.apache.beam.sdk.io/FileIOTest/testMatchWatchForNewFiles/] > Note: This test was flakey and fixed in BEAM-6491, filed this new ticket > since I am not sure if its the same issue. > > h3. Stacktrace > java.lang.AssertionError: > FileIO.MatchAll/Reshuffle.ViaRandomKey/Values/Values/Map/ParMultiDo(Anonymous).output: > Expected: iterable with items > [ sizeBytes=42, isReadSeekEfficient=true, lastModifiedMillis=0}>, > sizeBytes=37, isReadSeekEfficient=true, lastModifiedMillis=0}>, > sizeBytes=99, isReadSeekEfficient=true, lastModifiedMillis=0}>] in any order > but: not matched: > isReadSeekEfficient=true, lastModifiedMillis=0}> at > org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:169) > at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:393) at > org.apache.beam.sdk.testing.PAssert.that(PAssert.java:385) at > org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles(FileIOTest.java:262) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) at > org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:265) > at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:349) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:314) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:312) at > org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at > org.junit.runners.ParentRunner.run(ParentRunner.java:396) at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93) >
[jira] [Updated] (BEAM-7768) Formalize Python external transform API
[ https://issues.apache.org/jira/browse/BEAM-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7768: Labels: stale-P2 (was: ) > Formalize Python external transform API > --- > > Key: BEAM-7768 > URL: https://issues.apache.org/jira/browse/BEAM-7768 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Priority: P2 > Labels: stale-P2 > > As a python developer I'd like to have a more opinionated API for external > transforms to A) guide me toward more standardized solutions and B) reduce > the amount of boilerplate code that needs to be written. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8798) Contributing docs omit many steps
[ https://issues.apache.org/jira/browse/BEAM-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8798: Labels: stale-P2 (was: ) > Contributing docs omit many steps > - > > Key: BEAM-8798 > URL: https://issues.apache.org/jira/browse/BEAM-8798 > Project: Beam > Issue Type: Bug > Components: website >Affects Versions: 2.16.0 >Reporter: Elliotte Rusty Harold >Priority: P2 > Labels: stale-P2 > > So far I've noticed the following packages and tools that are not available > by default on a Debian system, need to be installed to build and test beam, > and are not yet mentioned in all our docs: > * tox > * Go > * Python (which versions?) > * Docker > There might be more. The docs such as README.md should be updated to include > these. This covers README.md, CONTRIBUTING.md, > https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ, and > https://beam.apache.org/contribute/ > That we have four such docs is itself an issue. We should probably > consolidate these. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7745) StreamingSideInputDoFnRunner/StreamingSideInputFetcher have suboptimal state access pattern during normal operation
[ https://issues.apache.org/jira/browse/BEAM-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122752#comment-17122752 ] Beam JIRA Bot commented on BEAM-7745: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > StreamingSideInputDoFnRunner/StreamingSideInputFetcher have suboptimal state > access pattern during normal operation > --- > > Key: BEAM-7745 > URL: https://issues.apache.org/jira/browse/BEAM-7745 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Priority: P2 > Labels: stale-P2 > > I spent some time tracking down sources of uncached state fetches in my job, > and one large category was the interaction of StreamingSideInputDoFnRunner + > StreamingSideInputFetcher. > Basically, during standard operations, when the main input is NOT blocked by > the side input, the side input fetcher will perform an uncached state read > for every input element. Changing it to cache the blockedMap state gave me a > ~30-40% increase in throughput in my job. > The interaction is a little complicated, and there's a couple optimizations > here I can see. > > Primarily, the blockedMap is only persisted if it is non-empty. Because the > WindmillStateCache won't cache a null value, this means that the "nothing is > blocked" signal is never actually cached, and will issue a state read to > windmill for each input element. The solution here seems like it is to > persist an empty map rather than a null when there are no blocked elements. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8680) Beam Dependency Update Request: com.google.api.grpc:proto-google-cloud-datastore-v1
[ https://issues.apache.org/jira/browse/BEAM-8680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122560#comment-17122560 ] Beam JIRA Bot commented on BEAM-8680: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: > com.google.api.grpc:proto-google-cloud-datastore-v1 > --- > > Key: BEAM-8680 > URL: https://issues.apache.org/jira/browse/BEAM-8680 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:38:47.772347 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:04:01.770240 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:09:03.579130 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:08:13.788049 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:08:14.850304 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:03:43.419080 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:07:18.957411 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:07:21.652290 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:06:59.020054 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-datastore-v1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The
[jira] [Updated] (BEAM-7745) StreamingSideInputDoFnRunner/StreamingSideInputFetcher have suboptimal state access pattern during normal operation
[ https://issues.apache.org/jira/browse/BEAM-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7745: Labels: stale-P2 (was: ) > StreamingSideInputDoFnRunner/StreamingSideInputFetcher have suboptimal state > access pattern during normal operation > --- > > Key: BEAM-7745 > URL: https://issues.apache.org/jira/browse/BEAM-7745 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Steve Niemitz >Priority: P2 > Labels: stale-P2 > > I spent some time tracking down sources of uncached state fetches in my job, > and one large category was the interaction of StreamingSideInputDoFnRunner + > StreamingSideInputFetcher. > Basically, during standard operations, when the main input is NOT blocked by > the side input, the side input fetcher will perform an uncached state read > for every input element. Changing it to cache the blockedMap state gave me a > ~30-40% increase in throughput in my job. > The interaction is a little complicated, and there's a couple optimizations > here I can see. > > Primarily, the blockedMap is only persisted if it is non-empty. Because the > WindmillStateCache won't cache a null value, this means that the "nothing is > blocked" signal is never actually cached, and will issue a state read to > windmill for each input element. The solution here seems like it is to > persist an empty map rather than a null when there are no blocked elements. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5676) Allow to change the PubsubClientFactory when using PubsubIO
[
https://issues.apache.org/jira/browse/BEAM-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-5676:
Labels: stale-P2 (was: )
> Allow to change the PubsubClientFactory when using PubsubIO
> ---
>
> Key: BEAM-5676
> URL: https://issues.apache.org/jira/browse/BEAM-5676
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
>Affects Versions: 2.7.0
>Reporter: Logan HAUSPIE
>Priority: P2
> Labels: stale-P2
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When we use PubSub to push or pull messages, we currently have no choice of
> serialization implementation because PubsubIO use internaly the
> *_PubsubJsonClient.FACTORY_* of type _*PubsubJsonClientFactory*_ to serialize
> or deserialize messages.
> It should be nice to be able to set a different factory (e.g.
> *_PubsubGrpcClientFactory_*) to decrease the size of messages (and then
> decrease the price of using Pubsub).
>
> I guess It could be possible to write something like:
> {{PubsubIO.Write write =}}
> {{ PubsubIO.writeMessages()}}
> {{ .to("projects/project/topics/topic")}}
> {{ *.withFactory(PubsubGrpcClient.FACTORY)*}}
> {{ .withTimestampAttribute("timestamp")}}{{;}}
> or
> {{PubsubIO.Read read = }}
> {{ PubsubIO.readMessages()}}
> {{ .fromSubscription("projects/project/subscriptions/name")}}
> {{ *.withFactory(PubsubGrpcClient.FACTORY)*}}
> {{ .withTimestampAttribute("timestamp");}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7692) Create a transform to assign an index when reading data from built in I/O transforms
[ https://issues.apache.org/jira/browse/BEAM-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123339#comment-17123339 ] SAILOKESH DIVI commented on BEAM-7692: -- This feature will be very helpful to assign index when reading data > Create a transform to assign an index when reading data from built in I/O > transforms > > > Key: BEAM-7692 > URL: https://issues.apache.org/jira/browse/BEAM-7692 > Project: Beam > Issue Type: New Feature > Components: io-ideas, io-java-text >Affects Versions: 2.13.0 >Reporter: SAILOKESH DIVI >Priority: P2 > Labels: beam > > As a beam user > when using any of the existing beams I/O transforms > I would like to add an index to each line read as part of the transform. > > As spark has zipWithIndex to assign an index when reading files with the beam > being abstraction layer for may runners. I would expect this feature should > be added to beam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7692) Create a transform to assign an index when reading data from built in I/O transforms
[ https://issues.apache.org/jira/browse/BEAM-7692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SAILOKESH DIVI updated BEAM-7692: - Labels: beam (was: beam stale-P2) > Create a transform to assign an index when reading data from built in I/O > transforms > > > Key: BEAM-7692 > URL: https://issues.apache.org/jira/browse/BEAM-7692 > Project: Beam > Issue Type: New Feature > Components: io-ideas, io-java-text >Affects Versions: 2.13.0 >Reporter: SAILOKESH DIVI >Priority: P2 > Labels: beam > > As a beam user > when using any of the existing beams I/O transforms > I would like to add an index to each line read as part of the transform. > > As spark has zipWithIndex to assign an index when reading files with the beam > being abstraction layer for may runners. I would expect this feature should > be added to beam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7789) :beam-test-tools project fails to build locally
[
https://issues.apache.org/jira/browse/BEAM-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122743#comment-17122743
]
Beam JIRA Bot commented on BEAM-7789:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> :beam-test-tools project fails to build locally
> ---
>
> Key: BEAM-7789
> URL: https://issues.apache.org/jira/browse/BEAM-7789
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Anton Kedin
>Priority: P2
> Labels: stale-P2
>
> Running the release-verification build (global build of everything) in turn
> triggers the build of the `beam-test-tools` project, which has some test
> infrastructure scripts that we run on Jenkins. It seems to work fine on
> jenkins. However running the build of the project locally fails:
> https://scans.gradle.com/s/kqhkzyozbpiua/console-log#L6
> What seems to happen is the gradle vendoring plugin caches the dependencies
> locally, but fails to cache simplelru.
> One workaround (based on ./gradlew :beam-test-tools:showGopathGoroot)
> {code}
> export GOPATH=$PWD/.test-infra/tools/.gogradle/project_gopath
> go get github.com/hashicorp/golang-lru/simplelru
> ./gradlew :beam-test-tools:build
> {code}
> It is able to find the `lrumap` and `simplelru` during the dependency
> resolution step, and I can see it mentioned in couple of artifacts produced
> by the `gogradle` plugin. But when it does `:installDepedencies` to actually
> copy them to `vendor` directory, this specific package is missing. This
> reproduces for me on a couple of different machines I tried, both on release
> and master branches
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7309) Adding AvroCoder to StandardCoders
[ https://issues.apache.org/jira/browse/BEAM-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122811#comment-17122811 ] Beam JIRA Bot commented on BEAM-7309: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Adding AvroCoder to StandardCoders > -- > > Key: BEAM-7309 > URL: https://issues.apache.org/jira/browse/BEAM-7309 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core, sdk-py-core >Reporter: Heejong Lee >Priority: P2 > Labels: stale-P2 > > Should AvroCoder be a standard coder? It could be useful as a common codec > for a simple dict type element. > https://github.com/apache/beam/pull/8342#discussion_r284025317 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9018) Beam Dependency Update Request: httplib2
[ https://issues.apache.org/jira/browse/BEAM-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122441#comment-17122441 ] Beam JIRA Bot commented on BEAM-9018: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: httplib2 > > > Key: BEAM-9018 > URL: https://issues.apache.org/jira/browse/BEAM-9018 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-12-23 12:05:07.739677 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.15.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:00:50.038093 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.15.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:04:16.448089 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.15.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:04:37.840332 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.15.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:04:14.892687 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.16.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:04:58.455497 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.17.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:07:25.549852 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.17.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-10 12:04:27.300461 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.17.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-17 12:05:17.548108 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.17.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-24 12:05:16.811254 > - > Please consider upgrading the dependency httplib2. > The current version is 0.12.0. The latest version is 0.17.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-03-24 13:47:56.776488 > - >
[jira] [Commented] (BEAM-5854) Separate Windowing from AggregationRel and make itself a Rel
[ https://issues.apache.org/jira/browse/BEAM-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123047#comment-17123047 ] Beam JIRA Bot commented on BEAM-5854: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Separate Windowing from AggregationRel and make itself a Rel > - > > Key: BEAM-5854 > URL: https://issues.apache.org/jira/browse/BEAM-5854 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7853) PubsubIO and FixedWindows
[
https://issues.apache.org/jira/browse/BEAM-7853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122733#comment-17122733
]
Beam JIRA Bot commented on BEAM-7853:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> PubsubIO and FixedWindows
> -
>
> Key: BEAM-7853
> URL: https://issues.apache.org/jira/browse/BEAM-7853
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Gregory Parsons
>Priority: P2
> Labels: stale-P2
>
> Hi all,
> I am having a potential issue with Windowing on cloud PubsubIO.
> I am finding that FixedWindows do not trigger on either DirectRunner or
> DataflowRunner after running a GroupBy transform.
> A basic pipeline with my use case would look like:
>
> {code:java}
> Pipeline p = Pipeline.create();
> PubsubIO.Read read = PubsubIO
> .readMessagesWithAttributes()
> .withTimestampAttribute("time")
> .fromTopic("test-topic");
> Window window = Window.into(
> FixedWindows.of(Duration.standardSeconds(10L))
> )
> .triggering(AfterWatermark.pastEndOfWindow())
> .withAllowedLateness(Duration.standardSeconds(10L))
> .discardingFiredPanes();
> PCollection>> windowedMessages = p
> .apply("Read Events", read)
> .apply("Apply Window", window)
> .apply("Convert to KV", ParDo.of(new ConvertToMapOnKey()))
> .apply("Group by key", GroupByKey.create())
> .apply("Log Pairs", ParDo.of(new LogGroupedEvents()));{code}
>
> LogGroupedEvents would log the key as a string, and the array of
> PubsubMessages in the grouped array. But this function never runs correctly.
> For simplicity I have simplified the pipeline to demonstrate the issue and
> have removed the actual use case of the pipeline. Therefore it may seem odd
> that I am grouping and logging simple messages but that is actually not what
> I am doing.
>
> If I swap the windowing function for one with triggers it works correctly.
> {code:java}
> Window getDefaultWindow(Long duration) {
> return Window.into(new GlobalWindows())
> .triggering(Repeatedly.forever(
> AfterProcessingTime
> .pastFirstElementInPane()
> .plusDelayOf(Duration.standardSeconds(duration)
> )
> ))
> .withAllowedLateness(Duration.standardSeconds(10L))
> .discardingFiredPanes()
> ;
> }
> {code}
>
> This could be due to me not understanding windowing and triggers but
> according the documentation and many examples online all that people use is a
> simple FixedWindow because it needs to automatically run a trigger at the end
> of the window per the beam docs:
>
> [https://beam.apache.org/documentation/programming-guide/#setting-your-pcollections-windowing-function]
> On example 7.3.1.
>
> I have been researching as much as I can about how windowing works
> internally. We arrived to our solution with triggering by looking at source
> code.
>
> Let me know if there is any other information you need from me to help look
> into this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8182) Run Nexmark Java based queries on Spark portable runner
[ https://issues.apache.org/jira/browse/BEAM-8182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8182: Labels: portability-spark stale-P2 (was: portability-spark) > Run Nexmark Java based queries on Spark portable runner > --- > > Key: BEAM-8182 > URL: https://issues.apache.org/jira/browse/BEAM-8182 > Project: Beam > Issue Type: Sub-task > Components: runner-spark >Reporter: Kyle Weaver >Priority: P2 > Labels: portability-spark, stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6061) When using unbounded PCollection from TextIO to BigQuery, data is stuck in Reshuffle/GroupByKey inside of BigQueryIO
[ https://issues.apache.org/jira/browse/BEAM-6061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6061: Labels: stale-P2 (was: ) > When using unbounded PCollection from TextIO to BigQuery, data is stuck in > Reshuffle/GroupByKey inside of BigQueryIO > > > Key: BEAM-6061 > URL: https://issues.apache.org/jira/browse/BEAM-6061 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.8.0 >Reporter: Florian Baumert >Priority: P2 > Labels: stale-P2 > > As a short summary, when reading from TextIO with watchForNewFiles (unbounded > collection), BigQueryIO in stream mode, BigQueryIO does not write data. It is > "stuck" in a GroupByKey internal to Reshuffle. > > The issue is detailed in stackoverflow with code and version information. > [https://stackoverflow.com/questions/53266689/when-using-unbounded-pcollection-from-textio-to-bigquery-data-is-stuck-in-reshu] > > Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8210) Python Integration tests: log test name
[ https://issues.apache.org/jira/browse/BEAM-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122646#comment-17122646 ] Beam JIRA Bot commented on BEAM-8210: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Python Integration tests: log test name > --- > > Key: BEAM-8210 > URL: https://issues.apache.org/jira/browse/BEAM-8210 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow, sdk-py-core, testing >Reporter: Udi Meiri >Priority: P2 > Labels: stale-P2 > > When creating a job (on any runner), log the originating test so it's easier > to debug. > Postcommits frequently run tens of pipelines at a time and it's getting > harder to tell them apart. > By logging I mean putting it somewhere in the job proto (such as in > parameter, job name, etc.). Using the worker logger on startup won't work if > the worker fails to start. > Ideally you should be able to see the name in the runner UI (such as Dataflow > cloud console). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7693) FILE_LOADS option for inserting rows in BigQuery creates a stuck process in Dataflow that saturates all the resources of the Job
[
https://issues.apache.org/jira/browse/BEAM-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7693:
Labels: stale-P2 (was: )
> FILE_LOADS option for inserting rows in BigQuery creates a stuck process in
> Dataflow that saturates all the resources of the Job
>
>
> Key: BEAM-7693
> URL: https://issues.apache.org/jira/browse/BEAM-7693
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, runner-dataflow
>Affects Versions: 2.13.0
> Environment: Dataflow
>Reporter: Juan Urrego
>Priority: P2
> Labels: stale-P2
> Attachments: Screenshot 2019-07-06 at 15.05.04.png,
> image-2019-07-06-15-04-17-593.png
>
>
> During a Stream Job, when you insert records to BigQuery in batch using the
> FILE_LOADS option and one of the jobs fail, the thread who failed is getting
> stuck and eventually it saturates the Job resources, making the autoscaling
> option useless (uses the max number of workers and the system latency always
> goes up). In some cases it has become ridiculous slow trying to process the
> incoming events.
> Here is an example:
> {code:java}
> BigQueryIO.writeTableRows()
> .to(destinationTableSerializableFunction)
> .withMethod(Method.FILE_LOADS)
> .withJsonSchema(tableSchema)
> .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
> .withWriteDisposition(WriteDisposition.WRITE_APPEND)
> .withTriggeringFrequency(Duration.standardMinutes(5))
> .withNumFileShards(25);
> {code}
> The pipeline works like a charm, but in the moment that I send a wrong
> tableRow (for instance a required value as null) the pipeline starts sending
> this messages:
> {code:java}
> Processing stuck in step FILE_LOADS: in
> BigQuery/BatchLoads/SinglePartitionWriteTables/ParMultiDo(WriteTables) for at
> least 10m00s without outputting or completing in state finish at
> java.lang.Thread.sleep(Native Method) at
> com.google.api.client.util.Sleeper$1.sleep(Sleeper.java:42) at
> com.google.api.client.util.BackOffUtils.next(BackOffUtils.java:48) at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.nextBackOff(BigQueryHelpers.java:159)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$PendingJobManager.waitForDone(BigQueryHelpers.java:145)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.finishBundle(WriteTables.java:255)
> at
> org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeFinishBundle(Unknown
> Source)
> {code}
> It's clear that the step keeps running even when it failed. The BigQuery Job
> mentions that the task failed, but DataFlow keeps trying to wait for a
> response, even when the job is never executed again.
> !image-2019-07-06-15-04-17-593.png|width=497,height=134!
> At the same time, no message is sent to the DropInputs step, even when I
> created my own step for DeadLetter, the process think that it hasn't failed
> yet.
> !Screenshot 2019-07-06 at 15.05.04.png|width=490,height=306!
> The only option that I have found so far, is to pre validate all the fields
> before, but I was expecting the DB to do that for me, especially in some
> extreme cases (like decimal numbers or constraint limitations). Please help
> fixing this issue, otherwise the batch option in stream jobs is almost
> useless, because I can't trust the own library to manage dead letters properly
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5947) BeamFnDataGrpcMultiplexer is never closed
[ https://issues.apache.org/jira/browse/BEAM-5947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123026#comment-17123026 ] Beam JIRA Bot commented on BEAM-5947: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > BeamFnDataGrpcMultiplexer is never closed > - > > Key: BEAM-5947 > URL: https://issues.apache.org/jira/browse/BEAM-5947 > Project: Beam > Issue Type: New Feature > Components: java-fn-execution >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > The BeamFnDataGrpcMultiplexer close code is never invoked. We should make > sure to invoke this on a shutdown code path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6728) Allow sharing of the job and artifact service address.
[ https://issues.apache.org/jira/browse/BEAM-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6728: Labels: stale-P2 (was: ) > Allow sharing of the job and artifact service address. > -- > > Key: BEAM-6728 > URL: https://issues.apache.org/jira/browse/BEAM-6728 > Project: Beam > Issue Type: New Feature > Components: runner-core >Reporter: Robert Bradshaw >Priority: P2 > Labels: stale-P2 > > GRPC can serve multiple services off of the same address. It would be ideal > to allow serving the artifact service off the same port as the job service, > especially when firewalls and port forwarding are involved. Of particular > note, the artifact staging service address provided in the job preparation > response could have a special value indicating "use the job server address" > as the job server itself may be incapable as providing an address accessible > to the client. > > [~mxm] [~angoenka] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6849) SpannerIO.Write "Part Running" at schemaView, never completes with Wait.on
[ https://issues.apache.org/jira/browse/BEAM-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6849: Labels: stale-P2 (was: ) > SpannerIO.Write "Part Running" at schemaView, never completes with Wait.on > -- > > Key: BEAM-6849 > URL: https://issues.apache.org/jira/browse/BEAM-6849 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.10.0 >Reporter: shashi reddy >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > Here is a gist: > [https://gist.github.com/sgireddy/859261f6335d303ec08049fcb2a7972a] > > I am using scio wrapper but I don't think this is scio issue given I am using > direct beam call and I see "Schema View" task is blocking on dataflow DAG. > > Here is my build.sbt: > name := "dataflow-spanner-template" > version := "0.1" > scalaVersion := "2.11.12" > val beamVersion = "2.10.0" > val scioVersion = "0.7.2" > libraryDependencies ++= Seq( > "org.apache.beam" % "beam-runners-google-cloud-dataflow-java" % beamVersion, > "com.spotify" %% "scio-core" % scioVersion, > "com.spotify" %% "scio-spanner" % scioVersion, > "com.google.cloud" % "google-cloud-spanner" % "1.4.0" > ) > > I saw the following exception with the above gist (but not in my original > code). > > java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: > java.lang.NoSuchMethodError: > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$Builder.setMaxInboundMetadataSize(Ljava/lang/Integer;)Lcom/google/api/gax/grpc/InstantiatingGrpcChannelProvider$Builder; > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:194) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:165) > > org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63) > > org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50) > > org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:125) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1149) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:144) > > org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:972) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > java.lang.Thread.run(Thread.java:745) Caused by: > org.apache.beam.sdk.util.UserCodeException: java.lang.NoSuchMethodError: > com.google.api.gax.grpc.InstantiatingGrpcChannelProvider$Builder.setMaxInboundMetadataSize(Ljava/lang/Integer;)Lcom/google/api/gax/grpc/InstantiatingGrpcChannelProvider$Builder; > org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:34) > org.apache.beam.sdk.io.gcp.spanner.ReadSpannerSchema$DoFnInvoker.invokeSetup(Unknown > Source) > org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:80) > > org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:62) > > org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:95) > > org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:264) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:86) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:183) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:165) > > org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63) > > org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50) > > org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87) > > org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:125) > >
[jira] [Commented] (BEAM-5150) MqttIOTest.testRead is flaky
[ https://issues.apache.org/jira/browse/BEAM-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123180#comment-17123180 ] Beam JIRA Bot commented on BEAM-5150: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > MqttIOTest.testRead is flaky > > > Key: BEAM-5150 > URL: https://issues.apache.org/jira/browse/BEAM-5150 > Project: Beam > Issue Type: Bug > Components: io-java-mqtt >Reporter: Mikhail Gryzykhin >Priority: P2 > Labels: flake, sickbay, stale-P2 > Time Spent: 20m > Remaining Estimate: 0h > > Failing job url: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1239/testReport/org.apache.beam.sdk.io.mqtt/MqttIOTest/testRead/] > Job history url: > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1234/testReport/junit/org.apache.beam.sdk.io.mqtt/MqttIOTest/testRead/history/ > Relevant log: > java.lang.AssertionError: > MqttIO.Read/Read(UnboundedMqttSource)/StripIds/ParMultiDo(StripIds).output: > Expected: iterable over [[<84>, <104>, <105>, <115>, <32>, <105>, <115>, > <32>, <116>, <101>, <115>, <116>, <32>, <48>], [<84>, <104>, <105>, <115>, > <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, <49>], [<84>, > <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, > <32>, <50>], [<84>, <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, > <101>, <115>, <116>, <32>, <51>], [<84>, <104>, <105>, <115>, <32>, <105>, > <115>, <32>, <116>, <101>, <115>, <116>, <32>, <52>], [<84>, <104>, <105>, > <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, <53>], > [<84>, <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, > <116>, <32>, <54>], [<84>, <104>, <105>, <115>, <32>, <105>, <115>, <32>, > <116>, <101>, <115>, <116>, <32>, <55>], [<84>, <104>, <105>, <115>, <32>, > <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, <56>], [<84>, <104>, > <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, > <57>]] in any order but: No item matches: [<84>, <104>, <105>, <115>, <32>, > <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, <48>], [<84>, <104>, > <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, > <49>], [<84>, <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, > <115>, <116>, <32>, <50>], [<84>, <104>, <105>, <115>, <32>, <105>, <115>, > <32>, <116>, <101>, <115>, <116>, <32>, <51>], [<84>, <104>, <105>, <115>, > <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, <52>] in [[<84>, > <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, > <32>, <53>], [<84>, <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, > <101>, <115>, <116>, <32>, <54>], [<84>, <104>, <105>, <115>, <32>, <105>, > <115>, <32>, <116>, <101>, <115>, <116>, <32>, <55>], [<84>, <104>, <105>, > <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, <116>, <32>, <56>], > [<84>, <104>, <105>, <115>, <32>, <105>, <115>, <32>, <116>, <101>, <115>, > <116>, <32>, <57>]] at > org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:168) > at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:374) at > org.apache.beam.sdk.testing.PAssert.that(PAssert.java:366) at > org.apache.beam.sdk.io.mqtt.MqttIOTest.testRead(MqttIOTest.java:148) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8771) Beam Dependency Update Request: org.apache.hadoop:hadoop-minicluster
[ https://issues.apache.org/jira/browse/BEAM-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122501#comment-17122501 ] Beam JIRA Bot commented on BEAM-8771: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.hadoop:hadoop-minicluster > > > Key: BEAM-8771 > URL: https://issues.apache.org/jira/browse/BEAM-8771 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-19 21:11:28.423753 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.7.3. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:18:11.423232 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.7.3. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:17:25.471611 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:17:48.468892 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:13:01.502930 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:16:38.903742 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:15:48.543744 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:13:11.212506 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:14:08.692904 > - > Please consider upgrading the dependency > org.apache.hadoop:hadoop-minicluster. > The current version is 2.8.5. The latest version is 3.2.1 > cc: [~timrobertson100], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The
[jira] [Updated] (BEAM-5854) Separate Windowing from AggregationRel and make itself a Rel
[ https://issues.apache.org/jira/browse/BEAM-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-5854: Labels: stale-P2 (was: ) > Separate Windowing from AggregationRel and make itself a Rel > - > > Key: BEAM-5854 > URL: https://issues.apache.org/jira/browse/BEAM-5854 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8062) Support array member accessor
[ https://issues.apache.org/jira/browse/BEAM-8062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8062: Labels: stale-P2 (was: ) > Support array member accessor > - > > Key: BEAM-8062 > URL: https://issues.apache.org/jira/browse/BEAM-8062 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > > array[] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8678) Beam Dependency Update Request: com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1
[ https://issues.apache.org/jira/browse/BEAM-8678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122567#comment-17122567 ] Beam JIRA Bot commented on BEAM-8678: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1 > -- > > Key: BEAM-8678 > URL: https://issues.apache.org/jira/browse/BEAM-8678 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:38:40.884685 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.44.0. The latest version is 0.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:03:46.619772 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.44.0. The latest version is 0.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:08:44.174983 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.44.0. The latest version is 0.84.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:07:49.948037 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.44.0. The latest version is 0.85.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-10 12:07:08.942518 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.85.1. The latest version is 0.89.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-17 12:08:01.955510 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.85.1. The latest version is 0.89.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-24 12:07:44.549333 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.85.1. The latest version is 0.90.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-03-24 13:50:14.186110 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.85.1. The latest version is 0.91.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-03-30 12:07:42.983407 > - > Please consider upgrading the dependency > com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1. > The current version is 0.85.1. The latest version is 0.91.0 > cc: > Please refer to
[jira] [Commented] (BEAM-6975) Merge portability status into capability matrix
[ https://issues.apache.org/jira/browse/BEAM-6975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122865#comment-17122865 ] Beam JIRA Bot commented on BEAM-6975: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Merge portability status into capability matrix > --- > > Key: BEAM-6975 > URL: https://issues.apache.org/jira/browse/BEAM-6975 > Project: Beam > Issue Type: Sub-task > Components: website >Reporter: Ahmet Altay >Priority: P2 > Labels: stale-P2 > > Should the portability status: > https://s.apache.org/apache-beam-portability-support-table > be merged into capability matrix > https://beam.apache.org/documentation/runners/capability-matrix/ ? > (That is add portable runners to the list of runners as columns in the > capability matrix.) > cc: [~kenn] [~lcwik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7582) TestPubsub.listSubscriptions is flaky
[ https://issues.apache.org/jira/browse/BEAM-7582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122776#comment-17122776 ] Beam JIRA Bot commented on BEAM-7582: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > TestPubsub.listSubscriptions is flaky > - > > Key: BEAM-7582 > URL: https://issues.apache.org/jira/browse/BEAM-7582 > Project: Beam > Issue Type: Improvement > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > > o.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline expired before > operation could complete. > > at > > io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) > > at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) > > at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) > > at > > com.google.pubsub.v1.SubscriberGrpc$SubscriberBlockingStub.listSubscriptions(SubscriberGrpc.java:1734) > > at > > org.apache.beam.sdk.io.gcp.pubsub.PubsubGrpcClient.listSubscriptions(PubsubGrpcClient.java:373) > > at > > org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.listSubscriptions(TestPubsub.java:165) > https://builds.apache.org/job/beam_PostCommit_SQL/1843/ > https://builds.apache.org/job/beam_PostCommit_SQL/1842/ > https://builds.apache.org/job/beam_PostCommit_SQL/1841/ > https://builds.apache.org/job/beam_PostCommit_SQL/1840/ > https://builds.apache.org/job/beam_PostCommit_SQL/1839/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7149) support OR
[ https://issues.apache.org/jira/browse/BEAM-7149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122849#comment-17122849 ] Beam JIRA Bot commented on BEAM-7149: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > support OR > -- > > Key: BEAM-7149 > URL: https://issues.apache.org/jira/browse/BEAM-7149 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > > OR supposed to be implement in rules. > Basically when LogicalRel is generated and being converted to PhysicalRel, OR > should be handled by a separate converter rule: > all conjunction clauses of OR should be converted to independent JOIN and > their results are combined by UNION rel with deduplication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4437) Maximum recursion depth with Apache Beam and Google Cloud SDK
[ https://issues.apache.org/jira/browse/BEAM-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4437: Labels: stale-P2 (was: ) > Maximum recursion depth with Apache Beam and Google Cloud SDK > - > > Key: BEAM-4437 > URL: https://issues.apache.org/jira/browse/BEAM-4437 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Affects Versions: 2.4.0 > Environment: LSB Version: > core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch > Distributor ID: Ubuntu > Description: Ubuntu 18.04 LTS > Release: 18.04 > Codename: bionic > Python 2.7.15rc1 > Google Cloud SDK 200.0.0 > app-engine-go > app-engine-python 1.9.69 > app-engine-python-extras 1.9.69 > bq 2.0.33 > core 2018.04.30 > gsutil 4.31 >Reporter: Jonathan Perron >Priority: P2 > Labels: stale-P2 > > I am working with a project which is using Google Cloud SDK and Cloud Engine. > This project is generating a DataStore which I need to access in order to > copy parts of it towards a Big Query instance, similar to what is describe in > [https://blog.papercut.com/google-cloud-dataflow-data-migration/]. > I install Apache Beam using pip install --upgrade apache_beam. However, when > I set up everything in order to access the DataStore using the ndb models and > want to import Beam using from airflow_beam import pipeline, I got an error > related to some part of the SDK described in the docs part. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7370) Beam Dependency Update Request: Sphinx
[ https://issues.apache.org/jira/browse/BEAM-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7370: Labels: stale-P2 (was: ) > Beam Dependency Update Request: Sphinx > -- > > Key: BEAM-7370 > URL: https://issues.apache.org/jira/browse/BEAM-7370 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Time Spent: 0.5h > Remaining Estimate: 0h > > - 2019-05-20 16:38:07.937770 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-17 12:32:27.855338 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-24 12:02:59.052884 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-07-01 12:04:13.113613 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-07-08 12:03:15.091005 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-07-15 12:03:09.406918 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-07-22 12:03:31.157859 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-07-29 12:05:13.023604 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-08-05 12:03:03.242767 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-08-12 12:04:01.647619 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.1.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-08-19 12:02:52.342008 > - > Please consider upgrading the dependency Sphinx. > The current version is 1.8.5. The latest version is 2.2.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-08-26 12:02:44.918642 > - >
[jira] [Commented] (BEAM-6031) Add retry logic to S3FileSystem
[ https://issues.apache.org/jira/browse/BEAM-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123019#comment-17123019 ] Beam JIRA Bot commented on BEAM-6031: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Add retry logic to S3FileSystem > > > Key: BEAM-6031 > URL: https://issues.apache.org/jira/browse/BEAM-6031 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.7.0, 2.8.0 >Reporter: Pawel Bartoszek >Priority: P2 > Labels: stale-P2 > > S3FileSystem should have some retry behaviour if ObjectsDelete fails. I have > seen such example in our job where 1 item from the delete batch cannot be > deleted due to S3 InternalError causing the whole job to restart. The source > code I am referring to: > [https://github.com/apache/beam/blob/8a88e72f293ef7f9be6c872aa0dda681458c7ca5/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java#L633] > > The retry logic might be added to other S3 calls in S3FileSystem as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6954) @Default not called if the options json has null value for a property
[ https://issues.apache.org/jira/browse/BEAM-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122866#comment-17122866 ] Beam JIRA Bot commented on BEAM-6954: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > @Default not called if the options json has null value for a property > - > > Key: BEAM-6954 > URL: https://issues.apache.org/jira/browse/BEAM-6954 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Affects Versions: 2.11.0 >Reporter: Balázs Németh >Priority: P2 > Labels: stale-P2 > > When a pipeline options get deserialized from a json with > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L738-L760] > it creates a map, where properties present in the json - even if with a null > value - will be added to the map. > So we can have String->NullNode mappings. > When you create a ProxyInvocationHandler with this Map ( > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L117-L125] > ) this map will be the backing jsonOptions map. > Later on when a getter is called on the options it will reach this code: > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L156-L158] > > Then the containsKey will return true, even for NullNodes. So we won't > execute the getDefault() method hence not calculating the default value. > > I'm not sure about the expected behaviour, but either: > - the containsKey check should be expanded with an !isNull check > OR > - when we serialize the json, it shouldn't serialize null values at > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L653-L655] > > Instinctively I would have expected the @Default.* annotations producing > values every single time, when the value is null - so a property with a > @Default.* annotation can't be null - but I was unable to find anything > explicit regarding this in the documentation. So I'm not sure which of the > suggested change has to be made. > --- > Okay, I have investigated further, and it seems the default value is indeed > calculated before the json serialization by calling the mentioned method. The > problem is that it returns a RuntimeValueProvider, which gets serialized as > null ( > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java#L275-L279] > ), because the isAccessible returns false ( > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java#L248-L250] > + > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java#L214-L220] > ) > ... and then during deserialization it is found in the jsonOptions at ( > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L156-L158] > ), so it executes the getValueFromJson which uses an ObjectMapper to create > a ValueProvider from a NullNode ( > [https://github.com/apache/beam/blob/a85ea07b719385ec185e4fc5e4cdcc67b3598599/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java#L498] > ) . > The problem here is that according to the JsonDeserializer documentation, the > deserialize method isn't executed for null nodes. -> > [https://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.6/com/fasterxml/jackson/databind/JsonDeserializer.html#deserialize(com.fasterxml.jackson.core.JsonParser,%20com.fasterxml.jackson.databind.DeserializationContext)] > For this part of the issue see BEAM-6963 (that still doesn't solve this issue > btw, but might be required for it) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-998) Consider asking Apache to register Apache Beam trademark
[ https://issues.apache.org/jira/browse/BEAM-998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davor Bonaci reassigned BEAM-998: - Assignee: (was: Davor Bonaci) > Consider asking Apache to register Apache Beam trademark > > > Key: BEAM-998 > URL: https://issues.apache.org/jira/browse/BEAM-998 > Project: Beam > Issue Type: Task > Components: project-management >Affects Versions: Not applicable >Reporter: Dan Halperin >Priority: P2 > > "Registered Trademarks If a PMC would like to request legal registration of > their project's trademarks, please registering their marks, please follow the > REGREQUEST instructions." > http://www.apache.org/foundation/marks/pmcs#other > The link to REGREQUEST: > http://www.apache.org/foundation/marks/register#register -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9499) test_multi_triggered_gbk_side_input is failing on head
[
https://issues.apache.org/jira/browse/BEAM-9499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruoyun Huang resolved BEAM-9499.
Fix Version/s: 2.21.0
Resolution: Fixed
> test_multi_triggered_gbk_side_input is failing on head
> --
>
> Key: BEAM-9499
> URL: https://issues.apache.org/jira/browse/BEAM-9499
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ruoyun Huang
>Priority: P2
> Labels: stale-assigned
> Fix For: 2.21.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> test_multi_triggered_gbk_side_input is failing after it was fixed to run on
> Dataflow runner. Earlier it was always running on DirectRunner.
> Example failure:
> [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/6004/testReport/junit/apache_beam.transforms.sideinputs_test/SideInputsTest/test_multi_triggered_gbk_side_input_7/]
> Error:
> h3. Error Message
> 'list' object has no attribute 'proto' >> begin captured
> logging << apache_beam.options.pipeline_options:
> WARNING: --region not set; will default to us-central1. Future releases of
> Beam will require the user to set --region explicitly, or else have a default
> set via the gcloud tool.
> {{[https://cloud.google.com/compute/docs/regions-zones]}}
> root: DEBUG: Unhandled type_constraint: Union[] root: DEBUG: Unhandled
> type_constraint: Union[] root: DEBUG: Unhandled type_constraint: Union[]
> apache_beam.runners.runner: ERROR: Error while visiting Main windowInto
> - >> end captured logging << -
> h3. Stacktrace
> File "/usr/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
> yield File "/usr/lib/python3.6/unittest/case.py", line 605, in run
> testMethod() File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/transforms/sideinputs_test.py",
> line 406, in test_multi_triggered_gbk_side_input p.run() File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 112, in run False if self.not_use_test_runner_api else
> test_runner_api)) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
> line 495, in run self._options).run(False) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
> line 508, in run return self.runner.run_pipeline(self, self._options) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 57, in run_pipeline self).run_pipeline(pipeline, options) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 536, in run_pipeline self.visit_transforms(pipeline, options) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/runner.py",
> line 224, in visit_transforms pipeline.visit(RunVisitor(self)) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
> line 545, in visit self._root_transform().visit(visitor, self, visited) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
> line 1033, in visit part.visit(visitor, pipeline, visited) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/pipeline.py",
> line 1036, in visit visitor.visit_transform(self) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/runner.py",
> line 219, in visit_transform self.runner.run_transform(transform_node,
> options) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/runner.py",
> line 246, in run_transform return m(transform_node, options) File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 957, in run_ParDo PropertyNames.STEP_NAME: input_step.proto.name,
> 'list' object has no attribute 'proto' >> begin captured
> logging << apache_beam.options.pipeline_options:
> WARNING: --region not set; will default to us-central1. Future releases of
> Beam will require the user to set --region explicitly, or else have a default
> set via the gcloud tool.
>
[jira] [Resolved] (BEAM-8645) TimestampCombiner incorrect in beam python
[
https://issues.apache.org/jira/browse/BEAM-8645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruoyun Huang resolved BEAM-8645.
Fix Version/s: 2.21.0
Resolution: Fixed
> TimestampCombiner incorrect in beam python
> --
>
> Key: BEAM-8645
> URL: https://issues.apache.org/jira/browse/BEAM-8645
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: P2
> Labels: stale-assigned
> Fix For: 2.21.0
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> When we have a TimestampValue on combine:
> {code:java}
> main_stream = (p
> | 'main TestStream' >> TestStream()
> .add_elements([window.TimestampedValue(('k', 100), 0)])
> .add_elements([window.TimestampedValue(('k', 400), 9)])
> .advance_watermark_to_infinity()
> | 'main windowInto' >> beam.WindowInto(
> window.FixedWindows(10),
> timestamp_combiner=TimestampCombiner.OUTPUT_AT_LATEST) |
> 'Combine' >> beam.CombinePerKey(sum))
> The expect timestamp should be:
> LATEST: (('k', 500), Timestamp(9)),
> EARLIEST: (('k', 500), Timestamp(0)),
> END_OF_WINDOW: (('k', 500), Timestamp(10)),
> But current py streaming gives following results:
> LATEST: (('k', 500), Timestamp(10)),
> EARLIEST: (('k', 500), Timestamp(10)),
> END_OF_WINDOW: (('k', 500), Timestamp(9.)),
> More details and discussions:
> https://lists.apache.org/thread.html/d3af1f2f84a2e59a747196039eae77812b78a991f0f293c717e5f4e1@%3Cdev.beam.apache.org%3E
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-6446) Clean up suppression rules in checkstyle suppressions.xml
[ https://issues.apache.org/jira/browse/BEAM-6446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruoyun Huang reassigned BEAM-6446: -- Assignee: (was: Ruoyun Huang) > Clean up suppression rules in checkstyle suppressions.xml > - > > Key: BEAM-6446 > URL: https://issues.apache.org/jira/browse/BEAM-6446 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Ruoyun Huang >Priority: P3 > Labels: stale-assigned > Time Spent: 1h 10m > Remaining Estimate: 0h > > When violations are addressed, clean up suppression rules in checkstyle > suppressions.xml -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8632) Beam Dependency Update Request: com.alibaba:fastjson
[ https://issues.apache.org/jira/browse/BEAM-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8632: Labels: stale-P2 (was: ) > Beam Dependency Update Request: com.alibaba:fastjson > > > Key: BEAM-8632 > URL: https://issues.apache.org/jira/browse/BEAM-8632 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-12 22:46:37.461771 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:02:17.756868 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:05:39.000472 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:06:07.944124 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:06:17.991295 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:01:44.683424 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:05:19.226837 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:05:40.928498 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:05:26.160069 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:06:02.225608 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:08:29.656259 > - > Please consider upgrading the dependency com.alibaba:fastjson. > The current version is 1.2.49. The latest version is 1.2.62 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Updated] (BEAM-8766) Beam Dependency Update Request: org.apache.activemq:activemq-kahadb-store
[ https://issues.apache.org/jira/browse/BEAM-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8766: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.apache.activemq:activemq-kahadb-store > - > > Key: BEAM-8766 > URL: https://issues.apache.org/jira/browse/BEAM-8766 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-19 21:10:17.844032 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.10 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:16:39.931067 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:15:55.631623 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:16:17.867202 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:11:27.413019 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:15:05.522684 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:14:26.376645 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:11:54.318743 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:12:50.498297 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:14:04.914305 > - > Please consider upgrading the dependency > org.apache.activemq:activemq-kahadb-store. > The current version is 5.13.1. The latest version is 5.15.11 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-10
[jira] [Commented] (BEAM-8112) Support passing stateBackend through pipeline options in python sdks
[
https://issues.apache.org/jira/browse/BEAM-8112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122662#comment-17122662
]
Beam JIRA Bot commented on BEAM-8112:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Support passing stateBackend through pipeline options in python sdks
>
>
> Key: BEAM-8112
> URL: https://issues.apache.org/jira/browse/BEAM-8112
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: Catlyn Kong
>Priority: P2
> Labels: stale-P2
>
> Currently the only way for python sdks to instruct flink to use a
> StateBackend different than the default (MemoryStateBackend) would be to
> specify state.backend in flink-conf.yaml, which creates the limitation of
> using the same statebackend for every job running on the same flink cluster.
> Ideally we should be able to pass it in to flink runner through
> PipelineOptions. Here's the error it spits out when I flag
> --state_backend=RocksDBStateBackend:
>
> {code:java}
> RuntimeError: Pipeline failed in state FAILED:
> com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot
> construct instance of `org.apache.flink.runtime.state.StateBackend` (no
> Creators, like default construct, exist): abstract types either need to be
> mapped to concrete types, have custom deserializer, or contain additional
> type information
> at [Source: (String)""RocksDBStateBackend""; line: 1, column: 1]
> {code}
> Acceptance Criteria:
> Flink StateBackend is configurable via command line options from python.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-8706) Beam Dependency Update Request: org.antlr
[ https://issues.apache.org/jira/browse/BEAM-8706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8706: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.antlr > - > > Key: BEAM-8706 > URL: https://issues.apache.org/jira/browse/BEAM-8706 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-15 19:43:02.726044 > - > Please consider upgrading the dependency org.antlr. > The current version is None. The latest version is None > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8772) Beam Dependency Update Request: org.apache.httpcomponents:httpcore
[ https://issues.apache.org/jira/browse/BEAM-8772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122500#comment-17122500 ] Beam JIRA Bot commented on BEAM-8772: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam Dependency Update Request: org.apache.httpcomponents:httpcore > -- > > Key: BEAM-8772 > URL: https://issues.apache.org/jira/browse/BEAM-8772 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-19 21:13:03.054236 > - > Please consider upgrading the dependency > org.apache.httpcomponents:httpcore. > The current version is 4.4.10. The latest version is 4.4.12 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:19:49.622331 > - > Please consider upgrading the dependency > org.apache.httpcomponents:httpcore. > The current version is 4.4.10. The latest version is 4.4.12 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:19:03.787573 > - > Please consider upgrading the dependency > org.apache.httpcomponents:httpcore. > The current version is 4.4.10. The latest version is 4.4.12 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:19:26.685211 > - > Please consider upgrading the dependency > org.apache.httpcomponents:httpcore. > The current version is 4.4.10. The latest version is 4.4.12 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8077) CONCAT function is broken
[ https://issues.apache.org/jira/browse/BEAM-8077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8077: Labels: stale-P2 (was: ) > CONCAT function is broken > - > > Key: BEAM-8077 > URL: https://issues.apache.org/jira/browse/BEAM-8077 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Priority: P2 > Labels: stale-P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6177) AfterProcessingTime not firing
[
https://issues.apache.org/jira/browse/BEAM-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-6177:
Labels: stale-P2 (was: )
> AfterProcessingTime not firing
> --
>
> Key: BEAM-6177
> URL: https://issues.apache.org/jira/browse/BEAM-6177
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.8.0
>Reporter: Arnaud T
>Priority: P2
> Labels: stale-P2
>
> Hi,
>
> Documentation says that a AfterProcessingTime(X) trigger should fire X
> seconds after the first element is processed, but it appears that this
> trigger never fires when using a Global window on a steady influx of elements.
> Here is my pipeline:
>
> {code:java}
> (p
> | 'pubsub' >>
> beam.io.ReadFromPubSub(topic=self._pubsub_topic).with_output_types(bytes)
> | 'window' >> beam.WindowInto(
> window.GlobalWindows(),
> trigger=Repeatedly(AfterProcessingTime(5)),
> accumulation_mode=AccumulationMode.DISCARDING
> )
> | 'decode' >> beam.FlatMap(converter.from_pubsub).with_output_types(PubSubRow)
> | 'row' >> beam.Map(lambda x: converter.to_clickhouse(x.type, x.data))
> | 'combine' >> beam.CombineGlobally(ListCombineFn()).without_defaults()
> | 'clickhouse' >> ClickHouseSink(self._clickhouse_host,
> self._clickhouse_port,self._clickhouse_database)
> )
> {code}
>
>
> I expect that every 5 seconds (as long as elements are pouring in), the
> trigger would fire and my data would be combined. The idea of this pipeline
> is simply to get messages from PubSub, transform them into ClickHouse ORM
> models and then batch save them into ClickHouse, using as much parallelism as
> possible - we do not care about order, etc... Elements can be inserted in any
> order and are not correlated to one another.
> The potential issue is in _class
> AfterProcessingTime(TriggerFn)::on_element(self, element, window, context) in
> trigger.py_:
>
> {code:java}
> context.set_timer(
> '', TimeDomain.REAL_TIME, context.get_current_time() + self.delay)
> {code}
> This will basically override the previously set timer every time a new
> element comes in, and in the case of a constant influx of elements, the
> trigger only fires once we have no more elements for X seconds.
>
>
> Please let me know if I understood the documentation right, and if I can
> further help.
>
> Thanks you,
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8135) Remove obsolete BigQuery publishers
[ https://issues.apache.org/jira/browse/BEAM-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122659#comment-17122659 ] Beam JIRA Bot commented on BEAM-8135: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Remove obsolete BigQuery publishers > --- > > Key: BEAM-8135 > URL: https://issues.apache.org/jira/browse/BEAM-8135 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kamil Wasilewski >Priority: P2 > Labels: stale-P2 > > After creating Prometheus metrics publisher, BQ publishers would become > obsolete and should be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8644) Beam Dependency Update Request: org.freemarker:freemarker
[ https://issues.apache.org/jira/browse/BEAM-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-8644: Labels: stale-P2 (was: ) > Beam Dependency Update Request: org.freemarker:freemarker > - > > Key: BEAM-8644 > URL: https://issues.apache.org/jira/browse/BEAM-8644 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2019-11-12 22:54:18.472454 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:15:39.185121 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:22:41.530326 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:22:15.167965 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:22:53.148923 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:17:29.309875 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:21:01.972648 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:20:30.044886 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:18:07.682746 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:19:07.123214 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-02-03 12:20:25.404910 > - > Please consider upgrading the dependency org.freemarker:freemarker. > The current version is 2.3.28. The latest version is 2.3.29 > cc: > Please refer to [Beam
[jira] [Commented] (BEAM-7127) Timer parameter not supported in timer callback
[
https://issues.apache.org/jira/browse/BEAM-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122851#comment-17122851
]
Beam JIRA Bot commented on BEAM-7127:
-
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> Timer parameter not supported in timer callback
> ---
>
> Key: BEAM-7127
> URL: https://issues.apache.org/jira/browse/BEAM-7127
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-harness
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Priority: P2
> Labels: portability, stale-P2
>
> Referencing the timer in its on_timer callback produces a recursive pickle
> error.
> {code:java}
> @userstate.on_timer(timer_spec)
> def process_timer(self, timer_1=beam.DoFn.TimerParam(timer_spec)):
> {code}
> Unit test:
> [https://github.com/apache/beam/blob/cbe4dfbdbe5d0da5152568853ee5e17334dd1b54/sdks/python/apache_beam/transforms/userstate_test.py#L67]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-6837) Portable Validates Runner Tests for Python Reference Runner
[ https://issues.apache.org/jira/browse/BEAM-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122889#comment-17122889 ] Beam JIRA Bot commented on BEAM-6837: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Portable Validates Runner Tests for Python Reference Runner > --- > > Key: BEAM-6837 > URL: https://issues.apache.org/jira/browse/BEAM-6837 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Priority: P2 > Labels: stale-P2 > > PVR are needed to make sure our reference runner stays in a healthy state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5530) Migrate to java.time lib instead of joda-time
[ https://issues.apache.org/jira/browse/BEAM-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123129#comment-17123129 ] Beam JIRA Bot commented on BEAM-5530: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Migrate to java.time lib instead of joda-time > - > > Key: BEAM-5530 > URL: https://issues.apache.org/jira/browse/BEAM-5530 > Project: Beam > Issue Type: Improvement > Components: dependencies, sdk-java-core >Reporter: Alexey Romanenko >Priority: P2 > Labels: stale-P2 > Fix For: 3.0.0 > > > Joda-time has been used till moving to Java 8. For now, these two time > libraries are used together. It will make sense finally to move everywhere to > only one lib - *java.time* - as a standard Java time library (see mail list > discussion: > [https://lists.apache.org/thread.html/b10f6f9daed44f5fa65e315a44b68b2f57c3e80225f5d549b84918af@%3Cdev.beam.apache.org%3E]). > > Since this migration will introduce breaking API changes, then we should > address it to 3.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6671) Beam 2.9.0 java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
[ https://issues.apache.org/jira/browse/BEAM-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17122922#comment-17122922 ] Beam JIRA Bot commented on BEAM-6671: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Beam 2.9.0 java.lang.NoSuchFieldError: > internal_static_google_rpc_LocalizedMessage_fieldAccessorTable > - > > Key: BEAM-6671 > URL: https://issues.apache.org/jira/browse/BEAM-6671 > Project: Beam > Issue Type: New Feature > Components: build-system >Reporter: Alex Amato >Priority: P2 > Labels: stale-P2 > > I received a report from a Dataflow user encountering this in Beam 2.9.0 when > creating a spanner instance. I wanted to post this here as this is known to > be related to dependency conflicts in the past > ([https://stackoverflow.com/questions/46684071/error-using-spannerio-in-apache-beam]). > > java.lang.NoSuchFieldError: > internal_static_google_rpc_LocalizedMessage_fieldAccessorTable > at > com.google.rpc.LocalizedMessage.internalGetFieldAccessorTable(LocalizedMessage.java:90) > at > com.google.protobuf.GeneratedMessageV3.getDescriptorForType(GeneratedMessageV3.java:121) > at io.grpc.protobuf.ProtoUtils.keyForProto(ProtoUtils.java:67) > at > com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor.(SpannerErrorInterceptor.java:47) > at > com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.(GrpcSpannerRpc.java:136) > at > com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:73) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7301) Beam transforms reorder fields
[ https://issues.apache.org/jira/browse/BEAM-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7301: Labels: stale-P2 (was: ) > Beam transforms reorder fields > -- > > Key: BEAM-7301 > URL: https://issues.apache.org/jira/browse/BEAM-7301 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Reuven Lax >Priority: P2 > Labels: stale-P2 > > Currently transforms such as Select, DropFields, RenameFields, and AddFields > can create schemas with unexpected order. The problem is that > FieldAccessDescriptor stores top-level fields and nested fields separately, > so there's no way to tell the relative order between them. To fix this we > should refactor FieldAccessDescriptor: instead of storing these separately it > should store a single list, where each item in the list might optionally have > a nested FieldAccessDescriptor. > Expected behavior from the transforms: > DropFields: preserves order in original schema > RenameFields: preserves order in original schema > AddFields: adds fields in order specified. If multiple nested fields are > selected, the first reference to the top field wins (e.g. adding "a.b", "c", > "a.d" results in adding a before c. > Select: Select fields in the order specified. -- This message was sent by Atlassian Jira (v8.3.4#803005)
