[jira] [Assigned] (BEAM-5496) MqttIO fails to deserialize checkpoint

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5496:
---

Assignee: Island Chen

> MqttIO fails to deserialize checkpoint
> --
>
> Key: BEAM-5496
> URL: https://issues.apache.org/jira/browse/BEAM-5496
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-mqtt
>Reporter: Luke Cwik
>Assignee: Island Chen
>Priority: Major
>
> Source of bug report: 
> [https://lists.apache.org/thread.html/3de5a946bcb539dea9f18a31f712d6af5b66f9fbb6b01eed452c5afb@%3Cdev.beam.apache.org%3E]
>  
> There is a bug of the built-in MqttIO, please check the 
> ,
>  this readObject() method forget to invoke the "stream.defaultReadObject()" 
> method.
>  
> {code:java}
> // set an empty list to messages when deserialize
> private void readObject(java.io.ObjectInputStream stream)
> throws IOException, ClassNotFoundException {
>   messages = new ArrayList<>();
> }{code}
>  
> So there is an exception while the runner tried to deserialize the checkpoint 
> object.
> {code:java}
> java.lang.RuntimeException: org.apache.beam.sdk.coders.CoderException: 95 
> unexpected extra bytes after decoding 
> org.apache.beam.sdk.io.mqtt.MqttIO$MqttCheckpointMark@6764e219
>  at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:340)
> ...{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5366) Vendor gRPC and Protobuf separately from beam-model-* Java packages

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5366:
---

Assignee: Luke Cwik  (was: Kenneth Knowles)

> Vendor gRPC and Protobuf separately from beam-model-* Java packages
> ---
>
> Key: BEAM-5366
> URL: https://issues.apache.org/jira/browse/BEAM-5366
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, build-system
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
>
> Each of the beam-model-* jars currently contains duplicate definitions of 
> gRPC and protobuf and their transitive dependencies.
>  
> By migrating those packages to a separate artifact, we could have a single 
> copy of those classes instead of 3.
>  
> This would reduce the size of the jars and prevent warnings such as:
>  
> {code:java}
> [WARNING] beam-model-fn-execution-2.5.0.jar, 
> beam-model-job-management-2.5.0.jar, beam-model-pipeline-2.5.0.jar define 
> 6660 overlapping classes:  [WARNING]   - 
> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1 
> [WARNING]   - 
> org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
>  [WARNING]   - 
> org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners
>  [WARNING]   - 
> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1 
> [WARNING]   - 
> org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4 
> [WARNING]   - 
> org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
>  [WARNING]   - 
> org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders 
> [WARNING]   - 
> org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
>  [WARNING]   - 
> org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap 
> [WARNING]   - 
> org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5496) MqttIO fails to deserialize checkpoint

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5496:
---

 Summary: MqttIO fails to deserialize checkpoint
 Key: BEAM-5496
 URL: https://issues.apache.org/jira/browse/BEAM-5496
 Project: Beam
  Issue Type: Bug
  Components: io-java-mqtt
Reporter: Luke Cwik


Source of bug report: 
[https://lists.apache.org/thread.html/3de5a946bcb539dea9f18a31f712d6af5b66f9fbb6b01eed452c5afb@%3Cdev.beam.apache.org%3E]
 

There is a bug of the built-in MqttIO, please check the 
,
 this readObject() method forget to invoke the "stream.defaultReadObject()" 
method.

 
{code:java}
// set an empty list to messages when deserialize
private void readObject(java.io.ObjectInputStream stream)
throws IOException, ClassNotFoundException {
  messages = new ArrayList<>();
}{code}
 


So there is an exception while the runner tried to deserialize the checkpoint 
object.
{code:java}
java.lang.RuntimeException: org.apache.beam.sdk.coders.CoderException: 95 
unexpected extra bytes after decoding 
org.apache.beam.sdk.io.mqtt.MqttIO$MqttCheckpointMark@6764e219
 at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:340)
...{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5468) Allow runner to set worker log level in Python SDK harness.

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626615#comment-16626615
 ] 

Luke Cwik commented on BEAM-5468:
-

Typically the user building the pipeline would want to control the log level on 
the SDK harness. The Java SDK harness does allow setting the log levels per:

[https://github.com/apache/beam/blob/f048354619f30c1dfaf7207a12c1f4ec2b10d4c6/sdks/java/core/src/main/java/org/apache/beam/sdk/options/SdkHarnessOptions.java#L56]

 

Logging is able to be configured globally and also per namespace. It would be 
convenient if Python used the same design of global + namespace log levels.

> Allow runner to set worker log level in Python SDK harness.
> ---
>
> Key: BEAM-5468
> URL: https://issues.apache.org/jira/browse/BEAM-5468
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5488) DirectRunner not producing output on TextIO withWindowedWrites() and withNumShards(1)

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5488:

Component/s: runner-direct

> DirectRunner not producing output on TextIO withWindowedWrites() and 
> withNumShards(1)
> -
>
> Key: BEAM-5488
> URL: https://issues.apache.org/jira/browse/BEAM-5488
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct, sdk-java-core
>Reporter: Luke Cwik
>Priority: Major
>
> Source of bug (Slack user [https://the-asf.slack.com/team/UCVN8DK7V]) and 
> [https://stackoverflow.com/questions/52445414/apache-beam-not-saving-unbounded-data-to-text-file].
> Example provided:
> {code:java}
> public static void main(String[] args) {
> ExerciseOptions options = 
> PipelineOptionsFactory.fromArgs(args).withValidation().as(ExerciseOptions.class);
> Pipeline pipeline = Pipeline.create(options);
> pipeline
>   .apply("Read Messages from Pubsub",
> PubsubIO
>   .readStrings()
>   .fromTopic(options.getTopicName()))
>   .apply("Set event timestamp", ParDo.of(new DoFn() {
> @ProcessElement
> public void processElement(ProcessContext context) {
>   context.outputWithTimestamp(context.element(), Instant.now());
> }
>   }))
>   .apply("Windowing", 
> Window.into(FixedWindows.of(Duration.standardMinutes(5
>   .apply("Write to File",
> TextIO
>   .write()
>   .withWindowedWrites()
>   .withNumShards(1)
>   .to(options.getOutputPrefix()));
> pipeline.run();
>   }
> {code}
> Produces output when executed on the DataflowRunner, does not produce output 
> on the DirectRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5488) DirectRunner not producing output on TextIO withWindowedWrites() and withNumShards(1)

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5488:
---

 Summary: DirectRunner not producing output on TextIO 
withWindowedWrites() and withNumShards(1)
 Key: BEAM-5488
 URL: https://issues.apache.org/jira/browse/BEAM-5488
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Luke Cwik


Source of bug (Slack user [https://the-asf.slack.com/team/UCVN8DK7V]) and 
[https://stackoverflow.com/questions/52445414/apache-beam-not-saving-unbounded-data-to-text-file].

Example provided:
{code:java}
public static void main(String[] args) {
ExerciseOptions options = 
PipelineOptionsFactory.fromArgs(args).withValidation().as(ExerciseOptions.class);

Pipeline pipeline = Pipeline.create(options);

pipeline
  .apply("Read Messages from Pubsub",
PubsubIO
  .readStrings()
  .fromTopic(options.getTopicName()))

  .apply("Set event timestamp", ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext context) {
  context.outputWithTimestamp(context.element(), Instant.now());
}
  }))

  .apply("Windowing", 
Window.into(FixedWindows.of(Duration.standardMinutes(5

  .apply("Write to File",
TextIO
  .write()
  .withWindowedWrites()
  .withNumShards(1)
  .to(options.getOutputPrefix()));

pipeline.run();
  }
{code}
Produces output when executed on the DataflowRunner, does not produce output on 
the DirectRunner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5487) ByteKeyRangeTracker restrictions do not cover the entire interval because of incorrect next key

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5487:

Description: 
The definition of next for a byte key is incorrectly set as old key + 1 when 
treating the key as the next smallest value, for example

0x01 is the next key for 0x00

 

The ByteKey compareTo operator treats two keys where one key is the proper 
prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
0x thus next is skipping over 0x as the next smallest key larger then 
0x00.

  was:
The definition of next for a byte key is incorrectly set as old key + 1 when 
treating the key as the next largest value, for example

0x01 is the next key for 0x00

 

The ByteKey compareTo operator treats two keys where one key is the proper 
prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
0x thus next is skipping over 0x as the next smallest key larger then 
0x00.


> ByteKeyRangeTracker restrictions do not cover the entire interval because of 
> incorrect next key
> ---
>
> Key: BEAM-5487
> URL: https://issues.apache.org/jira/browse/BEAM-5487
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The definition of next for a byte key is incorrectly set as old key + 1 when 
> treating the key as the next smallest value, for example
> 0x01 is the next key for 0x00
>  
> The ByteKey compareTo operator treats two keys where one key is the proper 
> prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
> 0x thus next is skipping over 0x as the next smallest key larger then 
> 0x00.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5487) ByteKeyRangeTracker restrictions do not cover the entire interval because of incorrect next key

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5487:

Summary: ByteKeyRangeTracker restrictions do not cover the entire interval 
because of incorrect next key  (was: Splitting ByteKeyRangeTracker restrictions 
do not cover the entire interval)

> ByteKeyRangeTracker restrictions do not cover the entire interval because of 
> incorrect next key
> ---
>
> Key: BEAM-5487
> URL: https://issues.apache.org/jira/browse/BEAM-5487
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>
> The definition of next for a byte key is incorrectly set as old key + 1 when 
> treating the key as the next largest value, for example
> 0x01 is the next key for 0x00
>  
> The ByteKey compareTo operator treats two keys where one key is the proper 
> prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
> 0x thus next is skipping over 0x as the next smallest key larger then 
> 0x00.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5487) Splitting ByteKeyRangeTracker restrictions do not cover the entire interval

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5487:

Description: 
The definition of next for a byte key is incorrectly set as old key + 1 when 
treating the key as the next largest value, for example

0x01 is the next key for 0x00

 

The ByteKey compareTo operator treats two keys where one key is the proper 
prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
0x thus next is skipping over 0x as the next smallest key larger then 
0x00.

  was:
The definition of next for a byte key is incorrectly set as +1 when treating 
the key as the next largest value, for example

0x01 is the next key for 0x00

 

The ByteKey compareTo operator treats two keys where one key is the proper 
prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
0x thus next is skipping over 0x as the next smallest key larger then 
0x00.


> Splitting ByteKeyRangeTracker restrictions do not cover the entire interval
> ---
>
> Key: BEAM-5487
> URL: https://issues.apache.org/jira/browse/BEAM-5487
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>
> The definition of next for a byte key is incorrectly set as old key + 1 when 
> treating the key as the next largest value, for example
> 0x01 is the next key for 0x00
>  
> The ByteKey compareTo operator treats two keys where one key is the proper 
> prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
> 0x thus next is skipping over 0x as the next smallest key larger then 
> 0x00.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5487) Splitting ByteKeyRangeTracker restrictions do not cover the entire interval

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5487:
---

 Summary: Splitting ByteKeyRangeTracker restrictions do not cover 
the entire interval
 Key: BEAM-5487
 URL: https://issues.apache.org/jira/browse/BEAM-5487
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik


The definition of next for a byte key is incorrectly set as +1 when treating 
the key as the next largest value, for example

0x01 is the next key for 0x00

 

The ByteKey compareTo operator treats two keys where one key is the proper 
prefix of the other as smaller hence 0x01 is greater then 0x00, but so is 
0x thus next is skipping over 0x as the next smallest key larger then 
0x00.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5126) PreCommit filtering broken based upon PR contents

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624190#comment-16624190
 ] 

Luke Cwik commented on BEAM-5126:
-

I'm not certain as to whether its broken or not anymore.

> PreCommit filtering broken based upon PR contents
> -
>
> Key: BEAM-5126
> URL: https://issues.apache.org/jira/browse/BEAM-5126
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Minor
>
> PR precommits used to be filtered by the contents of the PR.
>  
> Example PR that should have only spawned the Java PreCommit:
> https://github.com/apache/beam/pull/6159
>  
> This broke work done in BEAM-4445



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5461) Upgrade Gradle to 4.10

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5461.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Upgrade Gradle to 4.10
> --
>
> Key: BEAM-5461
> URL: https://issues.apache.org/jira/browse/BEAM-5461
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Scott Wegner
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We will get some improvements in build time.
>  
> I don't believe there are any features that we are explicitly looking for.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5446) SplittableDoFn: Remove runner time execution information from public API surface

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5446:
---

 Summary: SplittableDoFn: Remove runner time execution information 
from public API surface
 Key: BEAM-5446
 URL: https://issues.apache.org/jira/browse/BEAM-5446
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik


Move the setting of "claim observers" within RestrictionTracker to another 
location to clean up the RestrictionTracker interface.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4711) LocalFileSystem.delete doesn't support globbing

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4711.
-
   Resolution: Fixed
Fix Version/s: 2.8.0

> LocalFileSystem.delete doesn't support globbing
> ---
>
> Key: BEAM-4711
> URL: https://issues.apache.org/jira/browse/BEAM-4711
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Affects Versions: 2.5.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
> Fix For: 2.8.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> I attempted to run {{wordcount_it_test:WordCountIT.test_wordcount_it}} 
> locally with {{DirectRunner}}:
> {code}
> python setup.py nosetests \
>   --tests 
> apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
>   --test-pipeline-options="--output=foo"
> {code}
> It failed in [the {{delete_files}} cleanup 
> command|https://github.com/apache/beam/blob/a58f1ffaafb0e2ebcc73a1c5abfb05a15ec6a84b/sdks/python/apache_beam/examples/wordcount_it_test.py#L64]:
> {code}
> root: WARNING: Retry with exponential backoff: waiting for 11.1454450937 
> seconds before retrying delete_files because we caught exception: 
> BeamIOError: Delete operation failed with exceptions 
> {'foo/1530557644/results*': IOError(OSError(2, 'No such file or directory'),)}
>  Traceback for above exception (most recent call last):
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/utils/retry.py", line 184, 
> in wrapper
> return fun(*args, **kwargs)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/testing/test_utils.py", 
> line 136, in delete_files
> FileSystems.delete(file_paths)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/io/filesystems.py", line 
> 282, in delete
> return filesystem.delete(paths)
>   File "/Users/ryan/c/beam/sdks/python/apache_beam/io/localfilesystem.py", 
> line 304, in delete
> raise BeamIOError("Delete operation failed", exceptions)
> {code}
> The line:
> {code}
> self.addCleanup(delete_files, [output + '*'])
> {code}
> works as expected in GCS, and deletes a test's output-directory, but it fails 
> in on the local-filesystem, which doesn't expand globs before attempting to 
> delete paths.
> It would be good to make these consistent, presumably by adding glob-support 
> to {{LocalFileSystem}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5425) FileSystems contract rename/delete to allow prior partial successes to make continued progress

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5425:
---

Assignee: Ahmet Altay  (was: Kenneth Knowles)

> FileSystems contract rename/delete to allow prior partial successes to make 
> continued progress
> --
>
> Key: BEAM-5425
> URL: https://issues.apache.org/jira/browse/BEAM-5425
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ahmet Altay
>Priority: Major
>
> The filesystems contract for delete/rename say that an error should be raised 
> if the resources don't exist.
>  
> I believe the contract should be updated to not have failures if the 
> resources don't exist as we want them to be retried on failure without 
> needing the caller know that a prior call may have been partially successful.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5425) FileSystems contract rename/delete to allow prior partial successes to make continued progress

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5425:
---

 Summary: FileSystems contract rename/delete to allow prior partial 
successes to make continued progress
 Key: BEAM-5425
 URL: https://issues.apache.org/jira/browse/BEAM-5425
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core, sdk-py-core
Reporter: Luke Cwik
Assignee: Kenneth Knowles


The filesystems contract for delete/rename say that an error should be raised 
if the resources don't exist.

 

I believe the contract should be updated to not have failures if the resources 
don't exist as we want them to be retried on failure without needing the caller 
know that a prior call may have been partially successful.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4843) Incorrect docs on FileSystems.delete

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4843.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Incorrect docs on FileSystems.delete
> 
>
> Key: BEAM-4843
> URL: https://issues.apache.org/jira/browse/BEAM-4843
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0
>Reporter: Ryan Williams
>Assignee: Ryan Williams
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [The docs on {{FileSystems.delete}} 
> say|https://github.com/apache/beam/blob/b5e8335d982ee69d9f788f65f27356cddd5293d1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystems.java#L332-L333]:
> bq. It is allowed but not recommended to delete directories recursively. 
> Callers depends on {@link FileSystems} and uses {@code DeleteOptions}.
> However, the function actually takes a {{MoveOptions...}} param, there's 
> never been a {{DeleteOptions}} afaict, and there is no way to recursively 
> delete a {{ResourceId}}.
> The docs should be fixed, at a minimum; actually supporting recursive delete 
> would also be nice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5406) NullPointerException when converting Null Datetime to TableRow

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5406:

Affects Version/s: 2.7.0

> NullPointerException when converting Null Datetime to TableRow
> --
>
> Key: BEAM-5406
> URL: https://issues.apache.org/jira/browse/BEAM-5406
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Andrew Kujtan
>Assignee: Andrew Kujtan
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> BigQueryUtils.toTableRow() throws a NPE for null Datetime column data. It 
> should simply return null.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5406) NullPointerException when converting Null Datetime to TableRow

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5406:
---

Assignee: Andrew Kujtan  (was: Chamikara Jayalath)

> NullPointerException when converting Null Datetime to TableRow
> --
>
> Key: BEAM-5406
> URL: https://issues.apache.org/jira/browse/BEAM-5406
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Andrew Kujtan
>Assignee: Andrew Kujtan
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> BigQueryUtils.toTableRow() throws a NPE for null Datetime column data. It 
> should simply return null.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5406) NullPointerException when converting Null Datetime to TableRow

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5406.
-
   Resolution: Fixed
Fix Version/s: 2.8.0

> NullPointerException when converting Null Datetime to TableRow
> --
>
> Key: BEAM-5406
> URL: https://issues.apache.org/jira/browse/BEAM-5406
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.6.0
>Reporter: Andrew Kujtan
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> BigQueryUtils.toTableRow() throws a NPE for null Datetime column data. It 
> should simply return null.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5351:

Fix Version/s: (was: 2.8.0)
   2.7.0

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5351.
-
Resolution: Fixed

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5351:
---

Assignee: Luke Cwik  (was: Charles Chen)

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2395) BigtableIO for Python SDK

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik closed BEAM-2395.
---
   Resolution: Duplicate
Fix Version/s: Not applicable

> BigtableIO for Python SDK
> -
>
> Key: BEAM-2395
> URL: https://issues.apache.org/jira/browse/BEAM-2395
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Matthias Baetens
>Assignee: Matthias Baetens
>Priority: Major
>  Labels: features
> Fix For: Not applicable
>
>
> Developing a read and write IO for BigTable for the Python SDK. 
> Working / design document can be found here: 
> https://docs.google.com/document/d/1iXeQvIAsGjp9orleDy0o5ExU-eMqWesgvtt231UoaPg/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3342) Create a Cloud Bigtable Python connector

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612781#comment-16612781
 ] 

Luke Cwik commented on BEAM-3342:
-

Note that I closed the duplicate of this issue but wanted to capture the design 
doc proposal form that JIRA: 
[https://docs.google.com/document/d/1iXeQvIAsGjp9orleDy0o5ExU-eMqWesgvtt231UoaPg/edit?usp=sharing]

> Create a Cloud Bigtable Python connector
> 
>
> Key: BEAM-3342
> URL: https://issues.apache.org/jira/browse/BEAM-3342
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>Priority: Major
>
> I would like to create a Cloud Bigtable python connector.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5366) Vendor gRPC and Protobuf separately from beam-model-* Java packages

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5366:
---

 Summary: Vendor gRPC and Protobuf separately from beam-model-* 
Java packages
 Key: BEAM-5366
 URL: https://issues.apache.org/jira/browse/BEAM-5366
 Project: Beam
  Issue Type: Improvement
  Components: beam-model, build-system
Reporter: Luke Cwik
Assignee: Kenneth Knowles


Each of the beam-model-* jars currently contains duplicate definitions of gRPC 
and protobuf and their transitive dependencies.

 

By migrating those packages to a separate artifact, we could have a single copy 
of those classes instead of 3.

 

This would reduce the size of the jars and prevent warnings such as:
 
{code:java}
[WARNING] beam-model-fn-execution-2.5.0.jar, 
beam-model-job-management-2.5.0.jar, beam-model-pipeline-2.5.0.jar define 6660 
overlapping classes:  [WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1 
[WARNING]   - 
org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper
 [WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners 
[WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1 
[WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4 [WARNING]  
 - 
org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder
 [WARNING]   - 
org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders 
[WARNING]   - 
org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder
 [WARNING]   - 
org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap 
[WARNING]   - 
org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3098) Upgrade Java grpc version

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611006#comment-16611006
 ] 

Luke Cwik commented on BEAM-3098:
-

[~kbolton], the 2.7.0 release is ongoing right now. You can follow progress on 
the [d...@beam.apache.org|mailto:d...@beam.apache.org] list.

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5299) Define max global window as a shared value in protos like URN enums.

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610873#comment-16610873
 ] 

Luke Cwik commented on BEAM-5299:
-

I'm not exactly following but what if you did something like:

 
{code:java}
message Constants {
  enum Constants {
// Explanation of constant
    CONSTANT_VALUE = 0 [(beam_constant) = "1234567890"];
  }
}

{code}
 

We could update all the existing *beam_urn* to be *beam_constant* so that we 
have a general way of looking up any string metadata.

 

> Define max global window as a shared value in protos like URN enums.
> 
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
>  Labels: portability
>
> Instead of having each language define a max timestamp themselves, define the 
> max timestamps within proto to be shared across different languages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reopened BEAM-5351:
-

Please close and signal whether this was included in 2.7.0 or not

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.8.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5351.
-
   Resolution: Fixed
Fix Version/s: 2.8.0

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.8.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5351:

Description: It seems as though **generatePomFileForMavenJavaPublication** 
pom.xml was renamed to pom-default.xml as output which broke the the shadowJar 
task 
[here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Attachments: beam-examples-java-2.7.0.jar
>
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5351) 2.7.0 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5351:
---

 Summary: 2.7.0 jars missing 
META-INF/maven/groupId/artifactId/pom.xml
 Key: BEAM-5351
 URL: https://issues.apache.org/jira/browse/BEAM-5351
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 2.7.0
Reporter: Luke Cwik
Assignee: Charles Chen
 Attachments: beam-examples-java-2.7.0.jar





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5351:

Summary: 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml  
(was: 2.7.0 jars missing META-INF/maven/groupId/artifactId/pom.xml)

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Attachments: beam-examples-java-2.7.0.jar
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5351:

Attachment: beam-examples-java-2.7.0.jar

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Attachments: beam-examples-java-2.7.0.jar
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5299) Define max global window as a shared value in protos like URN enums.

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609621#comment-16609621
 ] 

Luke Cwik commented on BEAM-5299:
-

If you mean a lookup like [how 
this|https://github.com/apache/beam/blob/9b68f926628d727e917b6a33ccdafcfe693eef6a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java#L26]
 is used, that would be great. If you are talking about something else, feel 
free to give me a snippet as I'm not following by what else you could mean by 
lookup at runtime.

> Define max global window as a shared value in protos like URN enums.
> 
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
>  Labels: portability
>
> Instead of having each language define a max timestamp themselves, define the 
> max timestamps within proto to be shared across different languages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5328) Java starter archetype does not contain dependency versions

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605008#comment-16605008
 ] 

Luke Cwik commented on BEAM-5328:
-

[~alanmyrvold], can you link to where we test our examples archetypes so that 
they work. Would it be difficult to test the starter archetype as well?

> Java starter archetype does not contain dependency versions
> ---
>
> Key: BEAM-5328
> URL: https://issues.apache.org/jira/browse/BEAM-5328
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Luke Cwik
>Priority: Major
>  Labels: newbie, starter
>
> The starter archetype contains resource annotation markers instead of 
> versions:
> {code:java}
> @maven-compiler-plugin.version@
> @maven-exec-plugin.version@
> @slf4j.version@
> {code}
> in the properties block at the top.
>  
> This means that the starter project is broken without the user manually 
> editing the pom.xml that is generated and populating the versions at the top.
>  
> We also lack testing that validates that the starter archetype works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5328) Java starter archetype does not contain dependency versions

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5328:

Labels: newbie starter  (was: )

> Java starter archetype does not contain dependency versions
> ---
>
> Key: BEAM-5328
> URL: https://issues.apache.org/jira/browse/BEAM-5328
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Luke Cwik
>Priority: Major
>  Labels: newbie, starter
>
> The starter archetype contains resource annotation markers instead of 
> versions:
> {code:java}
> @maven-compiler-plugin.version@
> @maven-exec-plugin.version@
> @slf4j.version@
> {code}
> in the properties block at the top.
>  
> This means that the starter project is broken without the user manually 
> editing the pom.xml that is generated and populating the versions at the top.
>  
> We also lack testing that validates that the starter archetype works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5328) Java starter archetype does not contain dependency versions

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5328:
---

 Summary: Java starter archetype does not contain dependency 
versions
 Key: BEAM-5328
 URL: https://issues.apache.org/jira/browse/BEAM-5328
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 2.6.0, 2.7.0
Reporter: Luke Cwik


The starter archetype contains resource annotation markers instead of versions:
{code:java}
@maven-compiler-plugin.version@
@maven-exec-plugin.version@
@slf4j.version@
{code}
in the properties block at the top.

 

This means that the starter project is broken without the user manually editing 
the pom.xml that is generated and populating the versions at the top.

 

We also lack testing that validates that the starter archetype works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5299) Define max global window as a shared value in protos like URN enums.

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5299:
---

 Summary: Define max global window as a shared value in protos like 
URN enums.
 Key: BEAM-5299
 URL: https://issues.apache.org/jira/browse/BEAM-5299
 Project: Beam
  Issue Type: Improvement
  Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
Reporter: Luke Cwik
Assignee: Maximilian Michels


Instead of having each language define a max timestamp themselves, define the 
max timestamps within proto to be shared across different languages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603274#comment-16603274
 ] 

Luke Cwik commented on BEAM-5288:
-

I would suggest `StandardEnvironments` instead of `SupportedEnvironments` 
because these URNs say that there is a standard that is/will be documented and 
associated with it.

 

`Primitives` is also the wrong term. I would go with `Environments` unless 
something better is suggested.

 

 

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Priority: Major
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment. 
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message SupportedEnvironments {
>enum Primitives {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>repeated string params = 4; // parameters
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5288:

Description: 
As of mailing discussions and BEAM-5187, it has become clear that we need to 
extend the Environment information. In addition to the Docker environment, the 
extended environment holds deployment options for 1) a process-based 
environment, 2) an externally managed environment.

The proto definition, as of now, looks as follows:
{noformat}
 message Environment {

   // (Required) The URN of the payload
   string urn = 1;

   // (Optional) The data specifying any parameters to the URN. If
   // the URN does not require any arguments, this may be omitted.
   bytes payload = 2;
 }

 message StandardEnvironments {
   enum Environments {
 DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];

 PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];

 EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
   }
 }

 // The payload of a Docker image
 message DockerPayload {
   string container_image = 1;  // implicitly linux_amd64.
 }

 message ProcessPayload {
   string os = 1;  // "linux", "darwin", ..
   string arch = 2;  // "amd64", ..
   string command = 3; // process to execute
   repeated string params = 4; // parameters
 }
{noformat}

  was:
As of mailing discussions and BEAM-5187, it has become clear that we need to 
extend the Environment information. In addition to the Docker environment, the 
extended environment holds deployment options for 1) a process-based 
environment, 2) an externally managed environment. 

The proto definition, as of now, looks as follows:

{noformat}
 message Environment {

   // (Required) The URN of the payload
   string urn = 1;

   // (Optional) The data specifying any parameters to the URN. If
   // the URN does not require any arguments, this may be omitted.
   bytes payload = 2;
 }

 message SupportedEnvironments {
   enum Primitives {
 DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];

 PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];

 EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
   }
 }

 // The payload of a Docker image
 message DockerPayload {
   string container_image = 1;  // implicitly linux_amd64.
 }

 message ProcessPayload {
   string os = 1;  // "linux", "darwin", ..
   string arch = 2;  // "amd64", ..
   string command = 3; // process to execute
   repeated string params = 4; // parameters
 }
{noformat}


> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Priority: Major
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>repeated string params = 4; // parameters
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5191) Add support for writing to BigQuery clustered tables

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5191:
---

Assignee: Wout Scheepers  (was: Chamikara Jayalath)

> Add support for writing to BigQuery clustered tables
> 
>
> Key: BEAM-5191
> URL: https://issues.apache.org/jira/browse/BEAM-5191
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.6.0
>Reporter: Robert Sahlin
>Assignee: Wout Scheepers
>Priority: Minor
>  Labels: features, newbie
>
> Google recently added support for clustered tables in BigQuery. It would be 
> useful to set clustering columns the same way as for partitioning. It should 
> support multiple fields (4) for clustering.
> For example:
> [BigQueryIO.Write|https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html]<[T|https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html]>
>  .withClustering(new Clustering().setField("productId").setType("STRING"))



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4418) Improve gradle integration with IntelliJ

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595797#comment-16595797
 ] 

Luke Cwik commented on BEAM-4418:
-

Thanks [~rdub], just was providing context. Note that revisiting those 
decisions may be the right thing if we can't develop code well as a community. 
I am really looking for suggestions since I and others are annoyed by the 
current state of things and I haven't been able to come up with any solutions.

> Improve gradle integration with IntelliJ
> 
>
> Key: BEAM-4418
> URL: https://issues.apache.org/jira/browse/BEAM-4418
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Etienne Chauchot
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> To be able to work efficiently with gradle, the integration with intelliJ 
> (more common IDE in the community I think) needs to be improved.The aim of 
> this ticket is to gather areas of improvement discovered by people. Feel free 
> to comment on what you discovered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5247) Remove slf4j-simple binding from dependencies

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5247:
---

Assignee: Jozef Vilcek  (was: Aljoscha Krettek)

> Remove slf4j-simple binding from dependencies
> -
>
> Key: BEAM-5247
> URL: https://issues.apache.org/jira/browse/BEAM-5247
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Jozef Vilcek
>Assignee: Jozef Vilcek
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Flink runner declares a slf4j-simple binding in dependencies. This can break 
> logging of application if they have their own binding and does not exclude 
> this one from beam.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4418) Improve gradle integration with IntelliJ

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595749#comment-16595749
 ] 

Luke Cwik commented on BEAM-4418:
-

There is a lot of complexity in shading such that the Java artifacts produced 
are compatible with as many runners as possible and simplify dependency 
requirements for users.

I'm all ears for how to improve but vendoring has signficant tangible benefits 
in cleaning up the dependencies we export. I'm all ears for different proposals 
as vendoring has been discussed and agreed upon twice in the past: 

[https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E]

[https://lists.apache.org/thread.html/8b9b3768adfc40d3527d1ce5e8a51d90e5782a348a3abfb9e5dc85ef@%3Cdev.beam.apache.org%3E]

 

> Improve gradle integration with IntelliJ
> 
>
> Key: BEAM-4418
> URL: https://issues.apache.org/jira/browse/BEAM-4418
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Etienne Chauchot
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> To be able to work efficiently with gradle, the integration with intelliJ 
> (more common IDE in the community I think) needs to be improved.The aim of 
> this ticket is to gather areas of improvement discovered by people. Feel free 
> to comment on what you discovered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5184.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5184:

Summary: Multimap side inputs with duplicate keys and values are being lost 
 (was: Side inputs with duplicate values are being lost)

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5184) Side inputs with duplicate values are being lost

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5184:

Description: 
Side inputs with duplicate values are being lost due to the usage of a set 
based multimap.

[https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]

 

Originating thread: 
[https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]

 

Please update the existing tests to exercise this scenario as well: 
https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507

  was:
Side inputs with duplicate values are being lost due to the usage of a set 
based multimap.

[https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]

 

Originating thread: 
https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E


> Side inputs with duplicate values are being lost
> 
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5184) Side inputs with duplicate values are being lost

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5184:
---

Assignee: Vaclav Plajt  (was: Kenneth Knowles)

> Side inputs with duplicate values are being lost
> 
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5184) Side inputs with duplicate values are being lost

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5184:
---

 Summary: Side inputs with duplicate values are being lost
 Key: BEAM-5184
 URL: https://issues.apache.org/jira/browse/BEAM-5184
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Kenneth Knowles


Side inputs with duplicate values are being lost due to the usage of a set 
based multimap.

[https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]

 

Originating thread: 
https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5145) Make PTransform names stable in Join/CoGroupByKey

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5145.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Make PTransform names stable in Join/CoGroupByKey
> -
>
> Key: BEAM-5145
> URL: https://issues.apache.org/jira/browse/BEAM-5145
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Leen Toelen
>Priority: Trivial
> Fix For: 2.7.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5149) Add support to the Java SDK harness to merge windows

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5149:
---

 Summary: Add support to the Java SDK harness to merge windows
 Key: BEAM-5149
 URL: https://issues.apache.org/jira/browse/BEAM-5149
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-harness
Reporter: Luke Cwik
Assignee: Luke Cwik


Window merging function:

 
{code:java}
Input: KV>
Output: KV, 
iterable>>> 
{code}

For each set of original windows, a list of all unmerged windows is output 
alongside a map of merged window to set of consumed windows. All original 
windows must be contained in either the unmerged original window set or one of 
the consumed original window sets. Each original window can only be part of one 
output set. The nonce is used by a runner to associate each input with its 
output. The nonce is represented as an opaque set of bytes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5145) Make PTransform names stable in Join/CoGroupByKey

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5145:
---

Assignee: Leen Toelen  (was: Luke Cwik)

> Make PTransform names stable in Join/CoGroupByKey
> -
>
> Key: BEAM-5145
> URL: https://issues.apache.org/jira/browse/BEAM-5145
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Leen Toelen
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5145) Make PTransform names stable in Join/CoGroupByKey

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5145:
---

 Summary: Make PTransform names stable in Join/CoGroupByKey
 Key: BEAM-5145
 URL: https://issues.apache.org/jira/browse/BEAM-5145
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5126) PreCommit filtering broken based upon PR contents

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-5126:
---

 Summary: PreCommit filtering broken based upon PR contents
 Key: BEAM-5126
 URL: https://issues.apache.org/jira/browse/BEAM-5126
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Luke Cwik
Assignee: Alan Myrvold


PR precommits used to be filtered by the contents of the PR.

 

Example PR that should have only spawned the Java PreCommit:

https://github.com/apache/beam/pull/6159

 

This broke work done in BEAM-4445



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4849) Support running Beam Samza jobs in Yarn

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4849.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Support running Beam Samza jobs in Yarn
> ---
>
> Key: BEAM-4849
> URL: https://issues.apache.org/jira/browse/BEAM-4849
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-samza
>Reporter: Xinyu Liu
>Assignee: Xinyu Liu
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We need to some util classes and script to run beam job containers in Yarn. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4846) updateOfflineRepositoryRoot broken

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4846.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:136)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.execute(DefaultTaskPlanExecutor.java:201)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.executeWithTask(DefaultTaskPlanExecutor.java:192)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at 
> 

[jira] [Assigned] (BEAM-4846) updateOfflineRepositoryRoot broken

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4846:
---

Assignee: Pablo Estrada

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:136)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.execute(DefaultTaskPlanExecutor.java:201)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.executeWithTask(DefaultTaskPlanExecutor.java:192)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at 
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> at 
> 

[jira] [Resolved] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4658.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> Expose relevant timer information on *ExecutableStage* to runners
> Update fusion logic to handle timers.
> Provide execution time interface to fire timers into *RemoteBundle*s and also 
> to receive new timers that are being set.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4835) Add more flexible options for data loading to BigQueryIO.Write

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4835.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Add more flexible options for data loading to BigQueryIO.Write
> --
>
> Key: BEAM-4835
> URL: https://issues.apache.org/jira/browse/BEAM-4835
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Gene Peters
>Assignee: Gene Peters
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> As part of the BigQuery API, there are a few options exposed to end-users 
> which allow for more flexible data loading.
> For both 
> [streaming|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setIgnoreUnknownValues-java.lang.Boolean-]
>  and 
> [batch|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setIgnoreUnknownValues-java.lang.Boolean-]
>  inserts, the flag "ignoreUnknownValues" can be set, which indicates if 
> BigQuery should accept rows that contain values that do not match the schema. 
> [In 
> addition,|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setSkipInvalidRows-java.lang.Boolean-]
>  streaming inserts allow for the option of accepting an inserted batch of 
> rows even if some of of the rows are invalid.
> I've made the necessary code changes to make this available within 
> BigQueryIO.Write and will be attaching the pull request to this ticket for 
> review. Both flags are off by default.
> Let me know if you have any questions or feedback about this!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4835) Add more flexible options for data loading to BigQueryIO.Write

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4835:
---

Assignee: Gene Peters  (was: Chamikara Jayalath)

> Add more flexible options for data loading to BigQueryIO.Write
> --
>
> Key: BEAM-4835
> URL: https://issues.apache.org/jira/browse/BEAM-4835
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Gene Peters
>Assignee: Gene Peters
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> As part of the BigQuery API, there are a few options exposed to end-users 
> which allow for more flexible data loading.
> For both 
> [streaming|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setIgnoreUnknownValues-java.lang.Boolean-]
>  and 
> [batch|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setIgnoreUnknownValues-java.lang.Boolean-]
>  inserts, the flag "ignoreUnknownValues" can be set, which indicates if 
> BigQuery should accept rows that contain values that do not match the schema. 
> [In 
> addition,|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setSkipInvalidRows-java.lang.Boolean-]
>  streaming inserts allow for the option of accepting an inserted batch of 
> rows even if some of of the rows are invalid.
> I've made the necessary code changes to make this available within 
> BigQueryIO.Write and will be attaching the pull request to this ticket for 
> review. Both flags are off by default.
> Let me know if you have any questions or feedback about this!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3095) .withCompression() hinted at in docs, but not usable

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3095:
---

Assignee: Kyle Winkelman  (was: Chamikara Jayalath)

> .withCompression() hinted at in docs, but not usable
> 
>
> Key: BEAM-3095
> URL: https://issues.apache.org/jira/browse/BEAM-3095
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Rafael Fernandez
>Assignee: Kyle Winkelman
>Priority: Major
>  Labels: easyfix, newbie, starter
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> There is a FileBasedSink.CompressionType enum, and a comment in TextIO.java 
> that suggests .withCompression(...) is available. Alas, there does not seem 
> to be a documented way to write compressed output. It's unclear whether the 
> documentation is wrong, or the functionality is indeed missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3095) .withCompression() hinted at in docs, but not usable

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3095.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> .withCompression() hinted at in docs, but not usable
> 
>
> Key: BEAM-3095
> URL: https://issues.apache.org/jira/browse/BEAM-3095
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Rafael Fernandez
>Assignee: Kyle Winkelman
>Priority: Major
>  Labels: easyfix, newbie, starter
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> There is a FileBasedSink.CompressionType enum, and a comment in TextIO.java 
> that suggests .withCompression(...) is available. Alas, there does not seem 
> to be a documented way to write compressed output. It's unclear whether the 
> documentation is wrong, or the functionality is indeed missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5041) Java Fn SDK Harness skips unprocessed pCollections

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5041.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Java Fn SDK Harness skips unprocessed pCollections
> --
>
> Key: BEAM-5041
> URL: https://issues.apache.org/jira/browse/BEAM-5041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Java Sdk Harness used pCollections to keep track of computed consumers 
> [here|https://github.com/apache/beam/blob/ff95a82e461bd8319d9733be60e75992ba90cd7c/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java#L158].
>  This is incorrect as consumers are based on pTransforms so pTransforms 
> should be used to keep track of computed consumers.
> In case of Flatten, this creates an issue where pTransforms having same input 
> as that to flatten are not executed. This causes 
> [https://github.com/apache/beam/blob/ff95a82e461bd8319d9733be60e75992ba90cd7c/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/FlattenTest.java#L316]
>  to fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3095) .withCompression() hinted at in docs, but not usable

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3095:

Labels: easyfix newbie starter  (was: )

> .withCompression() hinted at in docs, but not usable
> 
>
> Key: BEAM-3095
> URL: https://issues.apache.org/jira/browse/BEAM-3095
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Rafael Fernandez
>Assignee: Chamikara Jayalath
>Priority: Major
>  Labels: easyfix, newbie, starter
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> There is a FileBasedSink.CompressionType enum, and a comment in TextIO.java 
> that suggests .withCompression(...) is available. Alas, there does not seem 
> to be a documented way to write compressed output. It's unclear whether the 
> documentation is wrong, or the functionality is indeed missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570954#comment-16570954
 ] 

Luke Cwik edited comment on BEAM-5035 at 8/7/18 12:15 AM:
--

I saw something like this before and it was caused by the jar being modified by 
a task while it was being consumed by another task.

This caused some internal corruption during reading if the process doing the 
reading didn't maintain an open file handle the entire time.

 

My specific failure was being caused by the main jar being overwritten by the 
shadow jar when we had the -Prelease enabled. You can validate that this is the 
same class of failure by using 
[inotifywatch|https://linux.die.net/man/1/inotifywatch] (assuming you can 
reproduce this failure locally).


was (Author: lcwik):
I saw something like this before and it was caused by the jar being modified by 
a task while it was being consumed by another task.

This caused some internal corruption during reading if the process doing the 
reading didn't maintain an open file handle the entire time.

 

My specific failure was being caused by the main jar being overwritten by the 
shadow jar when we had the -Prelease enabled. You can validate that this is the 
same failure by using [inotifywatch|https://linux.die.net/man/1/inotifywatch] 
(assuming you can reproduce this failure locally).

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Ahmet Altay
>Priority: Critical
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5035) beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava FAILED

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570954#comment-16570954
 ] 

Luke Cwik commented on BEAM-5035:
-

I saw something like this before and it was caused by the jar being modified by 
a task while it was being consumed by another task.

This caused some internal corruption during reading if the process doing the 
reading didn't maintain an open file handle the entire time.

 

My specific failure was being caused by the main jar being overwritten by the 
shadow jar when we had the -Prelease enabled. You can validate that this is the 
same failure by using [inotifywatch|https://linux.die.net/man/1/inotifywatch] 
(assuming you can reproduce this failure locally).

> beam_PostCommit_Java_GradleBuild/1105 :beam-examples-java:compileTestJava 
> FAILED
> 
>
> Key: BEAM-5035
> URL: https://issues.apache.org/jira/browse/BEAM-5035
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Assignee: Ahmet Altay
>Priority: Critical
>
> Compilation failed for
> [https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1105/]
>  > Task :beam-examples-java:compileTestJava FAILED
>  
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java:22:
>  error: cannot access BigqueryMatcher
>  import org.apache.beam.sdk.io.gcp.testing.BigqueryMatcher;
>  ^
>  bad class file: 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_GradleBuild/src/sdks/java/io/google-cloud-platform/build/libs/beam-sdks-java-io-google-cloud-platform-2.7.0-SNAPSHOT-tests.jar(/org/apache/beam/sdk/io/gcp/testing/BigqueryMatcher.class)
>  unable to access file: java.util.zip.ZipException: invalid stored block 
> lengths
>  Please remove or make sure it appears in the correct subdirectory of the 
> classpath.
>  1 error
>  
> https://github.com/apache/beam/blame/328129bf033bc6be16bc8e09af905f37b7516412/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesIT.java
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4826) Flink runner sends bad flatten to SDK

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570945#comment-16570945
 ] 

Luke Cwik commented on BEAM-4826:
-

This seems like a case where the GreedyPipelineFuser is incorrectly creating an 
ExecutableStage with a malformed flatten. Since the ExecutableStage contains 
the pipeline components, the flatten that is inserted into the ExecutableStage 
should have its inputs pruned there.

> Flink runner sends bad flatten to SDK
> -
>
> Key: BEAM-4826
> URL: https://issues.apache.org/jira/browse/BEAM-4826
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For a Go flatten test w/ 3 input, the Flink runner splits this into 3 bundle 
> descriptors. But it sends the original 3-input flatten but w/ 1 actual input 
> present in each bundle descriptor. This is inconsistent and the SDK shouldn't 
> expect dangling PCollections. In contrast, Dataflow removes the flatten when 
> it does the same split.
> Snippet:
> register: <
>   process_bundle_descriptor: <
> id: "3"
> transforms: <
>   key: "e4"
>   value: <
> unique_name: "github.com/apache/beam/sdks/go/pkg/beam.createFn'1"
> spec: <
>   urn: "urn:beam:transform:pardo:v1"
>   payload: [...]
> >
> inputs: <
>   key: "i0"
>   value: "n3"
> >
> outputs: <
>   key: "i0"
>   value: "n4"
> >
>   >
> >
> transforms: <
>   key: "e7"
>   value: <
> unique_name: "Flatten"
> spec: <
>   urn: "beam:transform:flatten:v1"
> >
> inputs: <
>   key: "i0"
>   value: "n2"
> >
> inputs: <
>   key: "i1"
>   value: "n4" . // <--- only one present.
> >
> inputs: <
>   key: "i2"
>   value: "n6"
> >
> outputs: <
>   key: "i0"
>   value: "n7"
> >
>   >
> >
> [...]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-1009:
---

Assignee: (was: Pei He)

> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Major
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3412) Update BigTable client version to 1.0

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3412:
---

Assignee: Garrett Jones  (was: Solomon Duskis)

> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3412) Update BigTable client version to 1.0

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3412.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Update BigTable client version to 1.0
> -
>
> Key: BEAM-3412
> URL: https://issues.apache.org/jira/browse/BEAM-3412
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Solomon Duskis
>Priority: Major
> Fix For: 2.7.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4262) Google suggests that upgrade version of bigtable-client to 1.3.0 [1]

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4262:
---

Assignee: Garrett Jones

> Google suggests that upgrade version of bigtable-client to 1.3.0 [1]
> 
>
> Key: BEAM-4262
> URL: https://issues.apache.org/jira/browse/BEAM-4262
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
> Environment: scio
>Reporter: keith
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We recently had a bug where a dataflow job hung at the end.  Google 
> engineering suggested that we upgrade our bigtable-client.  In particular:
>  
> "The Dataflow Team and the Bigtable team were able to determine the root 
> cause of this issue was due to the use of the client library of 
> bigtable-client-1.0.0-pre3 which is not recommended. They have suggested you 
> upgrade to bigtable-client-1.3.0 [1] which is the latest release"
>  
> Google: Case#15718936



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4262) Google suggests that upgrade version of bigtable-client to 1.3.0 [1]

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4262.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Google suggests that upgrade version of bigtable-client to 1.3.0 [1]
> 
>
> Key: BEAM-4262
> URL: https://issues.apache.org/jira/browse/BEAM-4262
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
> Environment: scio
>Reporter: keith
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We recently had a bug where a dataflow job hung at the end.  Google 
> engineering suggested that we upgrade our bigtable-client.  In particular:
>  
> "The Dataflow Team and the Bigtable team were able to determine the root 
> cause of this issue was due to the use of the client library of 
> bigtable-client-1.0.0-pre3 which is not recommended. They have suggested you 
> upgrade to bigtable-client-1.3.0 [1] which is the latest release"
>  
> Google: Case#15718936



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4818) Make the visteg plugin only run once

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4818.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Make the visteg plugin only run once
> 
>
> Key: BEAM-4818
> URL: https://issues.apache.org/jira/browse/BEAM-4818
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The visteg plugin is producing a dot report for each subproject which takes 
> several seconds on my machine when the build task is large.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5029) Gradle 5 compatibility

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5029:
---

Assignee: Alan Myrvold  (was: Luke Cwik)

> Gradle 5 compatibility
> --
>
> Key: BEAM-5029
> URL: https://issues.apache.org/jira/browse/BEAM-5029
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Elliotte Rusty Harold
>Assignee: Alan Myrvold
>Priority: Minor
>
> Warning when building:
> Deprecated Gradle features were used in this build, making it incompatible 
> with Gradle 5.0.
> See 
> https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings
> Probably worth getting rid of the deprecated features at some point.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5029) Gradle 5 compatibility

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558544#comment-16558544
 ] 

Luke Cwik commented on BEAM-5029:
-

It is typically from the use of our plugins and not from our build itself. Will 
need to get them upgraded.

> Gradle 5 compatibility
> --
>
> Key: BEAM-5029
> URL: https://issues.apache.org/jira/browse/BEAM-5029
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Elliotte Rusty Harold
>Assignee: Luke Cwik
>Priority: Minor
>
> Warning when building:
> Deprecated Gradle features were used in this build, making it incompatible 
> with Gradle 5.0.
> See 
> https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings
> Probably worth getting rid of the deprecated features at some point.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4629) RAT failures are hard to debug

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4629.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> RAT failures are hard to debug
> --
>
> Key: BEAM-4629
> URL: https://issues.apache.org/jira/browse/BEAM-4629
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Luke Cwik
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> RAT failures produces an error with a reference to log file that is not 
> accessible.
> For example:
> > Task :rat
> Build cache key for task ':rat' is 7a8f8655183c948a3f7a087e2f7c412b
> Caching disabled for task ':rat': Caching has not been enabled for the task
> Task ':rat' is not up-to-date because:
>   No history is available.
> Rat TXT report: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2/src/build/reports/rat/rat-report.txt
> This is not useful unless, the person has access to the jenkins VM. Tests 
> should include a copy of the failed rat-report.txt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4629) RAT failures are hard to debug

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4629:
---

Assignee: Luke Cwik  (was: Alan Myrvold)

> RAT failures are hard to debug
> --
>
> Key: BEAM-4629
> URL: https://issues.apache.org/jira/browse/BEAM-4629
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ahmet Altay
>Assignee: Luke Cwik
>Priority: Major
>
> RAT failures produces an error with a reference to log file that is not 
> accessible.
> For example:
> > Task :rat
> Build cache key for task ':rat' is 7a8f8655183c948a3f7a087e2f7c412b
> Caching disabled for task ':rat': Caching has not been enabled for the task
> Task ':rat' is not up-to-date because:
>   No history is available.
> Rat TXT report: 
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2/src/build/reports/rat/rat-report.txt
> This is not useful unless, the person has access to the jenkins VM. Tests 
> should include a copy of the failed rat-report.txt.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4867) Run :rat for all parts of repository as pre-commit

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4867.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> Run :rat for all parts of repository as pre-commit
> --
>
> Key: BEAM-4867
> URL: https://issues.apache.org/jira/browse/BEAM-4867
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Mikhail Gryzykhin
>Assignee: Jason Kuster
>Priority: Major
> Fix For: Not applicable
>
>
> The idea comes from:
> [https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4866]
> Apparently, we do not run pre-commits for all of our paths and not all 
> pre-commits are always relevant. However we still want to verify licence 
> headers an whole repo.
>  
> This can be done as extracting :rat as a separate job and run it on each 
> commit regardless of path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-1457) Enable rat plugin and findbugs plugin in default build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-1457:
---

Assignee: Jean-Baptiste Onofré

> Enable rat plugin and findbugs plugin in default build
> --
>
> Key: BEAM-1457
> URL: https://issues.apache.org/jira/browse/BEAM-1457
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Aviem Zur
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: Not applicable
>
>
> Today, maven rat plugin and findbugs plugin only run when `release` profile 
> is specified.
> Since these plugins do not add a large amount of time compared to the normal 
> build, and their checks are required to pass to approve pull requests - let's 
> enable them by default.
> [Original dev list 
> discussion|https://lists.apache.org/thread.html/e1f80e54b44b4a39630d978abe79fb6a6cecf71d9821ee1881b47afb@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-1457) Enable rat plugin and findbugs plugin in default build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-1457.
-
   Resolution: Fixed
Fix Version/s: Not applicable

During the migration to gradle the rat plugin now runs as part of the 
precommits.

> Enable rat plugin and findbugs plugin in default build
> --
>
> Key: BEAM-1457
> URL: https://issues.apache.org/jira/browse/BEAM-1457
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Aviem Zur
>Priority: Major
> Fix For: Not applicable
>
>
> Today, maven rat plugin and findbugs plugin only run when `release` profile 
> is specified.
> Since these plugins do not add a large amount of time compared to the normal 
> build, and their checks are required to pass to approve pull requests - let's 
> enable them by default.
> [Original dev list 
> discussion|https://lists.apache.org/thread.html/e1f80e54b44b4a39630d978abe79fb6a6cecf71d9821ee1881b47afb@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5022) Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to BeamModulePlugin

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5022:

Description: 
Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
BeamModulePlugin So that it can be used by other portable runners tests.

 

Also Create an interface TestJobserverDriver and make the drivers extend it 
instead of using reflection start the Jobserver.

  was:
Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
BeamGradlePlugin So that it can be used by other portable runners tests.

 

Also Create an interface TestJobserverDriver and make the drivers extend it 
instead of using reflection start the Jobserver.


> Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
> BeamModulePlugin
> --
>
> Key: BEAM-5022
> URL: https://issues.apache.org/jira/browse/BEAM-5022
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, runner-flink
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
> BeamModulePlugin So that it can be used by other portable runners tests.
>  
> Also Create an interface TestJobserverDriver and make the drivers extend it 
> instead of using reflection start the Jobserver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5022) Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to BeamModulePlugin

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5022:

Summary: Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask 
to BeamModulePlugin  (was: Move 
beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
BeamGradlePlugin)

> Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
> BeamModulePlugin
> --
>
> Key: BEAM-5022
> URL: https://issues.apache.org/jira/browse/BEAM-5022
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, runner-flink
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> Move beam-sdks-java-fn-execution#createPortableValidatesRunnerTask to 
> BeamGradlePlugin So that it can be used by other portable runners tests.
>  
> Also Create an interface TestJobserverDriver and make the drivers extend it 
> instead of using reflection start the Jobserver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4866) RAT tests fail

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556114#comment-16556114
 ] 

Luke Cwik commented on BEAM-4866:
-

:rat is already a dependency of the three precommits.

> RAT tests fail
> --
>
> Key: BEAM-4866
> URL: https://issues.apache.org/jira/browse/BEAM-4866
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
> Environment: Mac
>Reporter: Elliotte Rusty Harold
>Assignee: Luke Cwik
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> $ ./gradlew check
>  ...
>  4: Task failed with an exception.
>  ---
>  * What went wrong:
>  Execution failed for task ':rat'.
>  > Found 5 files with unapproved/unknown licenses. See 
> [file:/Users/elharo/beam/build/reports/rat/rat-report.txt|file:///Users/elharo/beam/build/reports/rat/rat-report.txt]
>  ```
> I don't think these are likely to be real issues. Likely the tool just needs 
> to be configured a little differently:
>  
>  $ more /Users/elharo/beam/build/reports/rat/rat-report.txt
> *
>  Summary
>  ---
>  Generated at: 2018-07-25T12:52:35-04:00
> Notes: 4
>  Binaries: 119
>  Archives: 4
>  Standards: 3935
> Apache Licensed: 3930
>  Generated Documents: 0
> JavaDocs are generated, thus a license header is optional.
>  Generated files do not require license headers.
> 5 Unknown Licenses
> *
> Files with unapproved licenses:
> /Users/elharo/beam/.test-infra/dockerized-jenkins/basic-security.groovy
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/config.xml
>  
> /Users/elharo/beam/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/plugins.txt
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/seedjobconfig.xml
> *
>  ```



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4866) RAT tests fail

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4866.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> RAT tests fail
> --
>
> Key: BEAM-4866
> URL: https://issues.apache.org/jira/browse/BEAM-4866
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
> Environment: Mac
>Reporter: Elliotte Rusty Harold
>Assignee: Luke Cwik
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> $ ./gradlew check
>  ...
>  4: Task failed with an exception.
>  ---
>  * What went wrong:
>  Execution failed for task ':rat'.
>  > Found 5 files with unapproved/unknown licenses. See 
> [file:/Users/elharo/beam/build/reports/rat/rat-report.txt|file:///Users/elharo/beam/build/reports/rat/rat-report.txt]
>  ```
> I don't think these are likely to be real issues. Likely the tool just needs 
> to be configured a little differently:
>  
>  $ more /Users/elharo/beam/build/reports/rat/rat-report.txt
> *
>  Summary
>  ---
>  Generated at: 2018-07-25T12:52:35-04:00
> Notes: 4
>  Binaries: 119
>  Archives: 4
>  Standards: 3935
> Apache Licensed: 3930
>  Generated Documents: 0
> JavaDocs are generated, thus a license header is optional.
>  Generated files do not require license headers.
> 5 Unknown Licenses
> *
> Files with unapproved licenses:
> /Users/elharo/beam/.test-infra/dockerized-jenkins/basic-security.groovy
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/config.xml
>  
> /Users/elharo/beam/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/plugins.txt
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/seedjobconfig.xml
> *
>  ```



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4866) RAT tests fail

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556063#comment-16556063
 ] 

Luke Cwik commented on BEAM-4866:
-

Broken by: https://github.com/apache/beam/pull/5989

> RAT tests fail
> --
>
> Key: BEAM-4866
> URL: https://issues.apache.org/jira/browse/BEAM-4866
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
> Environment: Mac
>Reporter: Elliotte Rusty Harold
>Assignee: Luke Cwik
>Priority: Major
>
> ```shell
> $ ./gradlew check
> ...
> 4: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':rat'.
> > Found 5 files with unapproved/unknown licenses. See 
> > file:/Users/elharo/beam/build/reports/rat/rat-report.txt
> ```
> I don't think these are likely to be real issues. Likely the tool just needs 
> to be configured a little differently:
> ```
> $ more /Users/elharo/beam/build/reports/rat/rat-report.txt
> *
> Summary
> ---
> Generated at: 2018-07-25T12:52:35-04:00
> Notes: 4
> Binaries: 119
> Archives: 4
> Standards: 3935
> Apache Licensed: 3930
> Generated Documents: 0
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
> 5 Unknown Licenses
> *
> Files with unapproved licenses:
> /Users/elharo/beam/.test-infra/dockerized-jenkins/basic-security.groovy
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/config.xml
>  
> /Users/elharo/beam/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/plugins.txt
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/seedjobconfig.xml
> *
> ```



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4866) RAT tests fail

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556061#comment-16556061
 ] 

Luke Cwik commented on BEAM-4866:
-

This is a legitimate failure, the groovy/xml/... need license headers as they 
are missing.

> RAT tests fail
> --
>
> Key: BEAM-4866
> URL: https://issues.apache.org/jira/browse/BEAM-4866
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
> Environment: Mac
>Reporter: Elliotte Rusty Harold
>Assignee: Luke Cwik
>Priority: Major
>
> ```shell
> $ ./gradlew check
> ...
> 4: Task failed with an exception.
> ---
> * What went wrong:
> Execution failed for task ':rat'.
> > Found 5 files with unapproved/unknown licenses. See 
> > file:/Users/elharo/beam/build/reports/rat/rat-report.txt
> ```
> I don't think these are likely to be real issues. Likely the tool just needs 
> to be configured a little differently:
> ```
> $ more /Users/elharo/beam/build/reports/rat/rat-report.txt
> *
> Summary
> ---
> Generated at: 2018-07-25T12:52:35-04:00
> Notes: 4
> Binaries: 119
> Archives: 4
> Standards: 3935
> Apache Licensed: 3930
> Generated Documents: 0
> JavaDocs are generated, thus a license header is optional.
> Generated files do not require license headers.
> 5 Unknown Licenses
> *
> Files with unapproved licenses:
> /Users/elharo/beam/.test-infra/dockerized-jenkins/basic-security.groovy
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/config.xml
>  
> /Users/elharo/beam/.test-infra/dockerized-jenkins/javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration.xml
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/plugins.txt
>  /Users/elharo/beam/.test-infra/dockerized-jenkins/seedjobconfig.xml
> *
> ```



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4863) Implement consistentWithEquals/structuralValue on FullWindowedValueCoder

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-4863:
---

 Summary: Implement consistentWithEquals/structuralValue on 
FullWindowedValueCoder
 Key: BEAM-4863
 URL: https://issues.apache.org/jira/browse/BEAM-4863
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik


Implementing *consistentWithEquals*/*structuralValue* boosts significantly the 
performance of using these values in comparison operations since it doesn't 
require encoding the values.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4830) Determine why go vet failures invoked by ./gradlew check were not caught be jenkins build build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4830:
---

Assignee: Robert Burke  (was: Luke Cwik)

> Determine why go vet failures invoked by ./gradlew check  were not caught be 
> jenkins build build
> 
>
> Key: BEAM-4830
> URL: https://issues.apache.org/jira/browse/BEAM-4830
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Alex Amato
>Assignee: Robert Burke
>Priority: Major
>
> The purpose of this is to catch errors developers see when they first start 
> contributing to beam. Let's ensure we run the same commands in the 
> [contributing guide|https://beam.apache.org/contribute/].
>  
> Note: check runs more than build, so we are not catching these problems in 
> the continuous Jenkins testing.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4819) Make portable Flink runner JobBundleFactory configurable

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553523#comment-16553523
 ] 

Luke Cwik commented on BEAM-4819:
-

Yes I believe it should be.

 

Its unfortunate but a lot of stuff was baked into PipelineOptions at a top 
level which makes it annoying to use as a top level options class for 
configuring runners. A few ideas came to mind:
 * (preferred) Create a new top level interface called options which has the 
same capabilities as PipelineOptions (the "as" method and a few others) 
allowing PipelineOptions to be decoupled from Options which has all the 
argument parsing and JSON conversion magic.
 ** Good: PipelineOptions can still be passed around to things like FileSystem 
configuration, paves the way for transform level options, ...
 * Use string[] args and another argument parsing library building a simpler 
version of PipelineOptionsFactory/PipelineOptions dedicated to runners.
 ** Good: Likely fastest solution to implement

> Make portable Flink runner JobBundleFactory configurable
> 
>
> Key: BEAM-4819
> URL: https://issues.apache.org/jira/browse/BEAM-4819
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>
> BEAM-4791 introduces factory override for testing, expand that to allow users 
> to configure a different factory via service loader to adopt alternative 
> execution environments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4847) Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is exhausted

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-4847:

Summary: Java PreCommit is flaky: Expiring Daemon because JVM Tenured space 
is exhausted  (was: Java PreCommit is flaky)

> Java PreCommit is flaky: Expiring Daemon because JVM Tenured space is 
> exhausted
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553357#comment-16553357
 ] 

Luke Cwik commented on BEAM-4847:
-

This seems to be more of an existing issue: 
[https://lists.apache.org/thread.html/b47799450c73d1c2588a8029c224d1c85278fd7431568345a7928983@%3Cdev.beam.apache.org%3E]
 that [~swegner] dealt a lot with.

> Java PreCommit is flaky
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4847) Java PreCommit is flaky

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4847:
---

Assignee: Alan Myrvold  (was: Luke Cwik)

> Java PreCommit is flaky
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4847) Java PreCommit is flaky

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553355#comment-16553355
 ] 

Luke Cwik commented on BEAM-4847:
-

Expiring Daemon because JVM Tenured space is exhausted

> Java PreCommit is flaky
> ---
>
> Key: BEAM-4847
> URL: https://issues.apache.org/jira/browse/BEAM-4847
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>
> Java PreCommit seems like flaky: 
> https://builds.apache.org/job/beam_PreCommit_Java_Commit/505/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4831) Fix broken go vet task in the gradle build

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4831:
---

Assignee: Robert Burke  (was: Luke Cwik)

> Fix broken go vet task in the gradle build
> --
>
> Key: BEAM-4831
> URL: https://issues.apache.org/jira/browse/BEAM-4831
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Alex Amato
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Reproduce by running
> {{./gradlew check}}
> {{Today this is failing.}}
> {{go vet seems to be trying to validate files in a vendor folder.}}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4846) updateOfflineRepositoryRoot broken

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553164#comment-16553164
 ] 

Luke Cwik commented on BEAM-4846:
-

Migrating the plugin application outside of the binary plugin did work around 
the issue. Example work around: 
https://github.com/lukecwik/incubator-beam/commit/b37fbe96f2d7f035010c5cb3a7a47659995d4aad

> updateOfflineRepositoryRoot broken
> --
>
> Key: BEAM-4846
> URL: https://issues.apache.org/jira/browse/BEAM-4846
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Major
>
> Migrating the [build script to a binary 
> plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
>  broke the updateOfflineRepositoryRoot task with a less then useful error 
> message:
> {code:java}
> * Exception is:
> org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
> ':updateOfflineRepository'.
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
> at 
> org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
> at 
> org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
> at 
> org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
> at 
> org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
> at 
> org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
> at 
> org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
> at 
> org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
> at 
> org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
> at 
> org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
> at 
> org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
> at 
> org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
> at 
> org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
> at 
> org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
> at 
> org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
> at 
> org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:136)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.execute(DefaultTaskPlanExecutor.java:201)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.executeWithTask(DefaultTaskPlanExecutor.java:192)
> at 
> org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
> at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> at 
> 

[jira] [Created] (BEAM-4846) updateOfflineRepositoryRoot broken

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-4846:
---

 Summary: updateOfflineRepositoryRoot broken
 Key: BEAM-4846
 URL: https://issues.apache.org/jira/browse/BEAM-4846
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Luke Cwik


Migrating the [build script to a binary 
plugin|https://github.com/apache/beam/commit/220b180132c8ec61ab5e09a49e26c3ae2e7191a7]
 broke the updateOfflineRepositoryRoot task with a less then useful error 
message:
{code:java}
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
':updateOfflineRepository'.
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:110)
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:77)
at 
org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
at 
org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
at 
org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
at 
org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
at 
org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
at 
org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
at 
org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
at 
org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
at 
org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
at 
org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
at 
org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
at 
org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.run(EventFiringTaskExecuter.java:51)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:317)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:309)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:185)
at 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:97)
at 
org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at 
org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:46)
at 
org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:262)
at 
org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$ExecuteTaskAction.execute(DefaultTaskExecutionGraph.java:246)
at 
org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:136)
at 
org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker$1.execute(DefaultTaskPlanExecutor.java:130)
at 
org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.execute(DefaultTaskPlanExecutor.java:201)
at 
org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.executeWithTask(DefaultTaskPlanExecutor.java:192)
at 
org.gradle.execution.taskgraph.DefaultTaskPlanExecutor$TaskExecutorWorker.run(DefaultTaskPlanExecutor.java:130)
at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: org.gradle.api.artifacts.ResolveException: Could not resolve all 
dependencies for configuration ':detachedConfiguration1'.
at 
org.gradle.api.internal.artifacts.ivyservice.ErrorHandlingConfigurationResolver.wrapException(ErrorHandlingConfigurationResolver.java:96)
at 

[jira] [Resolved] (BEAM-4769) Types$WildcardTypeImpl cannot be cast to java.lang.reflect.TypeVariable

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4769.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Types$WildcardTypeImpl cannot be cast to java.lang.reflect.TypeVariable
> ---
>
> Key: BEAM-4769
> URL: https://issues.apache.org/jira/browse/BEAM-4769
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.5.0
>Reporter: Dzmitry Lazerka
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Hi,
> I'm getting the following exception, probably when Beam cannot find a coder 
> for a wildcard generic type (although I don't think I use any, gotta find).
> {code:java}
> Exception in thread "main" java.lang.ClassCastException: 
> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.reflect.Types$WildcardTypeImpl
>  cannot be cast to java.lang.reflect.TypeVariable
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:625)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromParameterizedType(CoderRegistry.java:658)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:620)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromParameterizedType(CoderRegistry.java:658)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:620)
>  at org.apache.beam.sdk.coders.CoderRegistry.getCoder(CoderRegistry.java:254)
>  at 
> org.apache.beam.sdk.values.PCollection.inferCoderOrFail(PCollection.java:149)
>  at 
> org.apache.beam.sdk.values.PCollection.finishSpecifyingOutput(PCollection.java:89)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy.setOutput(TransformHierarchy.java:208)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:538)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
>  at org.apache.beam.sdk.transforms.ParDo$SingleOutput.expand(ParDo.java:649)
>  at org.apache.beam.sdk.transforms.ParDo$SingleOutput.expand(ParDo.java:584)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
>  at org.apache.beam.sdk.transforms.MapElements.expand(MapElements.java:123)
>  at org.apache.beam.sdk.transforms.MapElements.expand(MapElements.java:36)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
> {code} 
> The error seems to be in the following lines of CoderRegistry.java @ 629:
> {code:java|title=CoderRegistry.java|borderStyle=solid}
> } else if (type instanceof TypeVariable) {
>      coder = getCoderFromFactories(typeDescriptor, Collections.emptyList());
> } else if (type instanceof WildcardType) {
>      // No coder for an unknown generic type.
>      throw new CannotProvideCoderException(
>      String.format("Cannot provide a coder for type variable %s"
>      + " (declared by %s) because the actual type is unknown due to 
> erasure.",
>      type,
>      ((TypeVariable) type).getGenericDeclaration()),
>      ReasonCode.TYPE_ERASURE);
> {code}
> It tries to cast a WildcardType to TypeVariable, although that should've 
> been caught above on line 621.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4795) Support Spanner library 0.53

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4795.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Support Spanner library 0.53
> 
>
> Key: BEAM-4795
> URL: https://issues.apache.org/jira/browse/BEAM-4795
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Niel Markwick
>Assignee: Garrett Jones
>Priority: Minor
> Fix For: 2.7.0
>
>
> Beam 2.5 uses Spanner library 0.20
> Spanner (at time of writing) is up to version 0.53 with many additional 
> features... (eg commit timestamps).
> It is impossible to use this spanner library version due to Beam using 
> functions which have been removed between 0.20 and 0.53... 
> Please consider updating the spanner (and other GCP) library versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3321) Update gax-grpc dependency to latest

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3321.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Update gax-grpc dependency to latest
> 
>
> Key: BEAM-3321
> URL: https://issues.apache.org/jira/browse/BEAM-3321
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Beam uses 0.20 latest release is 1.15: 
> https://mvnrepository.com/artifact/com.google.api/gax-grpc
> This results in compatibility issues when Beam and Datastore client are used 
> together: 
> https://lists.apache.org/thread.html/8a99f4c0dbb1138f1067ef73b2ba7f44f035b309af41b1876ffc1ff2@%3Cuser.beam.apache.org%3E
> Assigning to Mairbek who added this dependency recently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)