[jira] [Commented] (BEAM-5526) Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the singleton
[ https://issues.apache.org/jira/browse/BEAM-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635093#comment-16635093 ] Romain Manni-Bucau commented on BEAM-5526: -- [~kenn] I didnt open a thread on the list but the fact to have an interface for a single implementation leads to adding noise to the source code (which is what "papers" opposed to your pdf say and I agree with them more than this pdf from my past experience). What I like with it being pluggable is that engine would provide it and reuse their own asm flavor (xbean for instance in several of them) which would drop bytebuddy from beam dependencies. That said my immediate issue is that on java11 beam fails - i didnt find an umbrella ticket but I think there is one since it was discussed on the list. > Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the > singleton > - > > Key: BEAM-5526 > URL: https://issues.apache.org/jira/browse/BEAM-5526 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Priority: Blocker > > org.apache.beam.sdk.transforms.reflect.DoFnInvokers + DoFnInvokerFactory > design is to be a SPI to let user plug their own bytecode manipulation > library, however in practise beam uses ByteBuddyDoFnInvokerFactory as a > singleton which makes all this design useless. > ByteBuddyDoFnInvokerFactory is also not configurable at all - typically the > injection strategy so it assumes it runs in an environment and on a JVM where > it will work - it does not on java 11 for instance. > This ticket is about fixing all these small inconsistency and blocker to tun > on java 11. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5526) Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the singleton
Romain Manni-Bucau created BEAM-5526: Summary: Make ByteBuddyDoFnInvokerFactory injection strategy configurable + drop the singleton Key: BEAM-5526 URL: https://issues.apache.org/jira/browse/BEAM-5526 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles org.apache.beam.sdk.transforms.reflect.DoFnInvokers + DoFnInvokerFactory design is to be a SPI to let user plug their own bytecode manipulation library, however in practise beam uses ByteBuddyDoFnInvokerFactory as a singleton which makes all this design useless. ByteBuddyDoFnInvokerFactory is also not configurable at all - typically the injection strategy so it assumes it runs in an environment and on a JVM where it will work - it does not on java 11 for instance. This ticket is about fixing all these small inconsistency and blocker to tun on java 11. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5495) PipelineResources algorithm is not working in most environments
Romain Manni-Bucau created BEAM-5495: Summary: PipelineResources algorithm is not working in most environments Key: BEAM-5495 URL: https://issues.apache.org/jira/browse/BEAM-5495 Project: Beam Issue Type: Bug Components: runner-spark Reporter: Romain Manni-Bucau Assignee: Amit Sela Issue are: 1. it assumes the classloader is an URLClassLoader (not always true and java >= 9 breaks that as well for the app loader) 2. it uses loader.getURLs() which leads to including the JRE itself in the staged file Looks like this detect resource algorithm can't work and should be replaced by a SPI rather than a built-in and not extensible algorithm. Another valid alternative is to just drop that "guess" logic and force the user to set staged files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-4381) Provide DoFn to convert a Row (record with schema) to a javax.json.JsonObject
Romain Manni-Bucau created BEAM-4381: Summary: Provide DoFn to convert a Row (record with schema) to a javax.json.JsonObject Key: BEAM-4381 URL: https://issues.apache.org/jira/browse/BEAM-4381 Project: Beam Issue Type: New Feature Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles This task is about ensuring the generic record of beam can be converted to a more mainstream format (JSON). The usage os javax.json.JsonObject is to ensure it is portable and implementation independent and integrates th emost probably with client libraries (most implementations have a bridge to the jsr anyway) It will likely lead to provide a Dofn Row -> JsonObject and the opposite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4057) Ensure generated pom don't break consumers
[ https://issues.apache.org/jira/browse/BEAM-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460591#comment-16460591 ] Romain Manni-Bucau commented on BEAM-4057: -- Nothing ready but checking from nexus it seems the meta are still missing in the produced jar at least so 2 is clearly not done. 5 seems not done too which breaks some toolings as mentionned on the list by a user. I didn't check other points but all but 4 are quite critical for all the world outside beam core devs IMHO. > Ensure generated pom don't break consumers > -- > > Key: BEAM-4057 > URL: https://issues.apache.org/jira/browse/BEAM-4057 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Romain Manni-Bucau >Priority: Major > > Out of my head here are the requirements: > 1. dependencies are all here (all scopes and well scoped: this means that > provided or test dependencies are not in compile scope for instance) > 2. META-INF should contain the pom.xml and pom.properties as maven generates > them (it is consumes by tools and libraries to grab the dependencies or scan > some classpath/lib folder) > 3. ensure the compiler plugin at least is defined with the java > version+compiler flags (a usage is to check if -parameters is activated for > instance) > 4. (nice to have) dont put all the boilerplate in all poms (license, etc) but > keep it in the parent pom as it was > 5. (if possible) respect the hierarchy (parents) - this is used sometimes as > a shortcut for dependencies analyzis cause it is faster than analyzing the > dependencies, probably not the best practise ever but it is efficient in > general > 6. ensure meta used by mainstream tools like mvnrepository are here > (description etc, should be a passthrough from gradle) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3992) Use JSON-B instead of an hardcoded Jackson
[ https://issues.apache.org/jira/browse/BEAM-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451750#comment-16451750 ] Romain Manni-Bucau commented on BEAM-3992: -- Hi [~kenn], I will send a mail to dev@ now. > Use JSON-B instead of an hardcoded Jackson > -- > > Key: BEAM-3992 > URL: https://issues.apache.org/jira/browse/BEAM-3992 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Affects Versions: 2.4.0 >Reporter: Romain Manni-Bucau >Priority: Major > > Currently beam uses jackson directly everywhere, it either messes up > environments where another version is here and can't be upgraded or it forces > to shade and have a fat app where it is not needed. Using JSON-B will allow > to switch the impl and beam will just rely on ~200k of API instead of megs of > dependencies making it smoother to integrated in all environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle
[ https://issues.apache.org/jira/browse/BEAM-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau reassigned BEAM-4088: Assignee: Etienne Chauchot (was: Romain Manni-Bucau) > ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 > does not pass in gradle > - > > Key: BEAM-4088 > URL: https://issues.apache.org/jira/browse/BEAM-4088 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > This is a new test being added to ensure threads don't leak. The failure > seems to indicate that threads do leak. > This test fails using gradle but previously passed using maven > PR: https://github.com/apache/beam/pull/4965 > Presubmit Failure: > * https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4059/ > * > https://scans.gradle.com/s/grha56432j3t2/tests/jqhvlvf72f7pg-ipde5etqqejoa?openStackTraces=WzBd -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle
[ https://issues.apache.org/jira/browse/BEAM-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441899#comment-16441899 ] Romain Manni-Bucau commented on BEAM-4088: -- Scott, this test illustrate that gradle test setup is wrong in respect of the tests isolation. Must be fixed to ensure build is deterministic ans reliable. See the list thread about it. > ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 > does not pass in gradle > - > > Key: BEAM-4088 > URL: https://issues.apache.org/jira/browse/BEAM-4088 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Etienne Chauchot >Assignee: Scott Wegner >Priority: Major > > This test always fail using gradle but always pass using maven -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle
[ https://issues.apache.org/jira/browse/BEAM-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau reassigned BEAM-4088: Assignee: Scott Wegner (was: Romain Manni-Bucau) > ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 > does not pass in gradle > - > > Key: BEAM-4088 > URL: https://issues.apache.org/jira/browse/BEAM-4088 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Etienne Chauchot >Assignee: Scott Wegner >Priority: Major > > This test always fail using gradle but always pass using maven -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4016) Direct runner incorrect lifecycle, @SplitRestriction should execute after @Setup on SplittableDoFn
[ https://issues.apache.org/jira/browse/BEAM-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439412#comment-16439412 ] Romain Manni-Bucau commented on BEAM-4016: -- PS: don't forget the teardown mapping for any instance (or use the same caching hack direct runer has) > Direct runner incorrect lifecycle, @SplitRestriction should execute after > @Setup on SplittableDoFn > -- > > Key: BEAM-4016 > URL: https://issues.apache.org/jira/browse/BEAM-4016 > Project: Beam > Issue Type: Bug > Components: runner-direct >Affects Versions: 2.4.0 >Reporter: Ismaël Mejía >Assignee: Thomas Groh >Priority: Major > Attachments: sdf-splitrestriction-lifeycle-test.patch > > > The method annotated with @SplitRestriction is the method where we can define > the RestrictionTrackers (splits) in advance in a SDF. It makes sense to > execute this after the @Setup method given that usually connections are > established at Setup and can be used to ask the different data stores about > the partitioning strategy. I added a test for this in the > SplittableDoFnTest.SDFWithLifecycle test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4057) Ensure generated pom don't break consumers
[ https://issues.apache.org/jira/browse/BEAM-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436178#comment-16436178 ] Romain Manni-Bucau commented on BEAM-4057: -- [~kenn] all tools allows native browsing (check idea) and it is common to browse a project like that to grab dep cause like 60 times faster for some cases so it is done by users you like it or not (note i didnt say i like it ;)). > Ensure generated pom don't break consumers > -- > > Key: BEAM-4057 > URL: https://issues.apache.org/jira/browse/BEAM-4057 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Romain Manni-Bucau >Assignee: Luke Cwik >Priority: Major > > Out of my head here are the requirements: > 1. dependencies are all here (all scopes and well scoped: this means that > provided or test dependencies are not in compile scope for instance) > 2. META-INF should contain the pom.xml and pom.properties as maven generates > them (it is consumes by tools and libraries to grab the dependencies or scan > some classpath/lib folder) > 3. ensure the compiler plugin at least is defined with the java > version+compiler flags (a usage is to check if -parameters is activated for > instance) > 4. (nice to have) dont put all the boilerplate in all poms (license, etc) but > keep it in the parent pom as it was > 5. (if possible) respect the hierarchy (parents) - this is used sometimes as > a shortcut for dependencies analyzis cause it is faster than analyzing the > dependencies, probably not the best practise ever but it is efficient in > general > 6. ensure meta used by mainstream tools like mvnrepository are here > (description etc, should be a passthrough from gradle) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-4057) Ensure generated pom don't break consumers
Romain Manni-Bucau created BEAM-4057: Summary: Ensure generated pom don't break consumers Key: BEAM-4057 URL: https://issues.apache.org/jira/browse/BEAM-4057 Project: Beam Issue Type: Sub-task Components: build-system Reporter: Romain Manni-Bucau Assignee: Luke Cwik Out of my head here are the requirements: 1. dependencies are all here (all scopes and well scoped: this means that provided or test dependencies are not in compile scope for instance) 2. META-INF should contain the pom.xml and pom.properties as maven generates them (it is consumes by tools and libraries to grab the dependencies or scan some classpath/lib folder) 3. ensure the compiler plugin at least is defined with the java version+compiler flags (a usage is to check if -parameters is activated for instance) 4. (nice to have) dont put all the boilerplate in all poms (license, etc) but keep it in the parent pom as it was 5. (if possible) respect the hierarchy (parents) - this is used sometimes as a shortcut for dependencies analyzis cause it is faster than analyzing the dependencies, probably not the best practise ever but it is efficient in general 6. ensure meta used by mainstream tools like mvnrepository are here (description etc, should be a passthrough from gradle) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4047) TransformExecutorServices should use Executor, not ExecutorService
[ https://issues.apache.org/jira/browse/BEAM-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau reassigned BEAM-4047: Assignee: Romain Manni-Bucau > TransformExecutorServices should use Executor, not ExecutorService > -- > > Key: BEAM-4047 > URL: https://issues.apache.org/jira/browse/BEAM-4047 > Project: Beam > Issue Type: Improvement > Components: runner-direct >Reporter: Romain Manni-Bucau >Assignee: Romain Manni-Bucau >Priority: Trivial > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4047) TransformExecutorServices should use Executor, not ExecutorService
[ https://issues.apache.org/jira/browse/BEAM-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau reassigned BEAM-4047: Assignee: (was: Thomas Groh) > TransformExecutorServices should use Executor, not ExecutorService > -- > > Key: BEAM-4047 > URL: https://issues.apache.org/jira/browse/BEAM-4047 > Project: Beam > Issue Type: Improvement > Components: runner-direct >Reporter: Romain Manni-Bucau >Priority: Trivial > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-4047) TransformExecutorServices should use Executor, not ExecutorService
Romain Manni-Bucau created BEAM-4047: Summary: TransformExecutorServices should use Executor, not ExecutorService Key: BEAM-4047 URL: https://issues.apache.org/jira/browse/BEAM-4047 Project: Beam Issue Type: Improvement Components: runner-direct Reporter: Romain Manni-Bucau Assignee: Thomas Groh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4042) Get rid of deprecated gradle API
[
https://issues.apache.org/jira/browse/BEAM-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Romain Manni-Bucau reassigned BEAM-4042:
Assignee: (was: Romain Manni-Bucau)
> Get rid of deprecated gradle API
>
>
> Key: BEAM-4042
> URL: https://issues.apache.org/jira/browse/BEAM-4042
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Romain Manni-Bucau
>Priority: Minor
>
> {code}
> > Task :beam-model-pipeline:shadowJar
> The SimpleWorkResult type has been deprecated and is scheduled to be removed
> in Gradle 5.0. Please use WorkResults.didWork() instead.
> at
> org.gradle.api.internal.tasks.SimpleWorkResult.(SimpleWorkResult.java:34)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
> at
> org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
> at
> com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction.execute(ShadowCopyAction.groovy:99)
> {code}
> to ensure the build output is as expected as possible (no exception in the
> build process when "green") this kind of stack should be fixed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-4042) Get rid of deprecated gradle API
Romain Manni-Bucau created BEAM-4042:
Summary: Get rid of deprecated gradle API
Key: BEAM-4042
URL: https://issues.apache.org/jira/browse/BEAM-4042
Project: Beam
Issue Type: Sub-task
Components: build-system
Reporter: Romain Manni-Bucau
Assignee: Davor Bonaci
{code}
> Task :beam-model-pipeline:shadowJar
The SimpleWorkResult type has been deprecated and is scheduled to be removed in
Gradle 5.0. Please use WorkResults.didWork() instead.
at
org.gradle.api.internal.tasks.SimpleWorkResult.(SimpleWorkResult.java:34)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
at
org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
at
com.github.jengelman.gradle.plugins.shadow.tasks.ShadowCopyAction.execute(ShadowCopyAction.groovy:99)
{code}
to ensure the build output is as expected as possible (no exception in the
build process when "green") this kind of stack should be fixed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-4039) ExecutorServiceParallelExecutor should have a unified shutdown collection
Romain Manni-Bucau created BEAM-4039: Summary: ExecutorServiceParallelExecutor should have a unified shutdown collection Key: BEAM-4039 URL: https://issues.apache.org/jira/browse/BEAM-4039 Project: Beam Issue Type: Improvement Components: runner-direct Reporter: Romain Manni-Bucau Assignee: Thomas Groh Goal of this ticket is to ensure the executor doesn't have to know each single instances of what it destroys at shutdown time. Proposal is to have a list of closeable (or autocloseable) which and a registerCloseable which would decouple it. Side note: it is still key to ensure the list is 1. sorted and 2. any error doesn't prevent to call next element -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3993) rat doesn't pass with gradle
Romain Manni-Bucau created BEAM-3993: Summary: rat doesn't pass with gradle Key: BEAM-3993 URL: https://issues.apache.org/jira/browse/BEAM-3993 Project: Beam Issue Type: Sub-task Components: build-system Reporter: Romain Manni-Bucau Assignee: Romain Manni-Bucau Fix For: 2.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3992) Use JSON-B instead of an hardcoded Jackson
Romain Manni-Bucau created BEAM-3992: Summary: Use JSON-B instead of an hardcoded Jackson Key: BEAM-3992 URL: https://issues.apache.org/jira/browse/BEAM-3992 Project: Beam Issue Type: Improvement Components: sdk-java-core Affects Versions: 2.4.0 Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles Currently beam uses jackson directly everywhere, it either messes up environments where another version is here and can't be upgraded or it forces to shade and have a fat app where it is not needed. Using JSON-B will allow to switch the impl and beam will just rely on ~200k of API instead of megs of dependencies making it smoother to integrated in all environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3119) direct-metrics-counter-committer threads might be leaking
[
https://issues.apache.org/jira/browse/BEAM-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416961#comment-16416961
]
Romain Manni-Bucau commented on BEAM-3119:
--
I confirm this issue, this is not "might" but "does"
> direct-metrics-counter-committer threads might be leaking
> -
>
> Key: BEAM-3119
> URL: https://issues.apache.org/jira/browse/BEAM-3119
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
>Reporter: Etienne Chauchot
>Assignee: Thomas Groh
>Priority: Major
>
> When I run ElasticsearchIOTests using ESv5, there is a thread leak control
> mechanism ({{com.carrotsearch.randomizedtesting.ThreadLeakControl}}). It
> waits for 5s for non-terminated threads at the end of a test. It detects
> leaked {{direct-metrics-counter-committer}} thread.
> {code}
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
> threads that couldn't be terminated:
>1) Thread[id=296, name=direct-metrics-counter-committer,
> state=TIMED_WAITING, group=TGRP-ElasticsearchIOTest]
> at sun.misc.Unsafe.park(Native Method)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at __randomizedtesting.SeedInfo.seed([59E504CA1B0DD6A8]:0){code}
> I tried to increase the timeout to 30s (by patching
> randomizedtesting-runner-2.5.0.jar) but still gets a zombie thread.
> To reproduce, just comment
> {code}
> @ThreadLeakScope(ThreadLeakScope.Scope.NONE)
> {code}
> in
> {code}
> beam/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTest.java
> {code}
> and run
> {code}
> testRead()
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-3876) unbounded source can not have any checkpoint when they end (no data case)
Romain Manni-Bucau created BEAM-3876: Summary: unbounded source can not have any checkpoint when they end (no data case) Key: BEAM-3876 URL: https://issues.apache.org/jira/browse/BEAM-3876 Project: Beam Issue Type: Bug Components: runner-direct Affects Versions: 2.4.0 Reporter: Romain Manni-Bucau Assignee: Romain Manni-Bucau -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-3852) Migrate existing users to new channel ASF slack channel
[ https://issues.apache.org/jira/browse/BEAM-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404129#comment-16404129 ] Romain Manni-Bucau edited comment on BEAM-3852 at 3/18/18 7:14 PM: --- members from apachebeam invited to the-asf #beam channel. [~davor] assigning it back to you for a last check otherwise with the work already done I guess we are good. was (Author: romain.manni-bucau): members from apachebeam invited to the-asf #beam channel. > Migrate existing users to new channel ASF slack channel > --- > > Key: BEAM-3852 > URL: https://issues.apache.org/jira/browse/BEAM-3852 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Luke Cwik >Assignee: Davor Bonaci >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Channel is https://the-asf.slack.com/messages/C9H0YNP3P/ > Created short link for #beam channel directly: > [https://s.apache.org/beam-slack-channel] > Created short link for self-enrollment: https://s.apache.org/slack-invite -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3852) Migrate existing users to new channel ASF slack channel
[ https://issues.apache.org/jira/browse/BEAM-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404129#comment-16404129 ] Romain Manni-Bucau commented on BEAM-3852: -- members from apachebeam invited to the-asf #beam channel. > Migrate existing users to new channel ASF slack channel > --- > > Key: BEAM-3852 > URL: https://issues.apache.org/jira/browse/BEAM-3852 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Luke Cwik >Assignee: Romain Manni-Bucau >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Channel is https://the-asf.slack.com/messages/C9H0YNP3P/ > Created short link for #beam channel directly: > [https://s.apache.org/beam-slack-channel] > Created short link for self-enrollment: https://s.apache.org/slack-invite -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3852) Migrate existing users to new channel ASF slack channel
[ https://issues.apache.org/jira/browse/BEAM-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau reassigned BEAM-3852: Assignee: Davor Bonaci (was: Romain Manni-Bucau) > Migrate existing users to new channel ASF slack channel > --- > > Key: BEAM-3852 > URL: https://issues.apache.org/jira/browse/BEAM-3852 > Project: Beam > Issue Type: New Feature > Components: website >Reporter: Luke Cwik >Assignee: Davor Bonaci >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Channel is https://the-asf.slack.com/messages/C9H0YNP3P/ > Created short link for #beam channel directly: > [https://s.apache.org/beam-slack-channel] > Created short link for self-enrollment: https://s.apache.org/slack-invite -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3794) StateInternals#state(StateNamespace, StateTag) should be a default method
[ https://issues.apache.org/jira/browse/BEAM-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-3794: - Description: Most of implementations just default this method to the favor with a statecontext using StateContexts.nullContext() so it would be beneficial to just do it by default. > StateInternals#state(StateNamespace, StateTag) should be a default method > - > > Key: BEAM-3794 > URL: https://issues.apache.org/jira/browse/BEAM-3794 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Romain Manni-Bucau >Priority: Major > > Most of implementations just default this method to the favor with a > statecontext using StateContexts.nullContext() so it would be beneficial to > just do it by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3794) StateInternals#state(StateNamespace, StateTag) should be a default method
Romain Manni-Bucau created BEAM-3794: Summary: StateInternals#state(StateNamespace, StateTag) should be a default method Key: BEAM-3794 URL: https://issues.apache.org/jira/browse/BEAM-3794 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Romain Manni-Bucau -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3786) Back FileSystem by VFS
Romain Manni-Bucau created BEAM-3786: Summary: Back FileSystem by VFS Key: BEAM-3786 URL: https://issues.apache.org/jira/browse/BEAM-3786 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles A ticket to not forget the discussion about reusing commons-vfs to benefit from its API and connectivity under the hood and automatically behind beam file[system|io] API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3729) Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 cannot access its superinterface"
[ https://issues.apache.org/jira/browse/BEAM-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372945#comment-16372945 ] Romain Manni-Bucau commented on BEAM-3729: -- https://github.com/apache/beam/pull/4729 > Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 > cannot access its superinterface" > - > > Key: BEAM-3729 > URL: https://issues.apache.org/jira/browse/BEAM-3729 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Romain Manni-Bucau >Priority: Blocker > Fix For: 2.4.0 > > > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/4243/console -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3728) Failing ParDoTest for Flink Runner
[ https://issues.apache.org/jira/browse/BEAM-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372944#comment-16372944 ] Romain Manni-Bucau commented on BEAM-3728: -- https://github.com/apache/beam/pull/4729 > Failing ParDoTest for Flink Runner > -- > > Key: BEAM-3728 > URL: https://issues.apache.org/jira/browse/BEAM-3728 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Aljoscha Krettek >Priority: Blocker > Fix For: 2.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest > for the Flink Runner: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/ > I don't currently understand what is causing this. [~lcwik] Do you have any > idea? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3725) Add a test rule which remembers and resets the thread context class loader
[ https://issues.apache.org/jira/browse/BEAM-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371725#comment-16371725 ] Romain Manni-Bucau commented on BEAM-3725: -- [~lcwik] if not urgent (like can wait 1 or 2 weeks) sure > Add a test rule which remembers and resets the thread context class loader > -- > > Key: BEAM-3725 > URL: https://issues.apache.org/jira/browse/BEAM-3725 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Luke Cwik >Priority: Minor > Labels: newbie, starter > > We have several tests that ensure the proper usage of the TCCL but they can > be brittle if the test is written improperly. > > This task is to create a JUnit test rule and to replace the existing > remember/reset TCCL usages in our tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3701) PipelineOptionsFactory doesn't use the right classloader for its SPI
[ https://issues.apache.org/jira/browse/BEAM-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-3701: - Fix Version/s: 2.4.0 > PipelineOptionsFactory doesn't use the right classloader for its SPI > > > Key: BEAM-3701 > URL: https://issues.apache.org/jira/browse/BEAM-3701 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Romain Manni-Bucau >Priority: Major > Fix For: 2.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > PipelineOptionsFactory uses its own classloader to load SPI, this should use > the TCCL to grab what is available in the launching context to enable actual > pluggability in most environment and not limit users to use a flat > classloader environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3479) Add a regression test for the DoFn classloader selection
[ https://issues.apache.org/jira/browse/BEAM-3479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-3479: - Fix Version/s: 2.4.0 > Add a regression test for the DoFn classloader selection > > > Key: BEAM-3479 > URL: https://issues.apache.org/jira/browse/BEAM-3479 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Romain Manni-Bucau >Priority: Major > Fix For: 2.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Follow up task after https://github.com/apache/beam/pull/4235 merge. This > task is about ensuring we test that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3700) PipelineOptionsFactory leaks memory
[ https://issues.apache.org/jira/browse/BEAM-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-3700: - Fix Version/s: 2.4.0 > PipelineOptionsFactory leaks memory > --- > > Key: BEAM-3700 > URL: https://issues.apache.org/jira/browse/BEAM-3700 > Project: Beam > Issue Type: Task > Components: sdk-java-core >Reporter: Romain Manni-Bucau >Assignee: Kenneth Knowles >Priority: Major > Fix For: 2.4.0 > > > PipelineOptionsFactory has a lot of cache but no way to reset it. This task > is about adding a public method to be able to control it in integrations > (runners likely). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests
[ https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-3409: - Fix Version/s: 2.4.0 > Unexpected behavior of DoFn teardown method running in unit tests > -- > > Key: BEAM-3409 > URL: https://issues.apache.org/jira/browse/BEAM-3409 > Project: Beam > Issue Type: Bug > Components: runner-direct >Affects Versions: 2.3.0 >Reporter: Alexey Romanenko >Assignee: Romain Manni-Bucau >Priority: Minor > Labels: test > Fix For: 2.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Writing a unit test, I found out a strange behaviour of Teardown method of > DoFn implementation when I run this method in unit tests using TestPipeline. > To be more precise, it doesn’t wait until teardown() method will be finished, > it just exits from this method after about 1 sec (on my machine) even if it > should take longer (very simple example - running infinite loop inside this > method or put thread in sleep). In the same time, when I run the same code > from main() with ordinary Pipeline and direct runner, then it’s ok and it > works as expected - teardown() method will be performed completely despite > how much time it will take. > I created two test cases to reproduce this issue - the first one to run with > main() and the second one to run with junit. They use the same implementation > of DoFn (class LongTearDownFn) and expects that teardown method will be > running at least for SLEEP_TIME ms. In case of running as junit test it's not > a case (see output log). > - run with main() > https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java > - run with junit > https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3702) Support system properties source for pipeline options
Romain Manni-Bucau created BEAM-3702: Summary: Support system properties source for pipeline options Key: BEAM-3702 URL: https://issues.apache.org/jira/browse/BEAM-3702 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Romain Manni-Bucau Fix For: 2.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3701) PipelineOptionsFactory doesn't use the right classloader for its SPI
Romain Manni-Bucau created BEAM-3701: Summary: PipelineOptionsFactory doesn't use the right classloader for its SPI Key: BEAM-3701 URL: https://issues.apache.org/jira/browse/BEAM-3701 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles PipelineOptionsFactory uses its own classloader to load SPI, this should use the TCCL to grab what is available in the launching context to enable actual pluggability in most environment and not limit users to use a flat classloader environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3700) PipelineOptionsFactory leaks memory
Romain Manni-Bucau created BEAM-3700: Summary: PipelineOptionsFactory leaks memory Key: BEAM-3700 URL: https://issues.apache.org/jira/browse/BEAM-3700 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles PipelineOptionsFactory has a lot of cache but no way to reset it. This task is about adding a public method to be able to control it in integrations (runners likely). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)
Romain Manni-Bucau created BEAM-3616: Summary: Ensure coders are updatable partially in a cluster (aka add versions to coders) Key: BEAM-3616 URL: https://issues.apache.org/jira/browse/BEAM-3616 Project: Beam Issue Type: Improvement Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Kenneth Knowles Fix For: 3.0.0 Follow up ticket for the following discussion: https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E Long story short: seems being able to update a live stream is a constraint beam must address. This implies no change in coder serialization. To do it the only way is to put a marker (like a version) otherwise the live update will be broken if part of a cluster is updated only. This task is about ensuring we support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3616) Ensure coders are updatable partially in a cluster (aka add versions to coders)
[ https://issues.apache.org/jira/browse/BEAM-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Manni-Bucau updated BEAM-3616: - Component/s: sdk-py-core sdk-go > Ensure coders are updatable partially in a cluster (aka add versions to > coders) > --- > > Key: BEAM-3616 > URL: https://issues.apache.org/jira/browse/BEAM-3616 > Project: Beam > Issue Type: Improvement > Components: sdk-go, sdk-java-core, sdk-py-core >Reporter: Romain Manni-Bucau >Assignee: Kenneth Knowles >Priority: Major > Fix For: 3.0.0 > > > Follow up ticket for the following discussion: > https://lists.apache.org/thread.html/f3b2daa740075cc39dc04f08d1eaacfcc2d550cecc04e27be024cf52@%3Cdev.beam.apache.org%3E > Long story short: seems being able to update a live stream is a constraint > beam must address. This implies no change in coder serialization. To do it > the only way is to put a marker (like a version) otherwise the live update > will be broken if part of a cluster is updated only. > This task is about ensuring we support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests
[ https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316634#comment-16316634 ] Romain Manni-Bucau commented on BEAM-3409: -- Hi started to hack on that in https://github.com/rmannibucau/incubator-beam/tree/fix/BEAM-3409_wait-for-teardown-execution-in-direct-runner still need some enhancements to have all tests passing but guess the issue is at least identified > Unexpected behavior of DoFn teardown method running in unit tests > -- > > Key: BEAM-3409 > URL: https://issues.apache.org/jira/browse/BEAM-3409 > Project: Beam > Issue Type: Bug > Components: runner-direct, testing >Affects Versions: 2.3.0 >Reporter: Alexey Romanenko >Assignee: Jean-Baptiste Onofré >Priority: Minor > Labels: test > > Writing a unit test, I found out a strange behaviour of Teardown method of > DoFn implementation when I run this method in unit tests using TestPipeline. > To be more precise, it doesn’t wait until teardown() method will be finished, > it just exits from this method after about 1 sec (on my machine) even if it > should take longer (very simple example - running infinite loop inside this > method or put thread in sleep). In the same time, when I run the same code > from main() with ordinary Pipeline and direct runner, then it’s ok and it > works as expected - teardown() method will be performed completely despite > how much time it will take. > I created two test cases to reproduce this issue - the first one to run with > main() and the second one to run with junit. They use the same implementation > of DoFn (class LongTearDownFn) and expects that teardown method will be > running at least for SLEEP_TIME ms. In case of running as junit test it's not > a case (see output log). > - run with main() > https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java > - run with junit > https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (BEAM-3415) JUnit5 support
Romain Manni-Bucau created BEAM-3415: Summary: JUnit5 support Key: BEAM-3415 URL: https://issues.apache.org/jira/browse/BEAM-3415 Project: Beam Issue Type: New Feature Components: testing Reporter: Romain Manni-Bucau Assignee: Romain Manni-Bucau -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (BEAM-3243) multiple anonymous DoFn lead to conflicting names
Romain Manni-Bucau created BEAM-3243: Summary: multiple anonymous DoFn lead to conflicting names Key: BEAM-3243 URL: https://issues.apache.org/jira/browse/BEAM-3243 Project: Beam Issue Type: Task Components: sdk-java-core Reporter: Romain Manni-Bucau Assignee: Romain Manni-Bucau Fix For: 2.2.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (BEAM-2782) Avoid overlapping "classes"
Romain Manni-Bucau created BEAM-2782:
Summary: Avoid overlapping "classes"
Key: BEAM-2782
URL: https://issues.apache.org/jira/browse/BEAM-2782
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Affects Versions: 2.1.0
Reporter: Romain Manni-Bucau
Assignee: Luke Cwik
Hi guys,
When shading beam we get this warning:
{code}
[INFO] [WARNING] beam-runners-core-java-2.1.0.jar,
beam-runners-core-construction-java-2.1.0.jar define 1 overlapping classes:
[INFO] [WARNING] - org.apache.beam.runners.core.metrics.package-info
{code}
Would be great to avoid it since this one is trivial and it wouldn't break
anything to fix it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (BEAM-2701) use a custom implementation of java.io.ObjectInputStream
Romain Manni-Bucau created BEAM-2701:
Summary: use a custom implementation of java.io.ObjectInputStream
Key: BEAM-2701
URL: https://issues.apache.org/jira/browse/BEAM-2701
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Romain Manni-Bucau
Assignee: Davor Bonaci
java.io.ObjectInputStream should override resolve[Proxy]Class using the TCCL to
support any classloader and not fallback into some JVM pitfall using another
classloader (default). This will enable beam to use any classloader instead of
requiring to run in the JVM using java serialization.
{code}
@Override
protected Class resolveClass(final ObjectStreamClass classDesc) throws
IOException, ClassNotFoundException {
final String n = classDesc.getName();
final ClassLoader classloader = getClassloader();
try {
return Class.forName(n, false, classloader);
} catch (ClassNotFoundException e) {
if (n.equals("boolean")) {
return boolean.class;
}
if (n.equals("byte")) {
return byte.class;
}
if (n.equals("char")) {
return char.class;
}
if (n.equals("short")) {
return short.class;
}
if (n.equals("int")) {
return int.class;
}
if (n.equals("long")) {
return long.class;
}
if (n.equals("float")) {
return float.class;
}
if (n.equals("double")) {
return double.class;
}
//Last try - Let runtime try and find it.
return Class.forName(n, false, null);
}
}
@Override
protected Class resolveProxyClass(final String[] interfaces) throws
IOException, ClassNotFoundException {
final Class[] cinterfaces = new Class[interfaces.length];
for (int i = 0; i < interfaces.length; i++) {
cinterfaces[i] = getClassloader().loadClass(interfaces[i]);
}
try {
return Proxy.getProxyClass(getClassloader(), cinterfaces);
} catch (IllegalArgumentException e) {
throw new ClassNotFoundException(null, e);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
