[jira] [Resolved] (APEXMALHAR-2508) CacheManager throws exception when the operator receives a shutdown request
[ https://issues.apache.org/jira/browse/APEXMALHAR-2508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXMALHAR-2508. --- Resolution: Fixed Fix Version/s: 3.8.0 > CacheManager throws exception when the operator receives a shutdown request > --- > > Key: APEXMALHAR-2508 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2508 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.8.0 > > > CacheManager initializes the Timer (refresher) only if refreshTime is not > null. However, in deactivate, before cancelling the timer, there is no null > check leading to NullPointerException. > Solution : Add a null check before cancelling the timer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (APEXMALHAR-2536) Support for setters on nested Pojo
Tushar Gosavi created APEXMALHAR-2536:
-
Summary: Support for setters on nested Pojo
Key: APEXMALHAR-2536
URL: https://issues.apache.org/jira/browse/APEXMALHAR-2536
Project: Apache Apex Malhar
Issue Type: Improvement
Reporter: Tushar Gosavi
Priority: Minor
Getters support getting values from nested Pojo by adding suitable
methods/fields in the chain
for example
{code}
class Inner {
private int b;
public int getB() { return b; }
public void setB(int b) { this.b = b; }
}
class Nested {
private in a;
private Inner inner;
public int getA() { return a; }
public void getA(int a) { this.a = a; }
public Inner getInner() { return inner; }
public void setInner(Inner i) { this.inner = i; }
}
{code}
With above objects If getter is specified as `inner.b` then the expression is
created with
`object.getInner().getB()` which compiles and gives result correctly.
If setter is specified with same expression it fails to compile as inner is a
private object.
We plan to include similar functionality for setters by following chain of
objects and constructing valid java expressions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (APEXMALHAR-2535) Timeouts in AbstractEnricher specified as int which limits duration of time which could be specified.
Tushar Gosavi created APEXMALHAR-2535: - Summary: Timeouts in AbstractEnricher specified as int which limits duration of time which could be specified. Key: APEXMALHAR-2535 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2535 Project: Apache Apex Malhar Issue Type: Bug Reporter: Tushar Gosavi Priority: Minor cacheExpirationInterval and cacheCleanupInterval timeout properties are using int as datatype which limits the amount of time duration could be specified using them. I am going to change they to long also provide a way for operator to specify expiryType on cacheStore -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (APEXCORE-720) In DAGExecution plugin, Context.getDAG returns null
[ https://issues.apache.org/jira/browse/APEXCORE-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXCORE-720. Resolution: Fixed Fix Version/s: 3.7.0 > In DAGExecution plugin, Context.getDAG returns null > --- > > Key: APEXCORE-720 > URL: https://issues.apache.org/jira/browse/APEXCORE-720 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Chinmay Kolhatkar >Assignee: Chinmay Kolhatkar > Fix For: 3.7.0 > > > When plugin calls context.getDAG method, it returns null. > The method should return a LogicalPlan object. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-718) How to Use one Kafka input operator and Feed to multiple operator
[
https://issues.apache.org/jira/browse/APEXCORE-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006246#comment-16006246
]
Tushar Gosavi commented on APEXCORE-718:
Please use us...@apex.apache.org mailing list for asking questions about apex
before opening an JIRA. Use JIRA for bug/improvement related to code and
documentation.
The stream for which exception is thrown is MapData, and the stream is not
present in the code snipped you have provided. In general you can't use the
same port twice if using addStream api. For sending data to multiple operators
you can connected same output port to multiple input ports.
dag.addStreams("streamname", outputPorts, in1, in2, in3 ...)
You can not connect single input port to multiple output ports.
> How to Use one Kafka input operator and Feed to multiple operator
> --
>
> Key: APEXCORE-718
> URL: https://issues.apache.org/jira/browse/APEXCORE-718
> Project: Apache Apex Core
> Issue Type: Question
>Reporter: ROHIT RAJKUMAR GARG
>
> I want to read from a topic using the kafka input operator and then feed it
> to multiple operator . in short it will be a sort of branch
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-585) Latency should be calculated only after the first window has been complete
[ https://issues.apache.org/jira/browse/APEXCORE-585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-585. -- Closing after 3.6.0 release > Latency should be calculated only after the first window has been complete > -- > > Key: APEXCORE-585 > URL: https://issues.apache.org/jira/browse/APEXCORE-585 > Project: Apache Apex Core > Issue Type: Bug >Reporter: David Yan >Assignee: David Yan >Priority: Minor > Fix For: 3.6.0 > > > From the latency unit test, it tries to calculate the latency by looking at > the difference between current window id and upstream's current window id > when the end window stats is not available. But when the current window id is > 0 (first window has not been complete), it would return some wacky latency > value because the current window id is 0. > This would not be an issue if the window id was not a simple sequence. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-715) Remove unnecessary @Evolving annotation in engine
[ https://issues.apache.org/jira/browse/APEXCORE-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-715. -- Closing after 3.6.0 release > Remove unnecessary @Evolving annotation in engine > - > > Key: APEXCORE-715 > URL: https://issues.apache.org/jira/browse/APEXCORE-715 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > \@Evolving annotation in the engine are not necessary and introduce > unnecessarily practice and wrong impression that some interfaces and/or > classes are more stable compared to others. All interfaces and classes in the > engine are not subject to the semantic version checks and may change from a > release to a release. The same applies to the buffer server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-617) InputNodeTest intermittently fails with ConcurrentModificationException
[ https://issues.apache.org/jira/browse/APEXCORE-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-617. -- Closing after 3.6.0 release > InputNodeTest intermittently fails with ConcurrentModificationException > --- > > Key: APEXCORE-617 > URL: https://issues.apache.org/jira/browse/APEXCORE-617 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > InputNodeTest.emitTestHelper uses deprecated inherently unsafe Thread.stop(). > Thread.stop() does not immediately stops the thread, the JVM waits for a > safepoint to raise ThreadDeath exception on the target thread. This leads to > a race condition and possible ConcurrentModificationException. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-641) Subscribers/DataListeners may not be scheduled to execute even when they have data to process
[ https://issues.apache.org/jira/browse/APEXCORE-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-641. -- Closing after 3.6.0 release > Subscribers/DataListeners may not be scheduled to execute even when they have > data to process > - > > Key: APEXCORE-641 > URL: https://issues.apache.org/jira/browse/APEXCORE-641 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.0, 3.3.0, 3.2.1, 3.4.0, 3.5.0, 3.6.0 >Reporter: Vlad Rozov >Assignee: Vlad Rozov > Fix For: 3.6.0 > > > Buffer server iterates over DataListeners aka LogicalNodes and each > LogicalNode tries to send to it's downstream all data that Publisher added to > the DataList. When an output port is connected to multiple partitions or > downstream operators (2 or more DataListeners/LogicalNodes) there may be more > data published to the DataList after first few DataListeners in the listeners > set iterated over DataList and reached the last block published so far. The > data published while the last DataListeners sends data to it's downstream > will not be processed by other DataListeners until Publisher adds more data > to the DataList. This may lead to blocked operators, as Buffer server may > stop processing data completely in case Publisher fills more than one Data > block while a single DataListener sends data to it's downstream and there are > more Subscribers/DataListeners than number of in memory blocks allowed (8). > In such case, Publisher will be suspended, and there will be no task > scheduled to process data already published to the DataList. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-596) Committed method on operators not called when stream locality is THREAD_LOCAL
[
https://issues.apache.org/jira/browse/APEXCORE-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-596.
--
Closing after 3.6.0 release
> Committed method on operators not called when stream locality is THREAD_LOCAL
> -
>
> Key: APEXCORE-596
> URL: https://issues.apache.org/jira/browse/APEXCORE-596
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Francis Fernandes
>Assignee: Francis Fernandes
>Priority: Minor
> Fix For: 3.6.0
>
>
> When the locality of the stream connecting the two operators is
> Locality.THREAD_LOCAL, the committed method is not called for some operators.
> These operators implement the Operator.CheckpointListener. e.g.
> AbstractFileOutputOperator
> For thread local during activate we do not set the thread in the node's
> context
> Because the thread is not set, we skip this operator in the
> processHeartBeatResponse and the committed is not called
> {code}
> if (thread == null || !thread.isAlive()) {
> continue;
> }
> {code}
> We need this condition for invalid operators (operator failures) in case of
> other localities.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-662) Raise StramEvent for heartbeat miss
[ https://issues.apache.org/jira/browse/APEXCORE-662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-662. -- Closing after 3.6.0 release > Raise StramEvent for heartbeat miss > --- > > Key: APEXCORE-662 > URL: https://issues.apache.org/jira/browse/APEXCORE-662 > Project: Apache Apex Core > Issue Type: Task >Reporter: Tushar Gosavi >Assignee: Hitesh Kapoor >Priority: Minor > Fix For: 3.6.0 > > > One of the cause of container kill is heartbeat miss. we can raise an > StramEvent in case of heartbeat miss to make debugging easier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-683) Apex client should support application packages on HDFS
[
https://issues.apache.org/jira/browse/APEXCORE-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-683.
--
Closing after 3.6.0 release
> Apex client should support application packages on HDFS
> ---
>
> Key: APEXCORE-683
> URL: https://issues.apache.org/jira/browse/APEXCORE-683
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
> Fix For: 3.6.0
>
>
> apex client supports only local file system, it is natural for native yarn
> application to support hdfs and other dfs file systems:
> {noformat}
> hdfs dfs -ls hdfs:///user/vrozov
> Found 4 items
> drwx-- - vrozov supergroup 0 2016-12-24 16:00
> hdfs:///user/vrozov/.Trash
> drwxrwxrwx - vrozov supergroup 0 2017-02-28 19:20
> hdfs:///user/vrozov/.flink
> -rw-r--r-- 3 vrozov supergroup 18862977 2017-03-25 09:20
> hdfs:///user/vrozov/appPackage.apa
> drwxrwxrwx - vrozov supergroup 0 2016-10-25 21:49
> hdfs:///user/vrozov/datatorrent
> [vrozov@node0 apex]$ apex
> Apex CLI 3.6.0-SNAPSHOT 25.03.2017 @ 09:17:45 PDT rev: b4a4e05 branch: master
> apex> get-app-package-info hdfs:///user/vrozov/appPackage.apa
> java.io.FileNotFoundException: hdfs:/user/vrozov/appPackage.apa (No such file
> or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:215)
> at java.util.zip.ZipFile.(ZipFile.java:145)
> at java.util.jar.JarFile.(JarFile.java:154)
> at java.util.jar.JarFile.(JarFile.java:118)
> at com.datatorrent.stram.client.AppPackage.(AppPackage.java:185)
> at com.datatorrent.stram.client.AppPackage.(AppPackage.java:258)
> at
> com.datatorrent.stram.cli.ApexCli.newAppPackageInstance(ApexCli.java:467)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageInfoCommand.execute(ApexCli.java:3502)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1520)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-655) Support RELEASE as archetype version when creating a project
[ https://issues.apache.org/jira/browse/APEXCORE-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-655. -- Closing after 3.6.0 release > Support RELEASE as archetype version when creating a project > - > > Key: APEXCORE-655 > URL: https://issues.apache.org/jira/browse/APEXCORE-655 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Thomas Weise >Priority: Minor > Fix For: 3.6.0 > > > Rather than keep on replacing the Apex archetype version number in various > examples, Maven supports LATEST to refer to the latest released archetype > version. > This requires a change to decouple resolution of Apex dependency version from > archetype version to replace the Apex version when the archetype is created > vs. when the project is created. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-511) DAG.addOperator, addStream and addModule should check for null and empty names
[ https://issues.apache.org/jira/browse/APEXCORE-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-511. -- Closing after 3.6.0 release > DAG.addOperator, addStream and addModule should check for null and empty names > -- > > Key: APEXCORE-511 > URL: https://issues.apache.org/jira/browse/APEXCORE-511 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Oliver Winke >Priority: Minor > Labels: newbie > Fix For: 3.6.0 > > > Operators, Streams and Modules should have valid not empty names. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-701) Upgrade Malhar version in archetype to 3.7.0
[ https://issues.apache.org/jira/browse/APEXCORE-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-701. -- Closing after 3.6.0 release > Upgrade Malhar version in archetype to 3.7.0 > - > > Key: APEXCORE-701 > URL: https://issues.apache.org/jira/browse/APEXCORE-701 > Project: Apache Apex Core > Issue Type: Task >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Trivial > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-671) DTConfiguration utility class ValueEntry access level was changed
[ https://issues.apache.org/jira/browse/APEXCORE-671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-671. -- Closing after 3.6.0 release > DTConfiguration utility class ValueEntry access level was changed > - > > Key: APEXCORE-671 > URL: https://issues.apache.org/jira/browse/APEXCORE-671 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni >Priority: Trivial > Fix For: 3.6.0 > > > DTConfiguration.ValueEntry that was public was changed to private. This needs > to be reverted back and a better solution needs to be found in future for > items that would fall into stram/api. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-644) get-app-package-operators with parent option does not work
[
https://issues.apache.org/jira/browse/APEXCORE-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-644.
--
Closing after 3.6.0 release
> get-app-package-operators with parent option does not work
> --
>
> Key: APEXCORE-644
> URL: https://issues.apache.org/jira/browse/APEXCORE-644
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Yatin Chaubal
>Assignee: Sergey Golovko
>Priority: Minor
> Fix For: 3.6.0
>
>
> Issue: get-app-package-operators with -parent option doesnot work
>
> Steps:
> 1) Start dtcli/apex
> 2) Run get-app-package-operators -parent com.datatorrent.demos.pi
> /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> Expected out output: valid JSON
> Actual output:
> {noformat}
> com.datatorrent.stram.cli.ApexCli$CliException:
> /home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
> at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
> at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
> {noformat}
> Reference:
> Without -parent option this work fine
> apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> {
> "operatorClasses": [
> {
> "name": "com.datatorrent.common.util.DefaultDelayOperator",
> "properties": [],
> "portTypeInfo": [
> {
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-294) Graceful application shutdown
[ https://issues.apache.org/jira/browse/APEXCORE-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-294. -- Closing after 3.6.0 release > Graceful application shutdown > - > > Key: APEXCORE-294 > URL: https://issues.apache.org/jira/browse/APEXCORE-294 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Thomas Weise >Assignee: Tushar Gosavi > Fix For: 3.6.0 > > > By injecting the end stream tuple into input operators, to replace the > current mechanism of forced operator undeploy. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-575) Improve application relaunch time.
[ https://issues.apache.org/jira/browse/APEXCORE-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-575. -- Closing after 3.6.0 release > Improve application relaunch time. > -- > > Key: APEXCORE-575 > URL: https://issues.apache.org/jira/browse/APEXCORE-575 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi > Fix For: 3.6.0 > > > Improve application relaunch time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-594) Plugin support in Apex
[ https://issues.apache.org/jira/browse/APEXCORE-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-594. -- Closing after 3.6.0 release > Plugin support in Apex > -- > > Key: APEXCORE-594 > URL: https://issues.apache.org/jira/browse/APEXCORE-594 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Tushar Gosavi > Fix For: 3.6.0 > > > Investigate adding new functionality to apex through user plugins. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-627) Unit test AtMostOnceTest intermittently fails
[ https://issues.apache.org/jira/browse/APEXCORE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-627. -- Closing after 3.6.0 release > Unit test AtMostOnceTest intermittently fails > - > > Key: APEXCORE-627 > URL: https://issues.apache.org/jira/browse/APEXCORE-627 > Project: Apache Apex Core > Issue Type: Bug > Environment: The test is reproducible on macOS Sierra, Processor 2.2 > GHz Intel Core i7, Memory 16GB 1600 MHz DDR3. >Reporter: Sergey Golovko >Assignee: Sergey Golovko >Priority: Minor > Fix For: 3.6.0 > > > The test AtMostOnceTest is not able to reach the criteria to stop the test. > And it continue to recover an input operator and rerun the test in a loop. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-616) Application fails to start Kerberised cluster
[ https://issues.apache.org/jira/browse/APEXCORE-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-616. -- Closing after 3.6.0 release > Application fails to start Kerberised cluster > - > > Key: APEXCORE-616 > URL: https://issues.apache.org/jira/browse/APEXCORE-616 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 > Environment: CDH - any >Reporter: Venkatesh Kottapalli >Assignee: Pramod Immaneni > Fix For: 3.6.0 > > > Application fails to start in a Kerberised cluster with the following > exception. > 2017-01-20 19:28:07,408 [main] ERROR stram.StreamingAppMaster main - Exiting > Application Master > java.lang.NoClassDefFoundError: > com/sun/jersey/client/apache4/ApacheHttpClient4Handler > at > com.datatorrent.stram.util.WebServicesClient.(WebServicesClient.java:153) > at > com.datatorrent.stram.util.WebServicesClient.(WebServicesClient.java:140) > at > com.datatorrent.stram.StreamingContainerManager.getAppMasterContainerInfo(StreamingContainerManager.java:481) > at > com.datatorrent.stram.StreamingContainerManager.init(StreamingContainerManager.java:448) > at > com.datatorrent.stram.StreamingContainerManager.(StreamingContainerManager.java:420) > at > com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:3065) > at > com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:552) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:102) > Caused by: java.lang.ClassNotFoundException: > com.sun.jersey.client.apache4.ApacheHttpClient4Handler > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 9 more -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-702) Mark ApexPlugin as Evolving
[ https://issues.apache.org/jira/browse/APEXCORE-702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-702. -- Closing after 3.6.0 release > Mark ApexPlugin as Evolving > --- > > Key: APEXCORE-702 > URL: https://issues.apache.org/jira/browse/APEXCORE-702 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi >Priority: Minor > Fix For: 3.6.0 > > > Make ApexPlugin and related classes as Evolving and this part of code is new > and might change in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-590) Failed to restart application on MapR
[ https://issues.apache.org/jira/browse/APEXCORE-590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-590. -- Closing after 3.6.0 release > Failed to restart application on MapR > - > > Key: APEXCORE-590 > URL: https://issues.apache.org/jira/browse/APEXCORE-590 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Pradeep A. Dalvi >Assignee: Pradeep A. Dalvi > Fix For: 3.6.0 > > > For restarting application, we try to copy previous app state i.e. > checkpoints directory from original app. However checkpoints are not being > copied due to incorrect check of source and destination directory path. > 16/12/16 13:28:32 ERROR fs.MapRFileSystem: Failed to delete path > maprfs:/user/dtadmin/datatorrent/apps/application_1481890072066_0006/checkpoints, > error: No such file or directory (2) > 16/12/16 13:28:32 INFO stram.FSRecoveryHandler: Creating > maprfs:/user/dtadmin/datatorrent/apps/application_1481890072066_0006/recovery/log > 16/12/16 13:28:32 INFO stram.StramClient: Ignoring > maprfs:///user/dtadmin/datatorrent/apps/application_1481890072066_0004/events > as it already exists under > maprfs:///user/dtadmin/datatorrent/apps/application_1481890072066_0004/events > 16/12/16 13:28:32 INFO stram.StramClient: Ignoring > maprfs:///user/dtadmin/datatorrent/apps/application_1481890072066_0004/recovery > as it already exists under > maprfs:///user/dtadmin/datatorrent/apps/application_1481890072066_0004/recovery > 16/12/16 13:28:32 INFO stram.StramClient: Ignoring > maprfs:///user/dtadmin/datatorrent/apps/application_1481890072066_0004/checkpoints > as it already exists under > maprfs:///user/dtadmin/datatorrent/apps/application_1481890072066_0004/checkpoints > 16/12/16 13:28:32 INFO stram.StramClient: Set the environment for the > application master -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-608) Streaming Containers use stale RPC proxy after connection is closed
[ https://issues.apache.org/jira/browse/APEXCORE-608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-608. -- Closing after 3.6.0 release > Streaming Containers use stale RPC proxy after connection is closed > --- > > Key: APEXCORE-608 > URL: https://issues.apache.org/jira/browse/APEXCORE-608 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > When Application is killed and Application Master is terminated, Streaming > Containers initiate container exit sequence and use disconnected RPC proxy to > report errors back to already terminated Application Master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-678) Shutdown of application should start from input nodes
[
https://issues.apache.org/jira/browse/APEXCORE-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-678.
--
Closing after 3.6.0 release
> Shutdown of application should start from input nodes
> -
>
> Key: APEXCORE-678
> URL: https://issues.apache.org/jira/browse/APEXCORE-678
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Bhupesh Chawda
>Assignee: Bhupesh Chawda
> Fix For: 3.6.0
>
>
> Streaming container calls shutdown() for all nodes instead of just input
> nodes.
> {code}
> private void stopInputNodes()
> {
> for (Entry e : nodes.entrySet()) {
> Node node = e.getValue();
> if (node instanceof InputNode) {
> final Thread thread = e.getValue().context.getThread();
> if (thread == null || !thread.isAlive()) {
> continue;
> }
> }
> node.shutdown(true);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-674) Change access specifier of DTConfiguration.ValueEntry to private
[ https://issues.apache.org/jira/browse/APEXCORE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-674. -- Closing after 3.6.0 release > Change access specifier of DTConfiguration.ValueEntry to private > > > Key: APEXCORE-674 > URL: https://issues.apache.org/jira/browse/APEXCORE-674 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Trivial > Fix For: 3.6.0 > > > There is no need for ValueEntry class to be public. DTConfiguration provides > methods to access the member variables of ValueEntry class. Also, the setter > functions set() and setInternal() will be changed to return void instead of > ValueEntry. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow
[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-570. -- Closing after 3.6.0 release > Prevent upstream operators from getting too far ahead when downstream > operators are slow > > > Key: APEXCORE-570 > URL: https://issues.apache.org/jira/browse/APEXCORE-570 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > Fix For: 3.6.0 > > > If the downstream operators are slower than upstream operators then the > upstream operators will get ahead and the gap can continue to increase. > Provide an option to slow down or temporarily pause the upstream operators > when they get too far ahead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-654) Recovery window is not updated when Delay Operator is used along with Partitioned Operators
[ https://issues.apache.org/jira/browse/APEXCORE-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-654. -- Closing after 3.6.0 release > Recovery window is not updated when Delay Operator is used along with > Partitioned Operators > --- > > Key: APEXCORE-654 > URL: https://issues.apache.org/jira/browse/APEXCORE-654 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 > Environment: Hadoop 2.7.2 > Apache Apex 3.5.0 > Apache Apex Malhar 3.6.0 >Reporter: Ambarish Pande >Assignee: Bhupesh Chawda > Labels: DelayOperator > Fix For: 3.6.0 > > Attachments: ProblemDag.png > > > Checkpointing is not happening when DefaultDelayOperator is used in a DAG in > which some upstream operators are Partitioned. > When used without partitioning, I can see the operators being check-pointed > properly. > Here is the link of the App source code and also the built apa file. > https://github.com/ambarishpande/delay-operator-test -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-579) Custom control tuple support
[ https://issues.apache.org/jira/browse/APEXCORE-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-579. -- Closing after 3.6.0 release > Custom control tuple support > > > Key: APEXCORE-579 > URL: https://issues.apache.org/jira/browse/APEXCORE-579 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > The discussion thread is here: > https://lists.apache.org/thread.html/9accc404a3cb17af2de73b860ed7dff4523a837b4d3c1c73fe0858c5@%3Cdev.apex.apache.org%3E > We will implement option 1 for now and park option 2 until we have a real use > case. > We probably need a LinkedHashMap to store all the control tuples received in > memory (to preserve the order and uniqueness). > The assumption is that we will not have a large number of control tuples > within a streaming window and the memory footprint of control tuples is > small. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-605) Suppress bootstrap compiler warning
[
https://issues.apache.org/jira/browse/APEXCORE-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-605.
--
Closing after 3.6.0 release
> Suppress bootstrap compiler warning
> ---
>
> Key: APEXCORE-605
> URL: https://issues.apache.org/jira/browse/APEXCORE-605
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
> Fix For: 3.6.0
>
>
> Apex uses animal-sniffer-maven-plugin to enforce 1.7 API compatibility. The
> following warning caused by {{1.7}} compiler plugin option
> should be suppressed.
> {noformat}
> [WARNING] bootstrap class path not set in conjunction with -source 1.7
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info
[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-592. -- Closing after 3.6.0 release > Returning description field in defaultProperties during apex cli call > get-app-package-info > -- > > Key: APEXCORE-592 > URL: https://issues.apache.org/jira/browse/APEXCORE-592 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.6.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Currently for an operator property, only the value field is returned. The > operator property can have a description describing the property. This task > is created to return this filed for that operator property. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-624) Shutdown does not work because of incorrect logic in the AppMaster
[ https://issues.apache.org/jira/browse/APEXCORE-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-624. -- Closing after 3.6.0 release > Shutdown does not work because of incorrect logic in the AppMaster > -- > > Key: APEXCORE-624 > URL: https://issues.apache.org/jira/browse/APEXCORE-624 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.3.1, 3.2.2, 3.5.1, 3.6.0, 3.4.1 > > > com.datatorrent.stram.StreamingAppMasterService.execute() calculates > numRequestedContainers incorrectly in some cases (e.g. RM container > allocation failure) which prevents an application from shutting down when it > is requested externally. An example is where we ask RM to remove previous > container allocation request (where the count should be decremented but is > NOT) and add a new one (where the count should be and IS incremented). > Another example is the "alreadyAllocated" case where we release the container > and still increment numRequestedContainers which seems wrong. > This bug is showing up in multiple Apex deployments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-663) Application restart not working.
[ https://issues.apache.org/jira/browse/APEXCORE-663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-663. -- Closing after 3.6.0 release > Application restart not working. > > > Key: APEXCORE-663 > URL: https://issues.apache.org/jira/browse/APEXCORE-663 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Assignee: Sandesh > Fix For: 3.6.0 > > > Application restart is not working. Steps to reproduce > - launch PiDemo > - Wait for some time (till recovery windowId reaches 120 >) > - kill the app > - Try to relaunch using 'launch -originalAppId' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-563) Have a pointer to log file name and offset in container/operator failure events
[ https://issues.apache.org/jira/browse/APEXCORE-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-563. -- Closing after 3.6.0 release > Have a pointer to log file name and offset in container/operator failure > events > --- > > Key: APEXCORE-563 > URL: https://issues.apache.org/jira/browse/APEXCORE-563 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Sanjay M Pujare >Assignee: Priyanka Gugale > Fix For: 3.6.0 > > > The default DailyRollingFileAppender does not take into account of how many > backup files to keep and it will result in unbounded growth of log files, > especially for long running applications. > The below is an interesting add-on to the default DailyRollingFileAppender > that supports maxBackupIndex. > http://wiki.apache.org/logging-log4j/DailyRollingFileAppender -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-680) Review container heartbeat timeout log level messages
[ https://issues.apache.org/jira/browse/APEXCORE-680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-680. -- Closing after 3.6.0 release > Review container heartbeat timeout log level messages > -- > > Key: APEXCORE-680 > URL: https://issues.apache.org/jira/browse/APEXCORE-680 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Vlad Rozov >Assignee: Hitesh Kapoor >Priority: Trivial > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-577) Plugin support to inspect DAG before launch.
[
https://issues.apache.org/jira/browse/APEXCORE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-577.
--
Closing after 3.6.0 release
> Plugin support to inspect DAG before launch.
>
>
> Key: APEXCORE-577
> URL: https://issues.apache.org/jira/browse/APEXCORE-577
> Project: Apache Apex Core
> Issue Type: Sub-task
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
> Fix For: 3.6.0
>
>
> Support an Visitor like api for DAG, which could be used to write plugins for
> Apex.
> change DAG interface to add a visitor
> {code}
> registerVisitor(DagVisitor visitor)
> {code}
> The DagVisitor interface
> {code}
> interface DagVisitor {
> boolean startDAG(DAG* dag);
> boolean visitOperator(OperatorMeta *ometa)
> boolean visitStream(StreamMeta *smeta);
> boolean endDAG()
> }
> {code}
> dev thread discussions
> https://lists.apache.org/thread.html/1c37e0954cee029c9de537cc35ecb35beebdf49aba17de89bd2ce9ed@%3Cdev.apex.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-583) Buffer Server LogicalNode should not be reused by Subscribers
[ https://issues.apache.org/jira/browse/APEXCORE-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-583. -- Closing after 3.6.0 release > Buffer Server LogicalNode should not be reused by Subscribers > - > > Key: APEXCORE-583 > URL: https://issues.apache.org/jira/browse/APEXCORE-583 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Vlad Rozov >Assignee: Vlad Rozov > Fix For: 3.6.0 > > > Currently LogicalNode supports only one Subscriber and it can't be reused by > a new Subscriber. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-700) Make the plugin registration interface uniform
[ https://issues.apache.org/jira/browse/APEXCORE-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-700. -- Closing after 3.6.0 release > Make the plugin registration interface uniform > -- > > Key: APEXCORE-700 > URL: https://issues.apache.org/jira/browse/APEXCORE-700 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni >Priority: Minor > Fix For: 3.6.0 > > > The user-facing plugin registration for DAG setup plugins is slightly > different from runtime plugins. It would be better to have a uniform way to > do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-597) BufferServer needs to shutdown all created execution services
[ https://issues.apache.org/jira/browse/APEXCORE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-597. -- Closing after 3.6.0 release > BufferServer needs to shutdown all created execution services > - > > Key: APEXCORE-597 > URL: https://issues.apache.org/jira/browse/APEXCORE-597 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Tushar Gosavi >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > JVM does not shutdown cleanly after receiving shutdown request from Stram. > The issue may be because BufferServer is not shutdown. The issue is seen > after commit d1646e42bdf5594ef34070594733a7ca10123a3f > The demo application to recreate the issue is at > https://github.com/tushargosavi/apex-malhar/blob/shutdownapp/demos/pi/src/main/java/com/datatorrent/demos/pi/Application.java -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-591) SubscribeRequestTuple has wrong buffer size when mask is zero
[ https://issues.apache.org/jira/browse/APEXCORE-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-591. -- Closing after 3.6.0 release > SubscribeRequestTuple has wrong buffer size when mask is zero > - > > Key: APEXCORE-591 > URL: https://issues.apache.org/jira/browse/APEXCORE-591 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-645) StramLocalCluster does not wait for master thread termination
[ https://issues.apache.org/jira/browse/APEXCORE-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-645. -- Closing after 3.6.0 release > StramLocalCluster does not wait for master thread termination > - > > Key: APEXCORE-645 > URL: https://issues.apache.org/jira/browse/APEXCORE-645 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > When StramLocalCluster is started asynchroniously it does not wait for the > master thread to terminate -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-598) Embedded mode execution does not use APPLICATION_PATH for checkpointing
[ https://issues.apache.org/jira/browse/APEXCORE-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-598. -- Closing after 3.6.0 release > Embedded mode execution does not use APPLICATION_PATH for checkpointing > --- > > Key: APEXCORE-598 > URL: https://issues.apache.org/jira/browse/APEXCORE-598 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 3.6.0 > > > Checkpoints are written to a default location, and when tests execute in > parallel they will interfere with each other. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-572) Remove dependency on hadoop-common test.jar
[ https://issues.apache.org/jira/browse/APEXCORE-572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-572. -- Closing after 3.6.0 release > Remove dependency on hadoop-common test.jar > --- > > Key: APEXCORE-572 > URL: https://issues.apache.org/jira/browse/APEXCORE-572 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > The dependency on hadoop-common test-jar can be avoided by implementing a few > helper methods. It is not necessary to have such dependency. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-558) Do not use yellow color to display command strings in help output
[ https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-558. -- Closing after 3.6.0 release > Do not use yellow color to display command strings in help output > - > > Key: APEXCORE-558 > URL: https://issues.apache.org/jira/browse/APEXCORE-558 > Project: Apache Apex Core > Issue Type: Bug > Environment: MacOS, Linux >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.6.0 > > > Apex CLI Help output (at least on MacOS and Linux terminals) shows command > strings in yellow which is extremely hard to read and reduces usability of > the CLI. Should be changed to red or purple or just be bolded in black. > In the following output "alias" and "begin-macro" are in yellow and hard to > read. > apex> help > GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN: > alias alias-name command > Create a command alias > begin-macro name > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-610) Avoid multiple getBytes() calls in Tuple.writeString
[ https://issues.apache.org/jira/browse/APEXCORE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-610. -- Closing after 3.6.0 release > Avoid multiple getBytes() calls in Tuple.writeString > > > Key: APEXCORE-610 > URL: https://issues.apache.org/jira/browse/APEXCORE-610 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi >Priority: Trivial > Fix For: 3.6.0 > > > Tuple.writeString calls getBytes() multiple times for identifier. each call > to getBytes returns new byte array hence the call is costly in terms of > memory usage. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-676) Show description for DefaultProperties only when user requests it
[ https://issues.apache.org/jira/browse/APEXCORE-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-676. -- Closing after 3.6.0 release > Show description for DefaultProperties only when user requests it > - > > Key: APEXCORE-676 > URL: https://issues.apache.org/jira/browse/APEXCORE-676 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.6.0 > > > Apex CLI shows description for default properties, this feature was added > recently as a part of > https://issues.apache.org/jira/browse/APEXCORE-592 > But description should be shown only when a user requests it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-456) Explicitly limit Server.Subscriber to one way communication
[ https://issues.apache.org/jira/browse/APEXCORE-456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-456. -- Closing after 3.6.0 release > Explicitly limit Server.Subscriber to one way communication > --- > > Key: APEXCORE-456 > URL: https://issues.apache.org/jira/browse/APEXCORE-456 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov > Fix For: 3.6.0 > > > In the DAG data always flows from upstream to downstream operators and buffer > server Server.Subscriber class may explicitly shutdown input stream, does not > need to allocate read buffer and listen to OP_READ events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-687) Update docs, change supported Hadoop version to 2.6
[ https://issues.apache.org/jira/browse/APEXCORE-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-687. -- Closing after 3.6.0 release > Update docs, change supported Hadoop version to 2.6 > --- > > Key: APEXCORE-687 > URL: https://issues.apache.org/jira/browse/APEXCORE-687 > Project: Apache Apex Core > Issue Type: Documentation > Components: Documentation >Reporter: Dean Lockgaard >Assignee: Dean Lockgaard >Priority: Minor > Fix For: 3.6.0 > > > The doc at > https://github.com/apache/apex-core/blob/master/docs/compatibility.md refers > to Hadoop version 2.2, but it is now 2.6. There may be other instances in > the docs where this version number would need to be changed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-695) Remove unnecessary interface modifiers
[ https://issues.apache.org/jira/browse/APEXCORE-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-695. -- Closing after 3.6.0 release > Remove unnecessary interface modifiers > -- > > Key: APEXCORE-695 > URL: https://issues.apache.org/jira/browse/APEXCORE-695 > Project: Apache Apex Core > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > public modifier is redundant in interfaces -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-690) Embedded app launcher does not apply HEARTBEAT_MONITORING setting
[ https://issues.apache.org/jira/browse/APEXCORE-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-690. -- Closing after 3.6.0 release > Embedded app launcher does not apply HEARTBEAT_MONITORING setting > - > > Key: APEXCORE-690 > URL: https://issues.apache.org/jira/browse/APEXCORE-690 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 3.6.0 > > > The setting is only applied when true (which is the default). The purpose of > the setting is to be able to turn it off though, such as for debugging the > DAG in the IDE. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-604) Extend DAG API to allow accessing DAG objects.
[ https://issues.apache.org/jira/browse/APEXCORE-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-604. -- Closing after 3.6.0 release > Extend DAG API to allow accessing DAG objects. > -- > > Key: APEXCORE-604 > URL: https://issues.apache.org/jira/browse/APEXCORE-604 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi > Fix For: 3.6.0 > > > I am planing extend the DAG api to export internals of the DAG. > Current DAG does not provide a way to get the list of operators and > streams with their attributes. Also streamMeta does not provide > API to access end ports. > This type of information is needed when external translator (like > Samoa) are constructing the DAG, > and we need to apply some transformation or configure the DAG before > it is run. Planing to extend > DAG API with following. > InputPortMeta > public Operator.InputPort getPortObject(); > OutputPortMeta > public Operator.OutputPort getPortObject(); > streamMeta > public T getSource(); > public Collection getSinks(); > DAG > public abstract Collection getOperators(); > public abstract Collection getStreams(); -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-593) apex cli get-app-package-info could not retrieve properties defined in properties.xml
[ https://issues.apache.org/jira/browse/APEXCORE-593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-593. -- Closing after 3.6.0 release > apex cli get-app-package-info could not retrieve properties defined in > properties.xml > - > > Key: APEXCORE-593 > URL: https://issues.apache.org/jira/browse/APEXCORE-593 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Yogi Devendra >Assignee: Vikram Patil >Priority: Minor > Fix For: 3.6.0 > > > If application defines properties in properties.xml; such properties should > be available to populateDAG() method when invoked from processAppDirectory() > in get-app-package-info. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-471) Requests for container allocation are not resubmitted
[ https://issues.apache.org/jira/browse/APEXCORE-471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-471. -- Closing after 3.6.0 release > Requests for container allocation are not resubmitted > - > > Key: APEXCORE-471 > URL: https://issues.apache.org/jira/browse/APEXCORE-471 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.3.0, 3.4.0 >Reporter: Vlad Rozov >Assignee: Sandesh > Fix For: 3.6.0 > > > When Yarn cluster has a limited number of available resources, requests > should be resubmitted. BlacklistBasedResourceRequestHandler does not properly > handle case when resources are limited. > Apex has 2 schedulers, BlacklistedResource* is a default option on CDH. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-691) Use type inference for generic instance creation
[
https://issues.apache.org/jira/browse/APEXCORE-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-691.
--
Closing after 3.6.0 release
> Use type inference for generic instance creation
>
>
> Key: APEXCORE-691
> URL: https://issues.apache.org/jira/browse/APEXCORE-691
> Project: Apache Apex Core
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
> Fix For: 3.6.0
>
>
> The current Java language level allows using type inference for generic
> instance creation making constructs like
> {noformat}
> Map map = new HashMap();
> {noformat}
> unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-634) Unifier attributes are not set for modules in DAG
[ https://issues.apache.org/jira/browse/APEXCORE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-634. -- Closing after 3.6.0 release > Unifier attributes are not set for modules in DAG > - > > Key: APEXCORE-634 > URL: https://issues.apache.org/jira/browse/APEXCORE-634 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Deepak Narkhede >Assignee: Deepak Narkhede >Priority: Minor > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-677) Avoid starting StramLocalCluster in StreamingContainerManagerTest.testAppDataSources
[ https://issues.apache.org/jira/browse/APEXCORE-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-677. -- Closing after 3.6.0 release > Avoid starting StramLocalCluster in > StreamingContainerManagerTest.testAppDataSources > > > Key: APEXCORE-677 > URL: https://issues.apache.org/jira/browse/APEXCORE-677 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Trivial > Fix For: 3.6.0 > > > testAppDataSources does not wait for the StramLocalCluster to be initialized, > so it does not need to start it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-611) Stram Event Log Levels
[ https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-611. -- Closing after 3.6.0 release > Stram Event Log Levels > -- > > Key: APEXCORE-611 > URL: https://issues.apache.org/jira/browse/APEXCORE-611 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.6.0 > > > Provide log levels for Stram events. Such as INFO, WARN, ERROR > Eg: > 1. Start Container, Start Operator are INFO level events > 2. OperatorError is ERROR level event > 3. Stop Container, Stop Operator are WARN level events > Log level for events can help in user experience when showing the event list > in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log > levels so that user can focus more on ERROR and WARN events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-580) Interface for processing and emitting control tuples
[ https://issues.apache.org/jira/browse/APEXCORE-580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-580. -- Closing after 3.6.0 release > Interface for processing and emitting control tuples > > > Key: APEXCORE-580 > URL: https://issues.apache.org/jira/browse/APEXCORE-580 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > DefaultOutputPort needs to have a emitControl method so that operator code > can call to emit a control tuple. > DefaultInputPort needs to have a processControl method so that operator would > be able to act on the arrival of a control tuple. > Similar to a regular data tuple, we also need to provide a way for the user > to provide custom serialization for the control tuple. > We need to design this so that the default behavior is to propagate control > tuples to all output ports, and it should allow the user to easily change > that behavior. The user can selectively propagate control tuples to certain > output ports, or block the propagation altogether. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-692) Apex Dev Setup doc should reference download page
[ https://issues.apache.org/jira/browse/APEXCORE-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-692. -- Closing after 3.6.0 release > Apex Dev Setup doc should reference download page > - > > Key: APEXCORE-692 > URL: https://issues.apache.org/jira/browse/APEXCORE-692 > Project: Apache Apex Core > Issue Type: Documentation > Components: Documentation >Reporter: Dean Lockgaard >Assignee: Dean Lockgaard >Priority: Minor > Fix For: 3.6.0 > > > In the Sandbox section of the Apache Apex Development Environment Setup > documentation > (https://apex.apache.org/docs/apex/apex_development_setup/#sandbox), > instructions are provided for a vendor-specific Sandbox. > These instructions should be changed to reference the Apache Apex Sandbox > instead (https://hub.docker.com/r/apacheapex/sandbox). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-504) Possible race condition in StreamingContainerAgent.getStreamCodec()
[
https://issues.apache.org/jira/browse/APEXCORE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-504.
--
Closing after 3.6.0 release
> Possible race condition in StreamingContainerAgent.getStreamCodec()
> ---
>
> Key: APEXCORE-504
> URL: https://issues.apache.org/jira/browse/APEXCORE-504
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.2.1, 3.4.0, 3.5.0
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
> Fix For: 3.3.1, 3.2.2, 3.5.1, 3.6.0, 3.4.1
>
>
> StreamingContainerAgent.getStreamCodec() may be called from multiple RPC
> heartbeat processing threads for the same inputPortMeta (for an input port of
> a partitioned operator) that leads to a race condition in the following code:
> {noformat}
> StreamCodec codec = inputPortMeta.getValue(PortContext.STREAM_CODEC);
> if (codec == null) {
> // it cannot be this object that gets returned. Depending on this
> value is dangerous
> codec = inputPortMeta.getPortObject().getStreamCodec();
> if (codec != null) {
> // don't create codec multiple times - it will assign a new
> identifier
> inputPortMeta.getAttributes().put(PortContext.STREAM_CODEC, codec);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-686) AppPackage fails when .apa contains zero length stored entry
[
https://issues.apache.org/jira/browse/APEXCORE-686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi closed APEXCORE-686.
--
Closing after 3.6.0 release
> AppPackage fails when .apa contains zero length stored entry
>
>
> Key: APEXCORE-686
> URL: https://issues.apache.org/jira/browse/APEXCORE-686
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
> Fix For: 3.6.0
>
>
> JarInputStream/ZipInputStream does not support valid formats of .jar/.zip file
> {noformat}
> java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
> at java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:309)
> at java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:121)
> at java.util.jar.JarInputStream.(JarInputStream.java:83)
> at java.util.jar.JarInputStream.(JarInputStream.java:62)
> at com.datatorrent.stram.client.AppPackage.(AppPackage.java:215)
> at com.datatorrent.stram.client.AppPackage.(AppPackage.java:196)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (APEXCORE-595) Master incorrectly updates committedWindowId when all partitions are terminated.
[ https://issues.apache.org/jira/browse/APEXCORE-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-595. -- Closing after 3.6.0 release > Master incorrectly updates committedWindowId when all partitions are > terminated. > > > Key: APEXCORE-595 > URL: https://issues.apache.org/jira/browse/APEXCORE-595 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi > Fix For: 3.6.0 > > > When all partitions are terminated application master incorrectly updates the > committedWindowId to Long.MAX_VALUE and send purge requet to StramChild > causing exception in StramChild. > The problematic code is in StreamingContainerManager.updateCheckpoints. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-581. -- Closing after 3.6.0 release > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-709) Refactor changes done through APEXCORE-575
[ https://issues.apache.org/jira/browse/APEXCORE-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-709. -- Closing after 3.6.0 release > Refactor changes done through APEXCORE-575 > -- > > Key: APEXCORE-709 > URL: https://issues.apache.org/jira/browse/APEXCORE-709 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi >Priority: Minor > Fix For: 3.6.0 > > > Handle additional review comments on pull request 422. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-649) Infrastructure for user define stram event listeners.
[ https://issues.apache.org/jira/browse/APEXCORE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-649. -- Closing after 3.6.0 release > Infrastructure for user define stram event listeners. > - > > Key: APEXCORE-649 > URL: https://issues.apache.org/jira/browse/APEXCORE-649 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi > Fix For: 3.6.0 > > > As suggested while working on Visitor API, I have came up with following > proposal. The idea is to support user defined DAG listeners. The plan is to > support limitated set of events for now and we could add more events > in future. > For the details functionality propvided check attached document. > Please provide feedback on provided proposal. > https://docs.google.com/document/d/1SAIE0EjnCumrB1jKJSnbGvcml47Po8ZHFthfcbNJQgU/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-522) Promote singleton usage pattern for String2String, Long2String and other StringCodecs
[ https://issues.apache.org/jira/browse/APEXCORE-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-522. -- Closing after 3.6.0 release > Promote singleton usage pattern for String2String, Long2String and other > StringCodecs > - > > Key: APEXCORE-522 > URL: https://issues.apache.org/jira/browse/APEXCORE-522 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > String2String codec does not hold state, a single instance of the same > String2String class may be reused. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-480) Change the container log name from dt.log to apex.log
[ https://issues.apache.org/jira/browse/APEXCORE-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-480. -- Closing after 3.6.0 release > Change the container log name from dt.log to apex.log > - > > Key: APEXCORE-480 > URL: https://issues.apache.org/jira/browse/APEXCORE-480 > Project: Apache Apex Core > Issue Type: Task >Reporter: Siyuan Hua >Assignee: Thomas Weise >Priority: Minor > Fix For: 3.6.0 > > > The log file name should be changed to avoid growing user dependency on > legacy naming. New users should not be exposed to this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-658) Replace dt. prefix with apex. for configuration keys
[ https://issues.apache.org/jira/browse/APEXCORE-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-658. -- Closing after 3.6.0 release > Replace dt. prefix with apex. for configuration keys > -- > > Key: APEXCORE-658 > URL: https://issues.apache.org/jira/browse/APEXCORE-658 > Project: Apache Apex Core > Issue Type: Task >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 3.6.0 > > > Users should be able to configure their applications with keys that use apex. > as prefix and this is how it should be shown in examples also. It should be > possible to make this change w/o breaking backward compatibility. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-648) Unnecessary byte array copy in DefaultStatefulStreamCodec.toDataStatePair()
[ https://issues.apache.org/jira/browse/APEXCORE-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi closed APEXCORE-648. -- Closing after 3.6.0 release > Unnecessary byte array copy in DefaultStatefulStreamCodec.toDataStatePair() > --- > > Key: APEXCORE-648 > URL: https://issues.apache.org/jira/browse/APEXCORE-648 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > DefaultStatefulStreamCodec.toDataStatePair() calls Kryo Output.toBytes() that > creates new byte[] and copies serialized data to the newly allocated array. > It is not necessary as the output of toDataStatePair() will be copied again > in PayloadTuple.getSerializedTuple(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-715) Remove unnecessary @Evolving annotation in engine
[ https://issues.apache.org/jira/browse/APEXCORE-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-715: --- Fix Version/s: 3.6.0 > Remove unnecessary @Evolving annotation in engine > - > > Key: APEXCORE-715 > URL: https://issues.apache.org/jira/browse/APEXCORE-715 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > Fix For: 3.6.0 > > > \@Evolving annotation in the engine are not necessary and introduce > unnecessarily practice and wrong impression that some interfaces and/or > classes are more stable compared to others. All interfaces and classes in the > engine are not subject to the semantic version checks and may change from a > release to a release. The same applies to the buffer server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-594) Plugin support in Apex
[ https://issues.apache.org/jira/browse/APEXCORE-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXCORE-594. Resolution: Fixed Fix Version/s: 3.6.0 > Plugin support in Apex > -- > > Key: APEXCORE-594 > URL: https://issues.apache.org/jira/browse/APEXCORE-594 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Tushar Gosavi > Fix For: 3.6.0 > > > Investigate adding new functionality to apex through user plugins. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-700) Make the plugin registration interface uniform
[ https://issues.apache.org/jira/browse/APEXCORE-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXCORE-700. Resolution: Fixed > Make the plugin registration interface uniform > -- > > Key: APEXCORE-700 > URL: https://issues.apache.org/jira/browse/APEXCORE-700 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni >Priority: Minor > Fix For: 3.6.0 > > > The user-facing plugin registration for DAG setup plugins is slightly > different from runtime plugins. It would be better to have a uniform way to > do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-563) Have a pointer to log file name and offset in container/operator failure events
[ https://issues.apache.org/jira/browse/APEXCORE-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-563: --- Summary: Have a pointer to log file name and offset in container/operator failure events (was: Have a pointer to container log file name and offset in stram events that deliver a container or operator failure event.) > Have a pointer to log file name and offset in container/operator failure > events > --- > > Key: APEXCORE-563 > URL: https://issues.apache.org/jira/browse/APEXCORE-563 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Sanjay M Pujare >Assignee: Priyanka Gugale > Fix For: 3.6.0 > > > The default DailyRollingFileAppender does not take into account of how many > backup files to keep and it will result in unbounded growth of log files, > especially for long running applications. > The below is an interesting add-on to the default DailyRollingFileAppender > that supports maxBackupIndex. > http://wiki.apache.org/logging-log4j/DailyRollingFileAppender -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-636) Ability to refresh tokens using user's own kerberos credentials in a managed environment
[ https://issues.apache.org/jira/browse/APEXCORE-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-636: --- Summary: Ability to refresh tokens using user's own kerberos credentials in a managed environment (was: Ability to refresh tokens using user's own kerberos credentials in a managed environment where the application is launched using an admin with impersonation) > Ability to refresh tokens using user's own kerberos credentials in a managed > environment > > > Key: APEXCORE-636 > URL: https://issues.apache.org/jira/browse/APEXCORE-636 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Pramod Immaneni >Assignee: devendra tagare > Fix For: 3.6.0 > > > When applications run in secure mode, they use delegation tokens to access > Hadoop resources. These delegation tokens have a lifetime, typically 7 days, > after which they no longer work and the application will not be able to > communicate with Hadoop. Apex can automatically refresh these tokens before > they expire. To do this it requires Kerberos credentials which should be > supplied during launch time. > In a managed environment the user launching the application may not be > intended runtime user for the application. Apex today supports impersonation > to achieve this. Typically, a management application uses its own > credentials, which typically have higher privilege, to launch the application > and impersonate as a regular user so that the application runs as the regular > user. However, the admin credentials are also packaged with the application > to for refreshing the tokens described above. This can cause a security > concern because a regular user has access to a higher privilege Kerberos > credentials. > We need a way to specify alternate kerberos credentials to be used for token > refresh. Today there is a partially implemented feature for this which allows > specification of the refresh keytab using a property but not the principal. > We would need to add support for the principal as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-656) Upgrade org.apache.httpcomponents.httpclient to 4.3.6 version
[ https://issues.apache.org/jira/browse/APEXCORE-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-656: --- Summary: Upgrade org.apache.httpcomponents.httpclient to 4.3.6 version (was: Upgrade org.apache.httpcomponents.httpclient) > Upgrade org.apache.httpcomponents.httpclient to 4.3.6 version > -- > > Key: APEXCORE-656 > URL: https://issues.apache.org/jira/browse/APEXCORE-656 > Project: Apache Apex Core > Issue Type: Dependency upgrade >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Trivial > Fix For: 3.6.0 > > > Upgrade org.apache.httpcomponents.httpclient dependency to the latest 4.3.x > version. Please see > [https://dist.apache.org/repos/dist/release/httpcomponents/httpclient/RELEASE_NOTES-4.3.x.txt]. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-634) Unifier attributes are not set for modules in DAG
[ https://issues.apache.org/jira/browse/APEXCORE-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-634: --- Summary: Unifier attributes are not set for modules in DAG (was: Apex Platform unable to set unifier attributes for modules in DAG) > Unifier attributes are not set for modules in DAG > - > > Key: APEXCORE-634 > URL: https://issues.apache.org/jira/browse/APEXCORE-634 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Deepak Narkhede >Assignee: Deepak Narkhede >Priority: Minor > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-594) Plugin support in Apex
[ https://issues.apache.org/jira/browse/APEXCORE-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-594: --- Summary: Plugin support in Apex (was: Investigate adding new functionality to apex through user plugins.) > Plugin support in Apex > -- > > Key: APEXCORE-594 > URL: https://issues.apache.org/jira/browse/APEXCORE-594 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Tushar Gosavi > > Investigate adding new functionality to apex through user plugins. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-710) Plugin manager does not report exceptions from plugins
[ https://issues.apache.org/jira/browse/APEXCORE-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-710: --- Issue Type: Bug (was: Sub-task) Parent: (was: APEXCORE-594) > Plugin manager does not report exceptions from plugins > -- > > Key: APEXCORE-710 > URL: https://issues.apache.org/jira/browse/APEXCORE-710 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Pramod Immaneni > > If there are exceptions during execution of a plugin they should be reported. > The plugin should possibly be taken down as it might be in a bad state. An > event may need to be raised in stram to notify users. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-577) Plugin support to inspect DAG before launch.
[
https://issues.apache.org/jira/browse/APEXCORE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi updated APEXCORE-577:
---
Summary: Plugin support to inspect DAG before launch. (was: Visitor API
for DAG)
> Plugin support to inspect DAG before launch.
>
>
> Key: APEXCORE-577
> URL: https://issues.apache.org/jira/browse/APEXCORE-577
> Project: Apache Apex Core
> Issue Type: Sub-task
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
> Fix For: 3.6.0
>
>
> Support an Visitor like api for DAG, which could be used to write plugins for
> Apex.
> change DAG interface to add a visitor
> {code}
> registerVisitor(DagVisitor visitor)
> {code}
> The DagVisitor interface
> {code}
> interface DagVisitor {
> boolean startDAG(DAG* dag);
> boolean visitOperator(OperatorMeta *ometa)
> boolean visitStream(StreamMeta *smeta);
> boolean endDAG()
> }
> {code}
> dev thread discussions
> https://lists.apache.org/thread.html/1c37e0954cee029c9de537cc35ecb35beebdf49aba17de89bd2ce9ed@%3Cdev.apex.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (APEXCORE-577) Plugin support to inspect DAG before launch.
[
https://issues.apache.org/jira/browse/APEXCORE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi resolved APEXCORE-577.
Resolution: Fixed
Fix Version/s: 3.6.0
> Plugin support to inspect DAG before launch.
>
>
> Key: APEXCORE-577
> URL: https://issues.apache.org/jira/browse/APEXCORE-577
> Project: Apache Apex Core
> Issue Type: Sub-task
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
> Fix For: 3.6.0
>
>
> Support an Visitor like api for DAG, which could be used to write plugins for
> Apex.
> change DAG interface to add a visitor
> {code}
> registerVisitor(DagVisitor visitor)
> {code}
> The DagVisitor interface
> {code}
> interface DagVisitor {
> boolean startDAG(DAG* dag);
> boolean visitOperator(OperatorMeta *ometa)
> boolean visitStream(StreamMeta *smeta);
> boolean endDAG()
> }
> {code}
> dev thread discussions
> https://lists.apache.org/thread.html/1c37e0954cee029c9de537cc35ecb35beebdf49aba17de89bd2ce9ed@%3Cdev.apex.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (APEXCORE-699) Investigate versioning for plugins
[ https://issues.apache.org/jira/browse/APEXCORE-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXCORE-699. Resolution: Won't Fix The solution suggested by [~vrozov] looks good. We have provided an API to plugin to get the version of Apex engine. So that plugin can decide to no register for any event if it founds an incompatible version. > Investigate versioning for plugins > -- > > Key: APEXCORE-699 > URL: https://issues.apache.org/jira/browse/APEXCORE-699 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Pramod Immaneni >Assignee: Tushar Gosavi >Priority: Minor > > Having versioning information in the plugin would help in dealing with > compatibility with older plugins when the plugin interface changes. This > needs to be investigated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-660) Documentation for Control tuple support changes
[ https://issues.apache.org/jira/browse/APEXCORE-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-660: --- Component/s: Documentation > Documentation for Control tuple support changes > --- > > Key: APEXCORE-660 > URL: https://issues.apache.org/jira/browse/APEXCORE-660 > Project: Apache Apex Core > Issue Type: Sub-task > Components: Documentation >Reporter: Bhupesh Chawda >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-709) Refactor changes done through APEXCORE-575
Tushar Gosavi created APEXCORE-709: -- Summary: Refactor changes done through APEXCORE-575 Key: APEXCORE-709 URL: https://issues.apache.org/jira/browse/APEXCORE-709 Project: Apache Apex Core Issue Type: Bug Reporter: Tushar Gosavi Assignee: Tushar Gosavi Priority: Minor Handle additional review comments on pull request 422. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-699) Investigate versioning for plugins
[
https://issues.apache.org/jira/browse/APEXCORE-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969449#comment-15969449
]
Tushar Gosavi commented on APEXCORE-699:
I did following experiments
- Added a static final long variable and added a getVersion() method in the
plugin interface. when it is compiled
the compiler optimized out the const value and replace it by the actual
value.
{code}
class A implement Plugin {
int getVersion() {
return Plugin.VERSION;
}
@Override
void sayHello(String name) { }
}
interface Plugin {
public static final long VERSION = 1;
void sayHello(String name);
}
{code}
If we change version in new Apex as below
{code}
interface Plugin {
public static final long VERSION = 2;
void sayHello(String name);
void newMethod();
}
{code}
But plugin compiled against old version is used then getVersion return version
of old Apex (1). This optimization works
only for the primitive types and require a getVersion() method in the
interface, or a version field in interface. We could
introduce a BasePlugin which will have a field and/or this method and suggest
to use extend plugin from BasePlugin class,
rather than implementing the interface directly.
There is not way to check for absence of method except to call it and catch
NoSuchMethod exception.
> Investigate versioning for plugins
> --
>
> Key: APEXCORE-699
> URL: https://issues.apache.org/jira/browse/APEXCORE-699
> Project: Apache Apex Core
> Issue Type: Sub-task
>Reporter: Pramod Immaneni
>Assignee: Tushar Gosavi
>Priority: Minor
>
> Having versioning information in the plugin would help in dealing with
> compatibility with older plugins when the plugin interface changes. This
> needs to be investigated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXCORE-702) Mark ApexPlugin as Evolving
Tushar Gosavi created APEXCORE-702: -- Summary: Mark ApexPlugin as Evolving Key: APEXCORE-702 URL: https://issues.apache.org/jira/browse/APEXCORE-702 Project: Apache Apex Core Issue Type: Bug Reporter: Tushar Gosavi Assignee: Tushar Gosavi Make ApexPlugin and related classes as Evolving and this part of code is new and might change in future. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application
[ https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968777#comment-15968777 ] Tushar Gosavi commented on APEXCORE-408: I have updated the document, please go through it. The approach I am taking is summarized below - extend dag interface with following methods - removeOperator/removeStream - add startTransaction method on LogicalPlan which will return a DAG object, which is a instance of TransactionalDAG which is clone of the dag with reference to operator and ports from the original dag. TransactionDAG maintains the reference to original dag and has a additional commit method. On commit if the new DAG is valid then the same changes are applied to the original DAG. - user can change the cloned dag as per his requirement. Currently I am providing the hooks to change the DAG through StatsListener by extending StatsListener interface having additional context object. later the hooks could also be provided through DAGExecutionPluginContext which has access to more information from stram than statslistener. - A Commit handler interface is added which commit method is call to notify type of changes performed to the DAG. The PhysicalPlan will be changed through this handler. > Ability to schedule Sub-DAG from running application > > > Key: APEXCORE-408 > URL: https://issues.apache.org/jira/browse/APEXCORE-408 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Thomas Weise >Assignee: Tushar Gosavi > > Today it is possible to add operators to a running application from the > CLI/client. It should be possible to do this from within the application > also. This can be used to expand/remove a Sub-DAG on demand, triggered by > application specific logic. It will enable use cases such as batch > applications that perform multiple stages of processing where not all > resources are required at the same time or the logic of subsequent steps > depends on the execution of previous steps. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-678) Shutdown of application should start from input nodes
[
https://issues.apache.org/jira/browse/APEXCORE-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tushar Gosavi updated APEXCORE-678:
---
Fix Version/s: 3.6.0
> Shutdown of application should start from input nodes
> -
>
> Key: APEXCORE-678
> URL: https://issues.apache.org/jira/browse/APEXCORE-678
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Bhupesh Chawda
>Assignee: Bhupesh Chawda
> Fix For: 3.6.0
>
>
> Streaming container calls shutdown() for all nodes instead of just input
> nodes.
> {code}
> private void stopInputNodes()
> {
> for (Entry e : nodes.entrySet()) {
> Node node = e.getValue();
> if (node instanceof InputNode) {
> final Thread thread = e.getValue().context.getThread();
> if (thread == null || !thread.isAlive()) {
> continue;
> }
> }
> node.shutdown(true);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (APEXCORE-408) Ability to schedule Sub-DAG from running application
[ https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958879#comment-15958879 ] Tushar Gosavi commented on APEXCORE-408: [~thw] I have opened a pull request https://github.com/apache/apex-core/pull/476 for statslistener changes. I am implementing the second approach from the document as it would be simple and won't change existing code much. There are few problems I am facing regarding the validation done by LogicalPlan while elements are being added to it, like extending a stream the port needs to be present in the DAG. When I try to extend an existing stream with new operator's input port , as new operator's input port is not in the original DAG , it throws as error as port does not belong to any operator in the DAG. Trying to solve this problem. > Ability to schedule Sub-DAG from running application > > > Key: APEXCORE-408 > URL: https://issues.apache.org/jira/browse/APEXCORE-408 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Thomas Weise >Assignee: Tushar Gosavi > > Today it is possible to add operators to a running application from the > CLI/client. It should be possible to do this from within the application > also. This can be used to expand/remove a Sub-DAG on demand, triggered by > application specific logic. It will enable use cases such as batch > applications that perform multiple stages of processing where not all > resources are required at the same time or the logic of subsequent steps > depends on the execution of previous steps. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-649) Infrastructure for user define stram event listeners.
[
https://issues.apache.org/jira/browse/APEXCORE-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938145#comment-15938145
]
Tushar Gosavi commented on APEXCORE-649:
# Example use case
- One common use case is pushing application metrics to third party monitoring
systems such as Graphite, OpenTSDB, etc ..., see the `Example Plugin` below how
this can be achieved.
- Taking decisions based on stats and events, such as kill the application in
case of container failure, which can be done by monitoring an container killed
event and then killing the application.
# Writing an Plugin
User can define a Apex Plugin by extending from DAGExecutionPlugin. The
important methods in the class as as below
- **setup(DAGExecutionPluginContext)**
In the setup user can register to the interested events the following events
are supported.
- *ContainerHeartbeat* - The heartbeat from StreamingContainer as passed to
the plugin for examination after it has been handled by the application master.
{code:language=java}
context.register(HEARTBEAT, new Handler<>(ContainerHeartbeat chb) {
...
});
{code}
- *StramEvent* - All Stram events generated by platform can be monitored
the platform.
{code:language=java}
context.register(STRAM_EVENT, new Handler<>(StreamEvent event) {
...
});
{code}
- *Committed* - When committed windowId is changed the plugin is notified
so thatplugin can cleanup cached data if required.
{code:language=java}
context.register(COMMIT_EVENT, new Handler<>(Long wid) {
...
});
{code}
- **teardown()**
clean additional resources created by the plugin.
The all the handlers should be thread-safe as there is no guarantee that plugin
execution environment provides for thread safety. Also the plugins should not
block for long time as it could prevent other plugins from
executing and may result in dropped events in case of full queue.
# Loading of plugin
The plugins are loaded by platform in application master. currently the plugins
are searched by two methods.
1. Through JavaServiceLoader functionality.
In this case user can create a jar of his plugin with
META-INF/services/org.apache.apex.engine.api.DAGExecutionPlugin file in
resource directory. This file should contain the fully classified name
of the class of the plugin. (See
https://docs.oracle.com/javase/tutorial/ext/basics/spi.html)
2. Providing class names of Plugins through application configuration file.
Alternatively user can provide fully qualified name of the class
implementing plugin in application configuration file as given below.
{code:language=java}
apex.plugin.stram.plugins
{fcn of plugin}
{code}
## Example Plugin.
A sample plugin to push the container free memory metric to Grapite monitoring
system is given below.
{code:language=java}
package com.tugo.apex.plugins;
import java.io.IOException;
import java.io.OutputStream;
import java.net.Socket;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import org.slf4j.Logger;
import org.apache.apex.engine.api.DAGExecutionPlugin;
import org.apache.apex.engine.api.DAGExecutionPluginContext;
import com.datatorrent.stram.api.StreamingContainerUmbilicalProtocol;
import static org.slf4j.LoggerFactory.getLogger;
public class GraphitePushPlugin implements DAGExecutionPlugin
{
private static final Logger LOG = getLogger(GraphitePushPlugin.class);
private DAGExecutionPluginContext context;
ScheduledExecutorService executorService;
private String appName;
private String host;
private int port;
private Socket socket;
private OutputStream output;
private boolean connected = false;
@Override
public void setup(DAGExecutionPluginContext context)
{
executorService = Executors.newSingleThreadScheduledExecutor();
context.register(DAGExecutionPluginContext.HEARTBEAT, new
DAGExecutionPluginContext.Handler()
{
@Override
public void handle(StreamingContainerUmbilicalProtocol.ContainerHeartbeat
heartbeat)
{
handleHeartbeat(heartbeat);
}
});
appName = context.getApplicationContext().getApplicationName();
host = context.getLaunchConfig().get("graphite-host");
port = Integer.parseInt(context.getLaunchConfig().get("graphite-port"));
}
synchronized void connect() throws IOException
{
if (!connected) {
socket = new Socket(host, port);
output = socket.getOutputStream();
connected = true;
}
}
private void
handleHeartbeat(StreamingContainerUmbilicalProtocol.ContainerHeartbeat
heartbeat)
{
StringBuilder builder = new StringBuilder(1024);
builder.append(appName).append(".").append(heartbeat.getContainerId()).append(".").append("freeMemory").append("
")
.append(heartbeat.memoryMBFree).append("
[jira] [Commented] (APEXCORE-663) Application restart not working.
[ https://issues.apache.org/jira/browse/APEXCORE-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900760#comment-15900760 ] Tushar Gosavi commented on APEXCORE-663: I believe this was introduced through 3b660c9c1dd639c49ae300fd3f70397d3f794d12. [~sandesh] can you take a look at it. > Application restart not working. > > > Key: APEXCORE-663 > URL: https://issues.apache.org/jira/browse/APEXCORE-663 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Priority: Critical > > Application restart is not working. Steps to reproduce > - launch PiDemo > - Wait for some time (till recovery windowId reaches 120 >) > - kill the app > - Try to relaunch using 'launch -originalAppId' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-663) Application restart not working.
Tushar Gosavi created APEXCORE-663: -- Summary: Application restart not working. Key: APEXCORE-663 URL: https://issues.apache.org/jira/browse/APEXCORE-663 Project: Apache Apex Core Issue Type: Bug Reporter: Tushar Gosavi Priority: Critical Application restart is not working. Steps to reproduce - launch PiDemo - Wait for some time (till recovery windowId reaches 120 >) - kill the app - Try to relaunch using 'launch -originalAppId' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-662) Raise StramEvent for heartbeat miss
Tushar Gosavi created APEXCORE-662: -- Summary: Raise StramEvent for heartbeat miss Key: APEXCORE-662 URL: https://issues.apache.org/jira/browse/APEXCORE-662 Project: Apache Apex Core Issue Type: Task Reporter: Tushar Gosavi Priority: Minor One of the cause of container kill is heartbeat miss. we can raise an StramEvent in case of heartbeat miss to make debugging easier. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-661) Raise StramEvent for blocked operator
Tushar Gosavi created APEXCORE-661: -- Summary: Raise StramEvent for blocked operator Key: APEXCORE-661 URL: https://issues.apache.org/jira/browse/APEXCORE-661 Project: Apache Apex Core Issue Type: Task Reporter: Tushar Gosavi Most common cause of container killed is blocked operator. The stramevent generated for container kill event does not have any reason for termination. instead we could generate an StramEvent when operator is detected as blocked. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-408) Ability to schedule Sub-DAG from running application
[ https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-408: --- Attachment: (was: dag.docx) > Ability to schedule Sub-DAG from running application > > > Key: APEXCORE-408 > URL: https://issues.apache.org/jira/browse/APEXCORE-408 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Thomas Weise >Assignee: Tushar Gosavi > > Today it is possible to add operators to a running application from the > CLI/client. It should be possible to do this from within the application > also. This can be used to expand/remove a Sub-DAG on demand, triggered by > application specific logic. It will enable use cases such as batch > applications that perform multiple stages of processing where not all > resources are required at the same time or the logic of subsequent steps > depends on the execution of previous steps. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-408) Ability to schedule Sub-DAG from running application
[ https://issues.apache.org/jira/browse/APEXCORE-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi updated APEXCORE-408: --- Attachment: (was: dag.docx) > Ability to schedule Sub-DAG from running application > > > Key: APEXCORE-408 > URL: https://issues.apache.org/jira/browse/APEXCORE-408 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Thomas Weise >Assignee: Tushar Gosavi > > Today it is possible to add operators to a running application from the > CLI/client. It should be possible to do this from within the application > also. This can be used to expand/remove a Sub-DAG on demand, triggered by > application specific logic. It will enable use cases such as batch > applications that perform multiple stages of processing where not all > resources are required at the same time or the logic of subsequent steps > depends on the execution of previous steps. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-577) Visitor API for DAG
[
https://issues.apache.org/jira/browse/APEXCORE-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897181#comment-15897181
]
Tushar Gosavi commented on APEXCORE-577:
Faced some issues while implementing suggested approach. The first entry point
to the user code
is populateDAG in the StreamingApplication. The plugins needs to be loaded
before prePopulateDAG
is called. We could go with ServiceLoader approach to load the plugin but it
has some issues, for example we can not have configuration for the plugin, as
ServiceLoader needs a default constructor to create an instance. And we do not
know from where to apply the plugin specific configuration before
calling plugins methods.
The preSetProperties/postSetProperty is tied up to the internal implementation
of DAG launching mechanism. We could have different implementation (python or
scala shell) which creates a DAG and submit for execution using launcher api
without setting configuration from xml files. for example a beam app launched
using maven cli, won't have preSetProperties/postSetProperty callbacks.
We have three implementation of preparing DAG as of now, all three needs to
call plugin specific functions in correct order before launch of DAG. Any new
mechanism to create DAG needs to remember to call plugin specific functions in
correct order.
I have my WIP changes in :
https://github.com/tushargosavi/apex-core/compare/master...tushargosavi:APEXCORE-577.visitor.new
Let me know your thoughts on this.
> Visitor API for DAG
> ---
>
> Key: APEXCORE-577
> URL: https://issues.apache.org/jira/browse/APEXCORE-577
> Project: Apache Apex Core
> Issue Type: Sub-task
>Reporter: Tushar Gosavi
>Assignee: Tushar Gosavi
>
> Support an Visitor like api for DAG, which could be used to write plugins for
> Apex.
> change DAG interface to add a visitor
> {code}
> registerVisitor(DagVisitor visitor)
> {code}
> The DagVisitor interface
> {code}
> interface DagVisitor {
> boolean startDAG(DAG* dag);
> boolean visitOperator(OperatorMeta *ometa)
> boolean visitStream(StreamMeta *smeta);
> boolean endDAG()
> }
> {code}
> dev thread discussions
> https://lists.apache.org/thread.html/1c37e0954cee029c9de537cc35ecb35beebdf49aba17de89bd2ce9ed@%3Cdev.apex.apache.org%3E
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (APEXCORE-579) Custom control tuple support
[ https://issues.apache.org/jira/browse/APEXCORE-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXCORE-579. Resolution: Fixed Fix Version/s: 3.6.0 > Custom control tuple support > > > Key: APEXCORE-579 > URL: https://issues.apache.org/jira/browse/APEXCORE-579 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > The discussion thread is here: > https://lists.apache.org/thread.html/9accc404a3cb17af2de73b860ed7dff4523a837b4d3c1c73fe0858c5@%3Cdev.apex.apache.org%3E > We will implement option 1 for now and park option 2 until we have a real use > case. > We probably need a LinkedHashMap to store all the control tuples received in > memory (to preserve the order and uniqueness). > The assumption is that we will not have a large number of control tuples > within a streaming window and the memory footprint of control tuples is > small. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi reassigned APEXCORE-581: -- Assignee: Bhupesh Chawda > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-581) Delivery of Custom Control Tuples
[ https://issues.apache.org/jira/browse/APEXCORE-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tushar Gosavi resolved APEXCORE-581. Resolution: Fixed Fix Version/s: 3.6.0 > Delivery of Custom Control Tuples > - > > Key: APEXCORE-581 > URL: https://issues.apache.org/jira/browse/APEXCORE-581 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: David Yan >Assignee: Bhupesh Chawda > Fix For: 3.6.0 > > > The behavior should be as follow: > - The control tuples should only be sent to downstream at streaming window > boundaries > - The control tuples should be sent to all partitions downstream > - The control tuples should be sent in the same order of arrival. > - Within a streaming window, do not send the same control tuple twice, even > if the same control tuple is received multiple times within that window. This > is possible if the operator has two input ports. (The LinkedHashMap should be > easily able to ensure both order and uniqueness.) > - The delivery of control tuples needs to stop at DelayOperator. > - When a streaming window is committed, remove the associated LinkedHashMap > that belong to windows with IDs that are less than the committed window > - It's safe to assume the control tuples are rare enough and can fit in memory > This will involve an additional MessageType to represent a custom control > tuple. > We probably need to have a data structure (possibly a LinkedHashMap) per > streaming window that stores the control tuple in the buffer server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
