[jira] [Updated] (APEXMALHAR-2068) FileSplitter- having multiple file splitters in an application results in incorrect behavior
[ https://issues.apache.org/jira/browse/APEXMALHAR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXMALHAR-2068: --- Assignee: (was: Chandni Singh) > FileSplitter- having multiple file splitters in an application results in > incorrect behavior > > > Key: APEXMALHAR-2068 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2068 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Chandni Singh > > If an application has multiple logical operators that use Window Data > Manager, then by default they share the same state. This is confusing and > cause issues. > We need to make this path unique by default for each logical instance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXMALHAR-2068) FileSplitter- having multiple file splitters in an application results in incorrect behavior
[ https://issues.apache.org/jira/browse/APEXMALHAR-2068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXMALHAR-2068: --- Assignee: Chandni Singh > FileSplitter- having multiple file splitters in an application results in > incorrect behavior > > > Key: APEXMALHAR-2068 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2068 > Project: Apache Apex Malhar > Issue Type: Bug >Reporter: Chandni Singh >Assignee: Chandni Singh > > If an application has multiple logical operators that use Window Data > Manager, then by default they share the same state. This is confusing and > cause issues. > We need to make this path unique by default for each logical instance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXMALHAR-2126) Suggest: Share Slice Buffer
[ https://issues.apache.org/jira/browse/APEXMALHAR-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351786#comment-15351786 ] Vlad Rozov commented on APEXMALHAR-2126: The intention of com.datatorrent.netlet.util.Slice was to avoid memory copy when raw data is received into a buffer and then parsed into messages, not memory allocation/deallocation. Memory for data structures/tuples still needs to be allocated and allocating large buffer may lead to excessive memory usage when a large buffer is allocated, but not fully used. > Suggest: Share Slice Buffer > --- > > Key: APEXMALHAR-2126 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2126 > Project: Apache Apex Malhar > Issue Type: Improvement >Reporter: bright chen > > I think the intention of Slice(com.datatorrent.netlet.util.Slice) was to > share the buffer and avoid unnecessary memory allocation/deallocation. But > the intension is not self-explain and lack of method to share the memory. And > the util class org.apache.apex.malhar.lib.utils.serde.SliceUtils also create > new memory and copy the data. > I suggest to implement another class(Say BufferSlice), which > - initialize buffer with relative large buffer > - support append(byte[] data, int offset, int length) > - dynamic reallocated buffer or throw exception when buffer is full ( based > on the management strategy) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXMALHAR-2138) Multiple declaration of org.mockito.mockito-all-1.8.5 in Malhar library pom
[
https://issues.apache.org/jira/browse/APEXMALHAR-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved APEXMALHAR-2138.
Resolution: Fixed
Fix Version/s: 3.5.0
> Multiple declaration of org.mockito.mockito-all-1.8.5 in Malhar library pom
>
>
> Key: APEXMALHAR-2138
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2138
> Project: Apache Apex Malhar
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Vlad Rozov
>Assignee: Harsh Nilesh Pathak
>Priority: Minor
> Fix For: 3.5.0
>
>
> {noformat}
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.mockito:mockito-all:jar -> duplicate declaration of version
> 1.8.5 @ org.apache.apex:malhar-library:[unknown-version],
> /Users/vrozov/Projects/DataTorrent/malhar/library/pom.xml, line 332, column 17
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (APEXCORE-506) Add trailing whitespace check to Apex checkstyle
Vlad Rozov created APEXCORE-506: --- Summary: Add trailing whitespace check to Apex checkstyle Key: APEXCORE-506 URL: https://issues.apache.org/jira/browse/APEXCORE-506 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXMALHAR-2146) Benchmark operators that support Beam window concepts
Vlad Rozov created APEXMALHAR-2146: -- Summary: Benchmark operators that support Beam window concepts Key: APEXMALHAR-2146 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2146 Project: Apache Apex Malhar Issue Type: Task Reporter: Vlad Rozov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXMALHAR-2145) Revisit Accumulation interface
Vlad Rozov created APEXMALHAR-2145: -- Summary: Revisit Accumulation interface Key: APEXMALHAR-2145 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2145 Project: Apache Apex Malhar Issue Type: Wish Reporter: Vlad Rozov Please see discussion regarding accumulation function and accumulator in https://github.com/apache/apex-malhar/pull/319. While accumulation function should not define storage and should provide a generic way to define how tuples are accumulated, accumulator represents actual aggregation and needs to define the most optimal storage for a particular aggregation type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXMALHAR-2138) Multiple declaration of org.mockito.mockito-all-1.8.5 in Malhar library pom
Vlad Rozov created APEXMALHAR-2138: -- Summary: Multiple declaration of org.mockito.mockito-all-1.8.5 in Malhar library pom Key: APEXMALHAR-2138 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2138 Project: Apache Apex Malhar Issue Type: Bug Affects Versions: 3.5.0 Reporter: Vlad Rozov Assignee: Priyanka Gugale Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXMALHAR-2138) Multiple declaration of org.mockito.mockito-all-1.8.5 in Malhar library pom
[
https://issues.apache.org/jira/browse/APEXMALHAR-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXMALHAR-2138:
---
Description:
{noformat}
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
be unique: org.mockito:mockito-all:jar -> duplicate declaration of version
1.8.5 @ org.apache.apex:malhar-library:[unknown-version],
/Users/vrozov/Projects/DataTorrent/malhar/library/pom.xml, line 332, column 17
{noformat}
> Multiple declaration of org.mockito.mockito-all-1.8.5 in Malhar library pom
>
>
> Key: APEXMALHAR-2138
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2138
> Project: Apache Apex Malhar
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Vlad Rozov
>Assignee: Priyanka Gugale
>Priority: Minor
>
> {noformat}
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.mockito:mockito-all:jar -> duplicate declaration of version
> 1.8.5 @ org.apache.apex:malhar-library:[unknown-version],
> /Users/vrozov/Projects/DataTorrent/malhar/library/pom.xml, line 332, column 17
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (APEXMALHAR-1978) Replace ${groupId} with ${project.groupId} in modules and project pom
[
https://issues.apache.org/jira/browse/APEXMALHAR-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved APEXMALHAR-1978.
Resolution: Done
Fix Version/s: 3.4.0
Was done as part of e38e144f2b3d25289c1aef128eb0c24b63362647 commit
> Replace ${groupId} with ${project.groupId} in modules and project pom
> -
>
> Key: APEXMALHAR-1978
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-1978
> Project: Apache Apex Malhar
> Issue Type: Task
>Reporter: Vlad Rozov
>Priority: Minor
> Fix For: 3.4.0
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (APEXCORE-485) Upgrade maven surefire plugin to the latest version
Vlad Rozov created APEXCORE-485: --- Summary: Upgrade maven surefire plugin to the latest version Key: APEXCORE-485 URL: https://issues.apache.org/jira/browse/APEXCORE-485 Project: Apache Apex Core Issue Type: Task Reporter: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (APEXCORE-485) Upgrade maven surefire plugin to the latest version
[ https://issues.apache.org/jira/browse/APEXCORE-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-485: --- Assignee: Vlad Rozov > Upgrade maven surefire plugin to the latest version > --- > > Key: APEXCORE-485 > URL: https://issues.apache.org/jira/browse/APEXCORE-485 > Project: Apache Apex Core > Issue Type: Task >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-486) Facing Method not found issue while launching Kafka application
[ https://issues.apache.org/jira/browse/APEXCORE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-486. - Resolution: Not A Bug Apache Apex requires mbassador-1.1.9. Please check your application for dependency on the mbassador-1.2.x version that is not binary compatible with the prior versions. > Facing Method not found issue while launching Kafka application > --- > > Key: APEXCORE-486 > URL: https://issues.apache.org/jira/browse/APEXCORE-486 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Anuj > > Facing Method not found issue while launching Kafka application. > java.lang.NoSuchMethodError: > net.engio.mbassy.bus.config.BusConfiguration.Default(III)Lnet/engio/mbassy/bus/config/BusConfiguration; > at > com.datatorrent.stram.StreamingContainerManager.(StreamingContainerManager.java:360) > at > com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:2979) > at > com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:550) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:101) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-486) Facing Method not found issue while launching Kafka application
[ https://issues.apache.org/jira/browse/APEXCORE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-486: Labels: (was: newbie) > Facing Method not found issue while launching Kafka application > --- > > Key: APEXCORE-486 > URL: https://issues.apache.org/jira/browse/APEXCORE-486 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Anuj > > Facing Method not found issue while launching Kafka application. > java.lang.NoSuchMethodError: > net.engio.mbassy.bus.config.BusConfiguration.Default(III)Lnet/engio/mbassy/bus/config/BusConfiguration; > at > com.datatorrent.stram.StreamingContainerManager.(StreamingContainerManager.java:360) > at > com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:2979) > at > com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:550) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:101) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-502) Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
Vlad Rozov created APEXCORE-502:
---
Summary: Unnecessary byte array copy in
DefaultKryoStreamCodec.toByteArray
Key: APEXCORE-502
URL: https://issues.apache.org/jira/browse/APEXCORE-502
Project: Apache Apex Core
Issue Type: Improvement
Reporter: Vlad Rozov
{noformat}
slice = new Slice(os.toByteArray(), 0, os.toByteArray().length);
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (APEXMALHAR-2045) Add bandwidth control feature to Apex
[ https://issues.apache.org/jira/browse/APEXMALHAR-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXMALHAR-2045. Resolution: Done Fix Version/s: 3.5.0 > Add bandwidth control feature to Apex > - > > Key: APEXMALHAR-2045 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2045 > Project: Apache Apex Malhar > Issue Type: Task >Reporter: Priyanka Gugale >Assignee: Priyanka Gugale > Fix For: 3.5.0 > > > bandwidth restrictions on input operator for number of bytes to be consumed > per second. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-509) OperatorContext.JVM_OPTIONS should preserve order
[ https://issues.apache.org/jira/browse/APEXCORE-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-509: Labels: newbie (was: ) > OperatorContext.JVM_OPTIONS should preserve order > - > > Key: APEXCORE-509 > URL: https://issues.apache.org/jira/browse/APEXCORE-509 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Priority: Minor > Labels: newbie > > While majority of JVM options don't require any particular order, few -XX > options are order sensitive. For example +UnlockCommercialFeatures must > precede +FlightRecorder or JVM will fail to start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-448) Make operator name available in OperatorContext
[ https://issues.apache.org/jira/browse/APEXCORE-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-448. - Resolution: Fixed Fix Version/s: 3.5.0 > Make operator name available in OperatorContext > --- > > Key: APEXCORE-448 > URL: https://issues.apache.org/jira/browse/APEXCORE-448 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Chandni Singh >Assignee: Chandni Singh > Fix For: 3.5.0 > > > Need name of the logical operator in the OperatorContext which can be used by > WindowDataManager to create a unique path per logical operator . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (APEXCORE-508) Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS
Vlad Rozov created APEXCORE-508: --- Summary: Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS Key: APEXCORE-508 URL: https://issues.apache.org/jira/browse/APEXCORE-508 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Priority: Minor Provide better documentation on DAGContext.CONTAINER_JVM_OPTIONS and OperatorContext.JVM_OPTIONS. Currently, it is not clear which one to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-508) Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS
[ https://issues.apache.org/jira/browse/APEXCORE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-508: Labels: documentation (was: ) > Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS > - > > Key: APEXCORE-508 > URL: https://issues.apache.org/jira/browse/APEXCORE-508 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov >Priority: Minor > Labels: documentation > > Provide better documentation on DAGContext.CONTAINER_JVM_OPTIONS and > OperatorContext.JVM_OPTIONS. Currently, it is not clear which one to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (APEXCORE-502) Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
[
https://issues.apache.org/jira/browse/APEXCORE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov reassigned APEXCORE-502:
---
Assignee: Vlad Rozov
> Unnecessary byte array copy in DefaultKryoStreamCodec.toByteArray
> -
>
> Key: APEXCORE-502
> URL: https://issues.apache.org/jira/browse/APEXCORE-502
> Project: Apache Apex Core
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>
> {noformat}
> slice = new Slice(os.toByteArray(), 0, os.toByteArray().length);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (APEXCORE-627) Unit test AtMostOnceTest intermittently fails
[ https://issues.apache.org/jira/browse/APEXCORE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-627. - Resolution: Fixed Fix Version/s: 3.6.0 > Unit test AtMostOnceTest intermittently fails > - > > Key: APEXCORE-627 > URL: https://issues.apache.org/jira/browse/APEXCORE-627 > Project: Apache Apex Core > Issue Type: Bug > Environment: The test is reproducible on macOS Sierra, Processor 2.2 > GHz Intel Core i7, Memory 16GB 1600 MHz DDR3. >Reporter: Sergey Golovko >Assignee: Sergey Golovko >Priority: Minor > Fix For: 3.6.0 > > > The test AtMostOnceTest is not able to reach the criteria to stop the test. > And it continue to recover an input operator and rerun the test in a loop. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-504) Possible race condition in StreamingContainerAgent.getStreamCodec()
[
https://issues.apache.org/jira/browse/APEXCORE-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-504:
Affects Version/s: 3.5.0
> Possible race condition in StreamingContainerAgent.getStreamCodec()
> ---
>
> Key: APEXCORE-504
> URL: https://issues.apache.org/jira/browse/APEXCORE-504
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.2.1, 3.4.0, 3.5.0
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>
> StreamingContainerAgent.getStreamCodec() may be called from multiple RPC
> heartbeat processing threads for the same inputPortMeta (for an input port of
> a partitioned operator) that leads to a race condition in the following code:
> {noformat}
> StreamCodec codec = inputPortMeta.getValue(PortContext.STREAM_CODEC);
> if (codec == null) {
> // it cannot be this object that gets returned. Depending on this
> value is dangerous
> codec = inputPortMeta.getPortObject().getStreamCodec();
> if (codec != null) {
> // don't create codec multiple times - it will assign a new
> identifier
> inputPortMeta.getAttributes().put(PortContext.STREAM_CODEC, codec);
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (APEXCORE-592) Returning description field in defaultProperties during apex cli call get-app-package-info
[ https://issues.apache.org/jira/browse/APEXCORE-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856274#comment-15856274 ] Vlad Rozov commented on APEXCORE-592: - I don't think it is necessary to use more complex approach for attribute description. It will not take too long to add a description to attributes. > Returning description field in defaultProperties during apex cli call > get-app-package-info > -- > > Key: APEXCORE-592 > URL: https://issues.apache.org/jira/browse/APEXCORE-592 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Currently for an operator property, only the value field is returned. The > operator property can have a description describing the property. This task > is created to return this filed for that operator property. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-641) Subscribers/DataListeners may not be scheduled to execute even when they have data to process
Vlad Rozov created APEXCORE-641: --- Summary: Subscribers/DataListeners may not be scheduled to execute even when they have data to process Key: APEXCORE-641 URL: https://issues.apache.org/jira/browse/APEXCORE-641 Project: Apache Apex Core Issue Type: Bug Components: Buffer Server Affects Versions: 3.5.0, 3.4.0, 3.2.1, 3.3.0, 3.2.0, 3.6.0 Reporter: Vlad Rozov Assignee: Vlad Rozov Buffer server iterates over DataListeners aka LogicalNodes and each LogicalNode tries to send to it's downstream all data that Publisher added to the DataList. When an output port is connected to multiple partitions or downstream operators (2 or more DataListeners/LogicalNodes) there may be more data published to the DataList after first few DataListeners in the listeners set iterated over DataList and reached the last block published so far. The data published while the last DataListeners sends data to it's downstream will not be processed by other DataListeners until Publisher adds more data to the DataList. This may lead to blocked operators, as Buffer server may stop processing data completely in case Publisher fills more than one Data block while a single DataListener sends data to it's downstream and there are more Subscribers/DataListeners than number of in memory blocks allowed (8). In such case, Publisher will be suspended, and there will be no task scheduled to process data already published to the DataList. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-643: --- Assignee: Vlad Rozov (was: Sandesh) > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-643: Component/s: Buffer Server > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-643: Priority: Minor (was: Major) > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-438) ConcurrentModificationException in LogicalNode
[ https://issues.apache.org/jira/browse/APEXCORE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-438: Priority: Minor (was: Major) > ConcurrentModificationException in LogicalNode > -- > > Key: APEXCORE-438 > URL: https://issues.apache.org/jira/browse/APEXCORE-438 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.1, 3.4.0, 3.3.1 >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > LogicalNode.physicalNodes may be modified or accessed on two different > threads - DefaultEventLoop and ServerHelperExecutor causing > ConcurrentModificationException or other issues caused by thread safety. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-642: Priority: Minor (was: Major) > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-558) Do not use yellow color to display command strings in help output
[ https://issues.apache.org/jira/browse/APEXCORE-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-558: --- Assignee: Sanjay M Pujare > Do not use yellow color to display command strings in help output > - > > Key: APEXCORE-558 > URL: https://issues.apache.org/jira/browse/APEXCORE-558 > Project: Apache Apex Core > Issue Type: Bug > Environment: MacOS, Linux >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.6.0 > > > Apex CLI Help output (at least on MacOS and Linux terminals) shows command > strings in yellow which is extremely hard to read and reduces usability of > the CLI. Should be changed to red or purple or just be bolded in black. > In the following output "alias" and "begin-macro" are in yellow and hard to > read. > apex> help > GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN: > alias alias-name command > Create a command alias > begin-macro name > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-643. - Resolution: Duplicate > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-643) BufferServer - Concurrent Modification Exception during Subscriber Teardown
[ https://issues.apache.org/jira/browse/APEXCORE-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858207#comment-15858207 ] Vlad Rozov commented on APEXCORE-643: - [~sandesh] Please test against https://github.com/apache/apex-core/pull/436 > BufferServer - Concurrent Modification Exception during Subscriber Teardown > --- > > Key: APEXCORE-643 > URL: https://issues.apache.org/jira/browse/APEXCORE-643 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Sandesh >Assignee: Vlad Rozov >Priority: Minor > > Saw the following exception, while running the BufferServer unit test > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$KeyIterator.next(HashMap.java:1461) > at > com.datatorrent.bufferserver.internal.LogicalNode.removeChannel(LogicalNode.java:116) > at com.datatorrent.bufferserver.server.Server$3.run(Server.java:329) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-438) ConcurrentModificationException in LogicalNode
[ https://issues.apache.org/jira/browse/APEXCORE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-438: --- Assignee: Vlad Rozov (was: Sandesh) > ConcurrentModificationException in LogicalNode > -- > > Key: APEXCORE-438 > URL: https://issues.apache.org/jira/browse/APEXCORE-438 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.1, 3.4.0, 3.3.1 >Reporter: Vlad Rozov >Assignee: Vlad Rozov > > LogicalNode.physicalNodes may be modified or accessed on two different > threads - DefaultEventLoop and ServerHelperExecutor causing > ConcurrentModificationException or other issues caused by thread safety. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858243#comment-15858243 ] Vlad Rozov commented on APEXCORE-642: - [~sanjaypujare] Is not the exception already logged in StreamingAppMaster.main()? > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-438) ConcurrentModificationException in LogicalNode
[ https://issues.apache.org/jira/browse/APEXCORE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-438: Component/s: Buffer Server > ConcurrentModificationException in LogicalNode > -- > > Key: APEXCORE-438 > URL: https://issues.apache.org/jira/browse/APEXCORE-438 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Affects Versions: 3.2.1, 3.4.0, 3.3.1 >Reporter: Vlad Rozov >Assignee: Vlad Rozov > > LogicalNode.physicalNodes may be modified or accessed on two different > threads - DefaultEventLoop and ServerHelperExecutor causing > ConcurrentModificationException or other issues caused by thread safety. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-637) NullPointerException in BlacklistBasedResourceRequestHandler
[
https://issues.apache.org/jira/browse/APEXCORE-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov reassigned APEXCORE-637:
---
Assignee: Vlad Rozov
> NullPointerException in BlacklistBasedResourceRequestHandler
>
>
> Key: APEXCORE-637
> URL: https://issues.apache.org/jira/browse/APEXCORE-637
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Minor
>
> Application master terminates due to NullPointerException:
> {noformat}
> 2017-02-04 08:15:37,594 INFO com.datatorrent.stram.ResourceRequestHandler:
> Found host INVALID_HOST
> 2017-02-04 08:15:37,595 ERROR com.datatorrent.stram.StreamingAppMaster:
> Exiting Application Master
> java.lang.NullPointerException
> at
> com.datatorrent.stram.BlacklistBasedResourceRequestHandler.recreateContainerRequest(BlacklistBasedResourceRequestHandler.java:108)
> at
> com.datatorrent.stram.BlacklistBasedResourceRequestHandler.reissueContainerRequests(BlacklistBasedResourceRequestHandler.java:79)
> at
> com.datatorrent.stram.StreamingAppMasterService.execute(StreamingAppMasterService.java:830)
> at
> com.datatorrent.stram.StreamingAppMasterService.run(StreamingAppMasterService.java:652)
> at
> com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:104)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXCORE-637) NullPointerException in BlacklistBasedResourceRequestHandler
Vlad Rozov created APEXCORE-637:
---
Summary: NullPointerException in
BlacklistBasedResourceRequestHandler
Key: APEXCORE-637
URL: https://issues.apache.org/jira/browse/APEXCORE-637
Project: Apache Apex Core
Issue Type: Bug
Reporter: Vlad Rozov
Priority: Minor
Application master terminates due to NullPointerException:
{noformat}
2017-02-04 08:15:37,594 INFO com.datatorrent.stram.ResourceRequestHandler:
Found host INVALID_HOST
2017-02-04 08:15:37,595 ERROR com.datatorrent.stram.StreamingAppMaster: Exiting
Application Master
java.lang.NullPointerException
at
com.datatorrent.stram.BlacklistBasedResourceRequestHandler.recreateContainerRequest(BlacklistBasedResourceRequestHandler.java:108)
at
com.datatorrent.stram.BlacklistBasedResourceRequestHandler.reissueContainerRequests(BlacklistBasedResourceRequestHandler.java:79)
at
com.datatorrent.stram.StreamingAppMasterService.execute(StreamingAppMasterService.java:830)
at
com.datatorrent.stram.StreamingAppMasterService.run(StreamingAppMasterService.java:652)
at
com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:104)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXCORE-638) FSEventRecorder logs misleading ERROR messages
Vlad Rozov created APEXCORE-638:
---
Summary: FSEventRecorder logs misleading ERROR messages
Key: APEXCORE-638
URL: https://issues.apache.org/jira/browse/APEXCORE-638
Project: Apache Apex Core
Issue Type: Bug
Reporter: Vlad Rozov
Priority: Minor
Every time Application Master records StramEvent, it logs ERROR message in case
web socket receiver is not properly configured. The error is misleading as it
does not cause any problem and the exception is swallowed. There should be only
one WARN message, when websocket tries to connect and fails and one INFO
message, when websocket is able to re-connect:
{noformat}
2017-02-04 08:15:09,206 ERROR com.datatorrent.stram.FSEventRecorder: Caught
Exception
java.net.ConnectException: Connection refused:
node0.morado.com/192.168.2.37:10092
at
org.apache.apex.shaded.ning19.com.ning.http.client.providers.netty.request.NettyConnectListener.onFutureFailure(NettyConnectListener.java:133)
at
org.apache.apex.shaded.ning19.com.ning.http.client.providers.netty.request.NettyConnectListener.operationComplete(NettyConnectListener.java:145)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:400)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.apache.apex.shaded.ning19.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.apache.apex.shaded.ning19.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused:
node0.morado.com/192.168.2.37:10092
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at
org.apache.apex.shaded.ning19.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
... 8 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXCORE-639) Application is reported as FINISHED (SUCCEEDED) even though it failed to start properly
Vlad Rozov created APEXCORE-639: --- Summary: Application is reported as FINISHED (SUCCEEDED) even though it failed to start properly Key: APEXCORE-639 URL: https://issues.apache.org/jira/browse/APEXCORE-639 Project: Apache Apex Core Issue Type: Bug Reporter: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-639) Application is reported as FINISHED (SUCCEEDED) even though it failed to start properly
[ https://issues.apache.org/jira/browse/APEXCORE-639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852852#comment-15852852 ] Vlad Rozov commented on APEXCORE-639: - Application Master terminates with NullPointerException, but Application is reported as FINISHED. > Application is reported as FINISHED (SUCCEEDED) even though it failed to > start properly > --- > > Key: APEXCORE-639 > URL: https://issues.apache.org/jira/browse/APEXCORE-639 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-617) InputNodeTest intermittently fails with ConcurrentModificationException
Vlad Rozov created APEXCORE-617: --- Summary: InputNodeTest intermittently fails with ConcurrentModificationException Key: APEXCORE-617 URL: https://issues.apache.org/jira/browse/APEXCORE-617 Project: Apache Apex Core Issue Type: Bug Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor InputNodeTest.emitTestHelper uses deprecated inherently unsafe Thread.stop(). Thread.stop() does not immediately stops the thread, the JVM waits for a safepoint to raise ThreadDeath exception on the target thread. This leads to a race condition and possible ConcurrentModificationException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow
[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-570: Component/s: Buffer Server > Prevent upstream operators from getting too far ahead when downstream > operators are slow > > > Key: APEXCORE-570 > URL: https://issues.apache.org/jira/browse/APEXCORE-570 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > If the downstream operators are slower than upstream operators then the > upstream operators will get ahead and the gap can continue to increase. > Provide an option to slow down or temporarily pause the upstream operators > when they get too far ahead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-609) Container runs out of memory when buffer spooling is disabled
[ https://issues.apache.org/jira/browse/APEXCORE-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-609: Component/s: Buffer Server > Container runs out of memory when buffer spooling is disabled > - > > Key: APEXCORE-609 > URL: https://issues.apache.org/jira/browse/APEXCORE-609 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > When buffer spooling is disabled and the specified buffer in memory limit is > reached, because of fault tolerance requirements, additional memory is > allocated to store the new data. This causes container to run out of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-611) Stram Event Log Levels
[ https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-611. - Resolution: Implemented Fix Version/s: 3.6.0 > Stram Event Log Levels > -- > > Key: APEXCORE-611 > URL: https://issues.apache.org/jira/browse/APEXCORE-611 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Minor > Fix For: 3.6.0 > > > Provide log levels for Stram events. Such as INFO, WARN, ERROR > Eg: > 1. Start Container, Start Operator are INFO level events > 2. OperatorError is ERROR level event > 3. Stop Container, Stop Operator are WARN level events > Log level for events can help in user experience when showing the event list > in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log > levels so that user can focus more on ERROR and WARN events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-611) Stram Event Log Levels
[ https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-611: Priority: Minor (was: Major) > Stram Event Log Levels > -- > > Key: APEXCORE-611 > URL: https://issues.apache.org/jira/browse/APEXCORE-611 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Assignee: Ajay Gupta >Priority: Minor > > Provide log levels for Stram events. Such as INFO, WARN, ERROR > Eg: > 1. Start Container, Start Operator are INFO level events > 2. OperatorError is ERROR level event > 3. Stop Container, Stop Operator are WARN level events > Log level for events can help in user experience when showing the event list > in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log > levels so that user can focus more on ERROR and WARN events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (APEXCORE-611) Stram Event Log Levels
[ https://issues.apache.org/jira/browse/APEXCORE-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-611: Assignee: Ajay Gupta > Stram Event Log Levels > -- > > Key: APEXCORE-611 > URL: https://issues.apache.org/jira/browse/APEXCORE-611 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Ajay Gupta >Assignee: Ajay Gupta > > Provide log levels for Stram events. Such as INFO, WARN, ERROR > Eg: > 1. Start Container, Start Operator are INFO level events > 2. OperatorError is ERROR level event > 3. Stop Container, Stop Operator are WARN level events > Log level for events can help in user experience when showing the event list > in a GUI. eg: In datatorrent gateway UI, we want to provide color-coded log > levels so that user can focus more on ERROR and WARN events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-620) Support Fat Jar without POM, allowing Gradle and other build systems
[ https://issues.apache.org/jira/browse/APEXCORE-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839111#comment-15839111 ] Vlad Rozov commented on APEXCORE-620: - What is the proposed mechanism to resolve dependencies? Apex application developer should provide .apa file with all dependencies (except the one that will be provided at run-time) resolved. > Support Fat Jar without POM, allowing Gradle and other build systems > > > Key: APEXCORE-620 > URL: https://issues.apache.org/jira/browse/APEXCORE-620 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Jose Casillas > > For a project that is built with Gradle to be used in Apex, we must convert > our suite of libraries to Maven or resort to: > apex> launch -ignorepom -libjars x,y,z,... > Can the launcher simply accept a jar and include all of the dependency jars > within it, without requiring a POM or explicit declaration with -libjars? > As a stop-gap, allowing -libjars to support a directory of dependencies would > free us from specifying each one in the launch statement. > The fat jar support would allow build systems other than Maven to power Apex > applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (APEXCORE-486) Facing Method not found issue while launching Kafka application
[ https://issues.apache.org/jira/browse/APEXCORE-486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov closed APEXCORE-486. --- > Facing Method not found issue while launching Kafka application > --- > > Key: APEXCORE-486 > URL: https://issues.apache.org/jira/browse/APEXCORE-486 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Anuj > > Facing Method not found issue while launching Kafka application. > java.lang.NoSuchMethodError: > net.engio.mbassy.bus.config.BusConfiguration.Default(III)Lnet/engio/mbassy/bus/config/BusConfiguration; > at > com.datatorrent.stram.StreamingContainerManager.(StreamingContainerManager.java:360) > at > com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingContainerManager.java:2979) > at > com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingAppMasterService.java:550) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:101) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (APEXCORE-390) Apex Build shows error in Eclipse
[ https://issues.apache.org/jira/browse/APEXCORE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov closed APEXCORE-390. --- > Apex Build shows error in Eclipse > - > > Key: APEXCORE-390 > URL: https://issues.apache.org/jira/browse/APEXCORE-390 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Jayaradha >Priority: Minor > > Checkout code from git and build the code. I am getting below error. > Plugin execution not covered by lifecycle configuration: > org.apache.maven.plugins:maven- > dependency-plugin:2.1:build-classpath (execution: > create-client-mvn-generated- > classpath-no-hadoop, phase: generate-resources) > Environment Used: > java version 1.8.0_60-b27 > Maven verion 3.3.3 > Eclipse IDE(Eclipse 4.4 for Mac OS X) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (APEXCORE-214) Containers Fail With RPC Timeout
[ https://issues.apache.org/jira/browse/APEXCORE-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov closed APEXCORE-214. --- > Containers Fail With RPC Timeout > > > Key: APEXCORE-214 > URL: https://issues.apache.org/jira/browse/APEXCORE-214 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Timothy Farkas >Assignee: Pramod Immaneni > > 2015-10-22 09:32:28,545 WARN com.datatorrent.stram.RecoverableRpcProxy: RPC > failure, attempting reconnect after 1 ms (remaining 28998 ms) > java.lang.reflect.UndeclaredThrowableException > at com.sun.proxy.$Proxy16.processHeartbeat(Unknown Source) > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:124) > at com.sun.proxy.$Proxy16.processHeartbeat(Unknown Source) > at > com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:676) > at > com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:275) > Caused by: java.io.EOFException: End of File Exception between local host is: > "node6.morado.com/192.168.2.10"; destination host is: > "node4.morado.com":33745; : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > ... 8 more > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:392) > at > org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (APEXCORE-596) Committed method on operators not called when stream locality is THREAD_LOCAL
[
https://issues.apache.org/jira/browse/APEXCORE-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved APEXCORE-596.
-
Resolution: Fixed
Fix Version/s: 3.6.0
> Committed method on operators not called when stream locality is THREAD_LOCAL
> -
>
> Key: APEXCORE-596
> URL: https://issues.apache.org/jira/browse/APEXCORE-596
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Francis Fernandes
>Assignee: Francis Fernandes
>Priority: Minor
> Fix For: 3.6.0
>
>
> When the locality of the stream connecting the two operators is
> Locality.THREAD_LOCAL, the committed method is not called for some operators.
> These operators implement the Operator.CheckpointListener. e.g.
> AbstractFileOutputOperator
> For thread local during activate we do not set the thread in the node's
> context
> Because the thread is not set, we skip this operator in the
> processHeartBeatResponse and the committed is not called
> {code}
> if (thread == null || !thread.isAlive()) {
> continue;
> }
> {code}
> We need this condition for invalid operators (operator failures) in case of
> other localities.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (APEXCORE-596) Committed method on operators not called when stream locality is THREAD_LOCAL
[
https://issues.apache.org/jira/browse/APEXCORE-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-596:
Priority: Minor (was: Major)
> Committed method on operators not called when stream locality is THREAD_LOCAL
> -
>
> Key: APEXCORE-596
> URL: https://issues.apache.org/jira/browse/APEXCORE-596
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Francis Fernandes
>Assignee: Francis Fernandes
>Priority: Minor
> Fix For: 3.6.0
>
>
> When the locality of the stream connecting the two operators is
> Locality.THREAD_LOCAL, the committed method is not called for some operators.
> These operators implement the Operator.CheckpointListener. e.g.
> AbstractFileOutputOperator
> For thread local during activate we do not set the thread in the node's
> context
> Because the thread is not set, we skip this operator in the
> processHeartBeatResponse and the committed is not called
> {code}
> if (thread == null || !thread.isAlive()) {
> continue;
> }
> {code}
> We need this condition for invalid operators (operator failures) in case of
> other localities.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (APEXCORE-625) Exception from StreamingAppMasterService.execute is lost if eventloop.stop thrown an exception.
[ https://issues.apache.org/jira/browse/APEXCORE-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843095#comment-15843095 ] Vlad Rozov commented on APEXCORE-625: - Please provide details (exception stack trace) for the exception raised in the eventloop.stop(); eventloop.stop() is asynchronous and actual execution is done on the event loop thread. I don't see why eventloop.stop() will be raising an exception. > Exception from StreamingAppMasterService.execute is lost if eventloop.stop > thrown an exception. > --- > > Key: APEXCORE-625 > URL: https://issues.apache.org/jira/browse/APEXCORE-625 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Assignee: Tushar Gosavi > > This was observed while debugging an application master crash. The > application master main loop terminated because of an exception, but this > exception was masked by exception thrown from eventloop stop in finally > block, causing difficulties in debugging the original issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-648) Unnecessary byte array copy in DefaultStatefulStreamCodec.toDataStatePair()
[ https://issues.apache.org/jira/browse/APEXCORE-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874903#comment-15874903 ] Vlad Rozov commented on APEXCORE-648: - [~brightchen] Please see https://github.com/vrozov/apex-core/tree/APEXCORE-648. Can you benchmark the fix and compare old and new behavior. > Unnecessary byte array copy in DefaultStatefulStreamCodec.toDataStatePair() > --- > > Key: APEXCORE-648 > URL: https://issues.apache.org/jira/browse/APEXCORE-648 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > DefaultStatefulStreamCodec.toDataStatePair() calls Kryo Output.toBytes() that > creates new byte[] and copies serialized data to the newly allocated array. > It is not necessary as the output of toDataStatePair() will be copied again > in PayloadTuple.getSerializedTuple(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-570) Prevent upstream operators from getting too far ahead when downstream operators are slow
[ https://issues.apache.org/jira/browse/APEXCORE-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-570. - Resolution: Implemented Fix Version/s: 3.6.0 > Prevent upstream operators from getting too far ahead when downstream > operators are slow > > > Key: APEXCORE-570 > URL: https://issues.apache.org/jira/browse/APEXCORE-570 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > Fix For: 3.6.0 > > > If the downstream operators are slower than upstream operators then the > upstream operators will get ahead and the gap can continue to increase. > Provide an option to slow down or temporarily pause the upstream operators > when they get too far ahead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-624) Shutdown does not work because of incorrect logic in the AppMaster
[ https://issues.apache.org/jira/browse/APEXCORE-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-624. - Resolution: Fixed Fix Version/s: 3.4.1 3.6.0 3.5.1 3.2.2 3.3.1 > Shutdown does not work because of incorrect logic in the AppMaster > -- > > Key: APEXCORE-624 > URL: https://issues.apache.org/jira/browse/APEXCORE-624 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.3.1, 3.2.2, 3.5.1, 3.6.0, 3.4.1 > > > com.datatorrent.stram.StreamingAppMasterService.execute() calculates > numRequestedContainers incorrectly in some cases (e.g. RM container > allocation failure) which prevents an application from shutting down when it > is requested externally. An example is where we ask RM to remove previous > container allocation request (where the count should be decremented but is > NOT) and add a new one (where the count should be and IS incremented). > Another example is the "alreadyAllocated" case where we release the container > and still increment numRequestedContainers which seems wrong. > This bug is showing up in multiple Apex deployments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-650) Remove ResetWindowTuple
[ https://issues.apache.org/jira/browse/APEXCORE-650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-650: Issue Type: Sub-task (was: Improvement) Parent: APEXCORE-653 > Remove ResetWindowTuple > --- > > Key: APEXCORE-650 > URL: https://issues.apache.org/jira/browse/APEXCORE-650 > Project: Apache Apex Core > Issue Type: Sub-task >Reporter: Sandesh > > ResetWindowTuple raises the complexity of the engine and development, without > much benefit. It is better to remove ResetWindowTuple from the engine. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Moved] (APEXMALHAR-2425) Error in document for PI example
[ https://issues.apache.org/jira/browse/APEXMALHAR-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov moved APEXCORE-652 to APEXMALHAR-2425: - Workflow: Default workflow, editable Closed status (was: jira) Key: APEXMALHAR-2425 (was: APEXCORE-652) Project: Apache Apex Malhar (was: Apache Apex Core) > Error in document for PI example > - > > Key: APEXMALHAR-2425 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2425 > Project: Apache Apex Malhar > Issue Type: Documentation >Reporter: Dongming Liang >Priority: Minor > Labels: newbie > > In Application_development document: > "PI is computed as N/number of values received" > should be: > "PI is computed as 4*N/number of values received" -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-653) windows ID should always increment by 1 (continuously)
Vlad Rozov created APEXCORE-653: --- Summary: windows ID should always increment by 1 (continuously) Key: APEXCORE-653 URL: https://issues.apache.org/jira/browse/APEXCORE-653 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov currently, wID are increment by 1 until they cross 16K boundary. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-633) Add unit tests for DataList
[ https://issues.apache.org/jira/browse/APEXCORE-633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-633: Priority: Minor (was: Major) > Add unit tests for DataList > --- > > Key: APEXCORE-633 > URL: https://issues.apache.org/jira/browse/APEXCORE-633 > Project: Apache Apex Core > Issue Type: Sub-task > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-630) Remove FastDataList & related components from BufferServer & Engine
[ https://issues.apache.org/jira/browse/APEXCORE-630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-630: Priority: Minor (was: Major) > Remove FastDataList & related components from BufferServer & Engine > --- > > Key: APEXCORE-630 > URL: https://issues.apache.org/jira/browse/APEXCORE-630 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > > FastDataList implementation is not used and there is no plan to complete it, > so there is no advantage in keeping it anymore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-632) Fix the existing unit test and move them to JUnit from testNg
[ https://issues.apache.org/jira/browse/APEXCORE-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-632: Priority: Minor (was: Major) > Fix the existing unit test and move them to JUnit from testNg > - > > Key: APEXCORE-632 > URL: https://issues.apache.org/jira/browse/APEXCORE-632 > Project: Apache Apex Core > Issue Type: Sub-task > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-631) Improving BufferServer tests
[ https://issues.apache.org/jira/browse/APEXCORE-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-631: Priority: Minor (was: Major) > Improving BufferServer tests > > > Key: APEXCORE-631 > URL: https://issues.apache.org/jira/browse/APEXCORE-631 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-648) Unnecessary byte array copy in DefaultStatefulStreamCodec.toDataStatePair()
Vlad Rozov created APEXCORE-648: --- Summary: Unnecessary byte array copy in DefaultStatefulStreamCodec.toDataStatePair() Key: APEXCORE-648 URL: https://issues.apache.org/jira/browse/APEXCORE-648 Project: Apache Apex Core Issue Type: Bug Reporter: Vlad Rozov Assignee: Vlad Rozov Priority: Minor DefaultStatefulStreamCodec.toDataStatePair() calls Kryo Output.toBytes() that creates new byte[] and copies serialized data to the newly allocated array. It is not necessary as the output of toDataStatePair() will be copied again in PayloadTuple.getSerializedTuple(). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-635) Proposal: Manage memory to avoid memory copy and garbage collection
[
https://issues.apache.org/jira/browse/APEXCORE-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873940#comment-15873940
]
Vlad Rozov commented on APEXCORE-635:
-
bq. As I understand, array is a kind of object in java. Does java implemented
differently? And how can JVM reuse allocated bytes without collection? Do you
have any reference I can take a look?
Yes, an array is an object in java that may hold either primitive types or
other objects. When an array is GC, it is GC as a single object. If an array
holds primitive types, the array's memory is reclaimed in a single step, there
is no GC of individual bytes/int/long in the array.
bq. I copied the implementation of kryo Output.toBytes() here ( kryo-2.24.0.jar
). From this implementation toBytes() do allocate new memory and copy.
I was referring to Output.getBuffer() and DefaultKryoStreamCodec that uses
Output.getBuffer(). You are right that Output.toBytes() allocates a new array
and copies bytes from Output internal buffer to the newly allocated buffer.
This is a bug in DefaultStatefulStreamCodec, it should use Output.getBuffer().
bq. I can get number, but that means need at least simply prototype. And based
on my previous test for serialization string of 1000 ascii, The performance can
gain 30%
As I pointed to you, the prototype has a significant bug (reusing byte arrays
that are not reusable due to fire and forget netlet behavior). What I would
like to see is a series of small prototypes that prove that the approach
provides significant performance gains.
I don't see how without modifying the current definition of StreamCodec and
StatefulStreamCodec the solution can be generalized. It may be possible to
improve DefaultStatefulStreamCodec (concrete implementation of
StatefulStreamCodec), but it seems to require a large amount of modifications
to netlet and buffer server that other StreamCodec implementations will not
benefit from. Additionally, the proposal seems to require netlet client to be
aware of codec implementation and any changes to tuple format will require
changes to serialization and de-serialization code.
> Proposal: Manage memory to avoid memory copy and garbage collection
> ---
>
> Key: APEXCORE-635
> URL: https://issues.apache.org/jira/browse/APEXCORE-635
> Project: Apache Apex Core
> Issue Type: Wish
>Reporter: bright chen
>Assignee: bright chen
>
> Manage memory to avoid memory copy and garbage collection
> The aim of this proposal is to reuse the memory to avoid the garbage
> collection and avoid unnecessary memory copy to increase the performance. In
> this proposal the term serde means serialization and deserialization. It’s
> same as codec.
> Currently, apex by default use DefaultStatefulStreamCodec for serde, which
> extends Kryo and optimize it by replace class by class id. And application
> developer can optimize serializer by implement interface StreamCodec.
> First, let’s look into the default codec DefaultStatefulStreamCodec. It
> basically optimize serde by replace class name by class id as my
> understanding. And the state information only send before sending first
> tuple, it’s kind like configuration for serde. So I suggest to separate this
> feature from serde. The benefit is the customized serde can still use this
> feature. And the kryo have some limitation which I’ll state later.
> Second, Let’s look at the customized serde. Let’s stand from application
> developer point of view and look at how to implement StreamCodec. I take a
> simple tuple List as example.
> The first solution is use kryo. This is basically same as apex default codec.
> The second solution is implement StreamCodec for String and List, and
> ListSerde delegate String to StringSerde. The benefit of this solution is the
> StringSerde ListSerde can be reused. The problem is there need a lot of
> temporary memory and memory copy. Following is the sample implement.
> Class StringSerde {
> Slice toByteArray(String o) {
> byte[] b = o.getBytes(“UTF8”); // new bytes
> byte[] b1 = new byte[b1.length + 4]; // new bytes
> set the length of the string at the first 4 bytes
> System.arrayCopy(b, 0, b1, 4, b.length); //copy bytes
> return new Slice(b1);
> }
> class ListSerde {
> StreamCodec itemSerde; //the serde for serialize/deserialize item
> Slice toByteArray(List list) {
> Slice[] itemSlices = new Slice[list.size()];
> int size = 0;
> int index = 0;
> for(T item : list) {
> Slice slice = itemSerde.toByteArray(item);
> size += slice.length;
> itemSlices[index++] = slice;
> }
> byte[] b = new byte[size+4]; //allocated the memory
> set the length of the list at the first 4 bytes
> copy the data
[jira] [Commented] (APEXCORE-635) Proposal: Manage memory to avoid memory copy and garbage collection
[
https://issues.apache.org/jira/browse/APEXCORE-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862609#comment-15862609
]
Vlad Rozov commented on APEXCORE-635:
-
- JVM does not garbage collect bytes, it garbage collects objects.
- Output.toBytes() does not copy anything
- Without actual numbers, the discussion is abstract and is not convincing.
- Complexity of the proposal is quite high
- Please see my prior comment, there are still issues not addressed
(n)
> Proposal: Manage memory to avoid memory copy and garbage collection
> ---
>
> Key: APEXCORE-635
> URL: https://issues.apache.org/jira/browse/APEXCORE-635
> Project: Apache Apex Core
> Issue Type: Wish
>Reporter: bright chen
>Assignee: bright chen
>
> Manage memory to avoid memory copy and garbage collection
> The aim of this proposal is to reuse the memory to avoid the garbage
> collection and avoid unnecessary memory copy to increase the performance. In
> this proposal the term serde means serialization and deserialization. It’s
> same as codec.
> Currently, apex by default use DefaultStatefulStreamCodec for serde, which
> extends Kryo and optimize it by replace class by class id. And application
> developer can optimize serializer by implement interface StreamCodec.
> First, let’s look into the default codec DefaultStatefulStreamCodec. It
> basically optimize serde by replace class name by class id as my
> understanding. And the state information only send before sending first
> tuple, it’s kind like configuration for serde. So I suggest to separate this
> feature from serde. The benefit is the customized serde can still use this
> feature. And the kryo have some limitation which I’ll state later.
> Second, Let’s look at the customized serde. Let’s stand from application
> developer point of view and look at how to implement StreamCodec. I take a
> simple tuple List as example.
> The first solution is use kryo. This is basically same as apex default codec.
> The second solution is implement StreamCodec for String and List, and
> ListSerde delegate String to StringSerde. The benefit of this solution is the
> StringSerde ListSerde can be reused. The problem is there need a lot of
> temporary memory and memory copy. Following is the sample implement.
> Class StringSerde {
> Slice toByteArray(String o) {
> byte[] b = o.getBytes(“UTF8”); // new bytes
> byte[] b1 = new byte[b1.length + 4]; // new bytes
> set the length of the string at the first 4 bytes
> System.arrayCopy(b, 0, b1, 4, b.length); //copy bytes
> return new Slice(b1);
> }
> class ListSerde {
> StreamCodec itemSerde; //the serde for serialize/deserialize item
> Slice toByteArray(List list) {
> Slice[] itemSlices = new Slice[list.size()];
> int size = 0;
> int index = 0;
> for(T item : list) {
> Slice slice = itemSerde.toByteArray(item);
> size += slice.length;
> itemSlices[index++] = slice;
> }
> byte[] b = new byte[size+4]; //allocated the memory
> set the length of the list at the first 4 bytes
> copy the data from itemSlices
> return new Slice(b);
> }
> }
>
> from above code, we can see that around 2 times of required memory were
> allocated and data copied twice( one copy maybe special to string, but
> another copy is mandatory). And when bytes written to the socket, all
> allocated memory can’t be reused but need to be garbage collected.
> The above tuple only have two levels, if the tuple have n level, n times of
> required memory would be allocated and n-1 time of data copy is required.
> The third solution could be allocate memory and then pass the memory and
> offset to item serde. There are some problems for this solution:
> How to pass the memory from caller? As our previous interface only pass the
> object but no way to pass memory. So the pass of memory will depends on
> implementation.
> Another big problem of this solution is it hard to reallocate proper
> memory(For this special case, it probably can allocate 2 times of all string
> length. ). And the memory allocated more than required would be wasted until
> data send to the socket(or allocate exact memory and copy the data to avoid
> waste memory). And the code also need to handle the case if memory is not
> enough.
> The fourth solution could be treat whole object as flat, allocate memory and
> handle it. For example as following. This solution solve the problem of pass
> memory. But it has other problems of third solution and introduced some other
> problems:
> Can’t reuse the code: we already have the StringSerde, but ListSerde
> have to implement almost same logic again.
> The serializeItemToMemory() method should be implemented depend on different
[jira] [Commented] (APEXCORE-635) Proposal: Manage memory to avoid memory copy and garbage collection
[
https://issues.apache.org/jira/browse/APEXCORE-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860770#comment-15860770
]
Vlad Rozov commented on APEXCORE-635:
-
- Memory copy is required as it moves data from JVM heap to direct (off-heap)
buffer. It is done prior to writing the buffer to a network channel.
- Please benchmark copying one large block of memory in a single step and in
multiple steps. What is the difference (in %)? Do the same for a direct buffer.
- How will control tuples be injected into a continuous block of memory?
- The current approach is fire and forget, the proposed requires some kind of
an acknowledgment. I am not convinced that the design that requires an
acknowledgment will be faster.
- There is a plan to move to netty from netlet and netty provides buffer pool
allocator that may better fit the proposed design.
- The current format uses variable length (1, 2, 3 or 4 bytes) prefix and it is
not possible to know offset where data needs to be written prior to the lenght
of the serialized data is known.
> Proposal: Manage memory to avoid memory copy and garbage collection
> ---
>
> Key: APEXCORE-635
> URL: https://issues.apache.org/jira/browse/APEXCORE-635
> Project: Apache Apex Core
> Issue Type: Wish
>Reporter: bright chen
>Assignee: bright chen
>
> Manage memory to avoid memory copy and garbage collection
> The aim of this proposal is to reuse the memory to avoid the garbage
> collection and avoid unnecessary memory copy to increase the performance. In
> this proposal the term serde means serialization and deserialization. It’s
> same as codec.
> Currently, apex by default use DefaultStatefulStreamCodec for serde, which
> extends Kryo and optimize it by replace class by class id. And application
> developer can optimize serializer by implement interface StreamCodec.
> First, let’s look into the default codec DefaultStatefulStreamCodec. It
> basically optimize serde by replace class name by class id as my
> understanding. And the state information only send before sending first
> tuple, it’s kind like configuration for serde. So I suggest to separate this
> feature from serde. The benefit is the customized serde can still use this
> feature. And the kryo have some limitation which I’ll state later.
> Second, Let’s look at the customized serde. Let’s stand from application
> developer point of view and look at how to implement StreamCodec. I take a
> simple tuple List as example.
> The first solution is use kryo. This is basically same as apex default codec.
> The second solution is implement StreamCodec for String and List, and
> ListSerde delegate String to StringSerde. The benefit of this solution is the
> StringSerde ListSerde can be reused. The problem is there need a lot of
> temporary memory and memory copy. Following is the sample implement.
> Class StringSerde {
> Slice toByteArray(String o) {
> byte[] b = o.getBytes(“UTF8”); // new bytes
> byte[] b1 = new byte[b1.length + 4]; // new bytes
> set the length of the string at the first 4 bytes
> System.arrayCopy(b, 0, b1, 4, b.length); //copy bytes
> return new Slice(b1);
> }
> class ListSerde {
> StreamCodec itemSerde; //the serde for serialize/deserialize item
> Slice toByteArray(List list) {
> Slice[] itemSlices = new Slice[list.size()];
> int size = 0;
> int index = 0;
> for(T item : list) {
> Slice slice = itemSerde.toByteArray(item);
> size += slice.length;
> itemSlices[index++] = slice;
> }
> byte[] b = new byte[size+4]; //allocated the memory
> set the length of the list at the first 4 bytes
> copy the data from itemSlices
> return new Slice(b);
> }
> }
>
> from above code, we can see that around 2 times of required memory were
> allocated and data copied twice( one copy maybe special to string, but
> another copy is mandatory). And when bytes written to the socket, all
> allocated memory can’t be reused but need to be garbage collected.
> The above tuple only have two levels, if the tuple have n level, n times of
> required memory would be allocated and n-1 time of data copy is required.
> The third solution could be allocate memory and then pass the memory and
> offset to item serde. There are some problems for this solution:
> How to pass the memory from caller? As our previous interface only pass the
> object but no way to pass memory. So the pass of memory will depends on
> implementation.
> Another big problem of this solution is it hard to reallocate proper
> memory(For this special case, it probably can allocate 2 times of all string
> length. ). And the memory allocated more than required would be wasted until
> data send to the socket(or
[jira] [Commented] (APEXCORE-599) Data not fully processed when operator terminates via ShutdownException
[ https://issues.apache.org/jira/browse/APEXCORE-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866753#comment-15866753 ] Vlad Rozov commented on APEXCORE-599: - [~thw] Can it be caused by APEXCORE-645? > Data not fully processed when operator terminates via ShutdownException > --- > > Key: APEXCORE-599 > URL: https://issues.apache.org/jira/browse/APEXCORE-599 > Project: Apache Apex Core > Issue Type: Bug >Affects Versions: 3.5.0 >Reporter: Thomas Weise > > Observed in the form of flaky unit tests in Beam. The pipelines terminate > using shutdown by the operator when there is no more input and occasionally > the test will fail because the cluster exits without having processed fully. > The bandaid is to have a delay before shutdown is issued, allowing the last > tuples to be pushed downstream. See BEAM-1140 for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-607) Add unit tests for BlackListedBasedResourceRequestHandler
[ https://issues.apache.org/jira/browse/APEXCORE-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-607: Priority: Minor (was: Major) > Add unit tests for BlackListedBasedResourceRequestHandler > - > > Key: APEXCORE-607 > URL: https://issues.apache.org/jira/browse/APEXCORE-607 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Priority: Minor > > There is no unit test coverage for the following class > BlackListedBasedResourceRequestHandler. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-606) Suggestion: Optimise Kryo Output
[ https://issues.apache.org/jira/browse/APEXCORE-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-606: Priority: Minor (was: Major) > Suggestion: Optimise Kryo Output > > > Key: APEXCORE-606 > URL: https://issues.apache.org/jira/browse/APEXCORE-606 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: bright chen >Assignee: bright chen >Priority: Minor > > The kryo Output has some limitation > - The size of the data is limited. kryo write data to the buffer, it will > throw the overflow exception if the data exceed the size > - The Output.toBytes() will copy the data to temporary buffer and output, > it will decrease the performance and introduce garbage collection. > When I was tuning Spillable Data structure and Manage State. I create a > mechanism to share and reuse the memory to avoid above problem. And it can > be reused in core serialization with small change. Please see jira: > https://issues.apache.org/jira/browse/APEXMALHAR-2190 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-622) event files are overwritten when master restarts.
[ https://issues.apache.org/jira/browse/APEXCORE-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-622: Priority: Minor (was: Major) > event files are overwritten when master restarts. > - > > Key: APEXCORE-622 > URL: https://issues.apache.org/jira/browse/APEXCORE-622 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Tushar Gosavi >Priority: Minor > > The events file written in application_path/events are overwritten in case of > master restart. The important debugging information is lost while debugging > cause of master restart. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-621) populate TIMEOUT_WINDOW_COUNT for thread local operators from downstreams.
[ https://issues.apache.org/jira/browse/APEXCORE-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-621: Priority: Minor (was: Major) > populate TIMEOUT_WINDOW_COUNT for thread local operators from downstreams. > -- > > Key: APEXCORE-621 > URL: https://issues.apache.org/jira/browse/APEXCORE-621 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Tushar Gosavi >Assignee: Deepak Narkhede >Priority: Minor > > A -> B -> C -> D > In above dag if we have set TIMEOUT_WINDOW_COUNT on 'C' and 'B' and 'C' are > in thread local, then 'B' uses default TIMEOUT_WINDOW_COUNT attribute and > marked as blocked opeator while C is performing a time cosuming operation. > The problem is more visible when operator B is partitioned and unifiers are > deployed thread local to C, in this case unifiers are declared are blocked, > and users need to remember to set TIMEOUT_WINDOW_COUNT on unifiers. > Instead platform could inherit TIMEOUT_WINDOW_COUNT attribute from downstream > operator in case of threadlocal/container local case to avoid getting > detected as blocked early. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-620) Support Fat Jar without POM, allowing Gradle and other build systems
[ https://issues.apache.org/jira/browse/APEXCORE-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-620: Priority: Minor (was: Major) > Support Fat Jar without POM, allowing Gradle and other build systems > > > Key: APEXCORE-620 > URL: https://issues.apache.org/jira/browse/APEXCORE-620 > Project: Apache Apex Core > Issue Type: New Feature >Reporter: Jose Casillas >Priority: Minor > > For a project that is built with Gradle to be used in Apex, we must convert > our suite of libraries to Maven or resort to: > apex> launch -ignorepom -libjars x,y,z,... > Can the launcher simply accept a jar and include all of the dependency jars > within it, without requiring a POM or explicit declaration with -libjars? > As a stop-gap, allowing -libjars to support a directory of dependencies would > free us from specifying each one in the launch statement. > The fat jar support would allow build systems other than Maven to power Apex > applications. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-615) Out of sequence tuple exception - kill the whole app
[ https://issues.apache.org/jira/browse/APEXCORE-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-615: Priority: Minor (was: Major) > Out of sequence tuple exception - kill the whole app > > > Key: APEXCORE-615 > URL: https://issues.apache.org/jira/browse/APEXCORE-615 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sandesh >Assignee: Sandesh >Priority: Minor > > In case of out of sequence tuple, killing the containers will not solve the > problem as container will continue to fail after recovery. Killing the app > is a better option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-601) provide log level in the event
[ https://issues.apache.org/jira/browse/APEXCORE-601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-601. - Resolution: Duplicate > provide log level in the event > --- > > Key: APEXCORE-601 > URL: https://issues.apache.org/jira/browse/APEXCORE-601 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Sanjay M Pujare > > Provide log levels for stram events. Such as INFO, WARN, ERROR > Eg: > 1. Start Container, Start Operator are INFO level events > 2. OperatorError is ERROR level event > 3. Stop Container, Stop Operator are WARN level events > These levels are used in the DT log file entries so the intent is to make the > same level available to the event consumer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-600) Dynamic change of operator property not adhering to constraint annotations
[
https://issues.apache.org/jira/browse/APEXCORE-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-600:
Priority: Minor (was: Major)
> Dynamic change of operator property not adhering to constraint annotations
> --
>
> Key: APEXCORE-600
> URL: https://issues.apache.org/jira/browse/APEXCORE-600
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Francis Fernandes
>Assignee: Francis Fernandes
>Priority: Minor
>
> Create an operator with a field having some constraint
> {code}
> @ Min(0)
> protected int nonNegativeVal = 10;
> {code}
> When launching the application the constraint is adhered and an exception is
> thrown which prevents the application from launching:
> {code}
>
> dt.operator.test.prop.nonNegativeVal
> -10
>
> {code}
> The application is launched with a valid value and enters into running state.
> The constraint is not adhered to on updating the property with an invalid
> (negative) value.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (APEXCORE-597) BufferServer needs to shutdown all created execution services
[ https://issues.apache.org/jira/browse/APEXCORE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-597: Summary: BufferServer needs to shutdown all created execution services (was: Container does not terminate after shutdown request from master.) > BufferServer needs to shutdown all created execution services > - > > Key: APEXCORE-597 > URL: https://issues.apache.org/jira/browse/APEXCORE-597 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Tushar Gosavi >Assignee: Vlad Rozov >Priority: Minor > > JVM does not shutdown cleanly after receiving shutdown request from Stram. > The issue may be because BufferServer is not shutdown. The issue is seen > after commit d1646e42bdf5594ef34070594733a7ca10123a3f > The demo application to recreate the issue is at > https://github.com/tushargosavi/apex-malhar/blob/shutdownapp/demos/pi/src/main/java/com/datatorrent/demos/pi/Application.java -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (APEXCORE-642) Catch all exceptions in StreamingAppMasterService.serviceInit() and create a StramEvent
[ https://issues.apache.org/jira/browse/APEXCORE-642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862989#comment-15862989 ] Vlad Rozov commented on APEXCORE-642: - I'd prefer to log exception in the main as it is the top most method and logging exception in serviceInit will lead to double logging as serviceInit() is not called directly by StreamingAppMaster, so there may be exceptions raised by a middle man and it will be impossible to distinguish exceptions raised by a middle man from exceptions raised by StreamingAppMasterService and already logged. It is OK to create StramEvent in the serviceInit() assuming that there is a way to record the event. As events are reported asynchronously, reporting an event and terminating app master will likely lead to the event being lost. > Catch all exceptions in StreamingAppMasterService.serviceInit() and create a > StramEvent > --- > > Key: APEXCORE-642 > URL: https://issues.apache.org/jira/browse/APEXCORE-642 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > When the AM (Stram) starts, it executes > StreamingAppMasterService.serviceInit() to perform service initialization as > per the Hadoop service contract. In this, the Stram deserializes the DAG > which can fail (e.g. bad jar versions or other deserialization issues) and > any exception is thrown is not logged in Apex log files or events. It is > proposed to catch these exceptions and log them to the dt log file as well as > Stram events so the Apex user knows about these exceptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-624) Shutdown does not work because of incorrect logic in the AppMaster
[ https://issues.apache.org/jira/browse/APEXCORE-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-624: Priority: Minor (was: Major) > Shutdown does not work because of incorrect logic in the AppMaster > -- > > Key: APEXCORE-624 > URL: https://issues.apache.org/jira/browse/APEXCORE-624 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > > com.datatorrent.stram.StreamingAppMasterService.execute() calculates > numRequestedContainers incorrectly in some cases (e.g. RM container > allocation failure) which prevents an application from shutting down when it > is requested externally. An example is where we ask RM to remove previous > container allocation request (where the count should be decremented but is > NOT) and add a new one (where the count should be and IS incremented). > Another example is the "alreadyAllocated" case where we release the container > and still increment numRequestedContainers which seems wrong. > This bug is showing up in multiple Apex deployments. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Closed] (APEXCORE-628) One Yarn with Multiple Apex Applications
[ https://issues.apache.org/jira/browse/APEXCORE-628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov closed APEXCORE-628. --- Resolution: Not A Bug > One Yarn with Multiple Apex Applications > > > Key: APEXCORE-628 > URL: https://issues.apache.org/jira/browse/APEXCORE-628 > Project: Apache Apex Core > Issue Type: Bug >Reporter: santhoshi > > Launching more than one (multiple) apex engine in one node with multiple > terminals and one yarn running. > 8042 port is used by one application which is making other application not to > run as port 8042 is already in use. > How to run more than one apex engine with same yarn? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-609) Container runs out of memory when buffer spooling is disabled
[ https://issues.apache.org/jira/browse/APEXCORE-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-609: Issue Type: Improvement (was: Bug) > Container runs out of memory when buffer spooling is disabled > - > > Key: APEXCORE-609 > URL: https://issues.apache.org/jira/browse/APEXCORE-609 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > When buffer spooling is disabled and the specified buffer in memory limit is > reached, because of fault tolerance requirements, additional memory is > allocated to store the new data. This causes container to run out of memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-609) Drop DataList blocks when BufferServer spooling is disabled
[ https://issues.apache.org/jira/browse/APEXCORE-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-609: Summary: Drop DataList blocks when BufferServer spooling is disabled (was: Container runs out of memory when buffer spooling is disabled) > Drop DataList blocks when BufferServer spooling is disabled > --- > > Key: APEXCORE-609 > URL: https://issues.apache.org/jira/browse/APEXCORE-609 > Project: Apache Apex Core > Issue Type: Improvement > Components: Buffer Server >Reporter: Pramod Immaneni >Assignee: Pramod Immaneni > > When buffer spooling is disabled and the specified buffer in memory limit is > reached, because of fault tolerance requirements, additional memory is > allocated to store the new data. This causes container to run out of memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-608) Streaming Containers use stale RPC proxy after connection is closed
[ https://issues.apache.org/jira/browse/APEXCORE-608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-608: Priority: Minor (was: Major) > Streaming Containers use stale RPC proxy after connection is closed > --- > > Key: APEXCORE-608 > URL: https://issues.apache.org/jira/browse/APEXCORE-608 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > When Application is killed and Application Master is terminated, Streaming > Containers initiate container exit sequence and use disconnected RPC proxy to > report errors back to already terminated Application Master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-597) Container does not terminate after shutdown request from master.
[ https://issues.apache.org/jira/browse/APEXCORE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-597: Component/s: Buffer Server > Container does not terminate after shutdown request from master. > > > Key: APEXCORE-597 > URL: https://issues.apache.org/jira/browse/APEXCORE-597 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Tushar Gosavi >Priority: Minor > > JVM does not shutdown cleanly after receiving shutdown request from Stram. > The issue may be because BufferServer is not shutdown. The issue is seen > after commit d1646e42bdf5594ef34070594733a7ca10123a3f > The demo application to recreate the issue is at > https://github.com/tushargosavi/apex-malhar/blob/shutdownapp/demos/pi/src/main/java/com/datatorrent/demos/pi/Application.java -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-597) Container does not terminate after shutdown request from master.
[ https://issues.apache.org/jira/browse/APEXCORE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-597: Priority: Minor (was: Major) > Container does not terminate after shutdown request from master. > > > Key: APEXCORE-597 > URL: https://issues.apache.org/jira/browse/APEXCORE-597 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Tushar Gosavi >Priority: Minor > > JVM does not shutdown cleanly after receiving shutdown request from Stram. > The issue may be because BufferServer is not shutdown. The issue is seen > after commit d1646e42bdf5594ef34070594733a7ca10123a3f > The demo application to recreate the issue is at > https://github.com/tushargosavi/apex-malhar/blob/shutdownapp/demos/pi/src/main/java/com/datatorrent/demos/pi/Application.java -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-597) Container does not terminate after shutdown request from master.
[ https://issues.apache.org/jira/browse/APEXCORE-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-597: --- Assignee: Vlad Rozov > Container does not terminate after shutdown request from master. > > > Key: APEXCORE-597 > URL: https://issues.apache.org/jira/browse/APEXCORE-597 > Project: Apache Apex Core > Issue Type: Bug > Components: Buffer Server >Reporter: Tushar Gosavi >Assignee: Vlad Rozov >Priority: Minor > > JVM does not shutdown cleanly after receiving shutdown request from Stram. > The issue may be because BufferServer is not shutdown. The issue is seen > after commit d1646e42bdf5594ef34070594733a7ca10123a3f > The demo application to recreate the issue is at > https://github.com/tushargosavi/apex-malhar/blob/shutdownapp/demos/pi/src/main/java/com/datatorrent/demos/pi/Application.java -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (APEXCORE-628) One Yarn with Multiple Apex Applications
[ https://issues.apache.org/jira/browse/APEXCORE-628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-628: Fix Version/s: (was: 3.6.0) > One Yarn with Multiple Apex Applications > > > Key: APEXCORE-628 > URL: https://issues.apache.org/jira/browse/APEXCORE-628 > Project: Apache Apex Core > Issue Type: Bug >Reporter: santhoshi > > Launching more than one (multiple) apex engine in one node with multiple > terminals and one yarn running. > 8042 port is used by one application which is making other application not to > run as port 8042 is already in use. > How to run more than one apex engine with same yarn? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (APEXCORE-628) One Yarn with Multiple Apex Applications
[ https://issues.apache.org/jira/browse/APEXCORE-628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reopened APEXCORE-628: - > One Yarn with Multiple Apex Applications > > > Key: APEXCORE-628 > URL: https://issues.apache.org/jira/browse/APEXCORE-628 > Project: Apache Apex Core > Issue Type: Bug >Reporter: santhoshi > Fix For: 3.6.0 > > > Launching more than one (multiple) apex engine in one node with multiple > terminals and one yarn running. > 8042 port is used by one application which is making other application not to > run as port 8042 is already in use. > How to run more than one apex engine with same yarn? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (APEXCORE-645) StramLocalCluster does not wait for master thread termination
[ https://issues.apache.org/jira/browse/APEXCORE-645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov reassigned APEXCORE-645: --- Assignee: Vlad Rozov > StramLocalCluster does not wait for master thread termination > - > > Key: APEXCORE-645 > URL: https://issues.apache.org/jira/browse/APEXCORE-645 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Minor > > When StramLocalCluster is started asynchroniously it does not wait for the > master thread to terminate -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (APEXCORE-645) StramLocalCluster does not wait for master thread termination
Vlad Rozov created APEXCORE-645: --- Summary: StramLocalCluster does not wait for master thread termination Key: APEXCORE-645 URL: https://issues.apache.org/jira/browse/APEXCORE-645 Project: Apache Apex Core Issue Type: Bug Reporter: Vlad Rozov Priority: Minor When StramLocalCluster is started asynchroniously it does not wait for the master thread to terminate -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (APEXCORE-644) get-app-package-operators with parent option does not work
[
https://issues.apache.org/jira/browse/APEXCORE-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov resolved APEXCORE-644.
-
Resolution: Fixed
Fix Version/s: 3.6.0
> get-app-package-operators with parent option does not work
> --
>
> Key: APEXCORE-644
> URL: https://issues.apache.org/jira/browse/APEXCORE-644
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Yatin Chaubal
>Assignee: Sergey Golovko
>Priority: Minor
> Fix For: 3.6.0
>
>
> Issue: get-app-package-operators with -parent option doesnot work
>
> Steps:
> 1) Start dtcli/apex
> 2) Run get-app-package-operators -parent com.datatorrent.demos.pi
> /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> Expected out output: valid JSON
> Actual output:
> {noformat}
> com.datatorrent.stram.cli.ApexCli$CliException:
> /home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
> at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
> at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
> {noformat}
> Reference:
> Without -parent option this work fine
> apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> {
> "operatorClasses": [
> {
> "name": "com.datatorrent.common.util.DefaultDelayOperator",
> "properties": [],
> "portTypeInfo": [
> {
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (APEXCORE-644) get-app-package-operators with parent option does not work
[
https://issues.apache.org/jira/browse/APEXCORE-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-644:
Priority: Minor (was: Major)
> get-app-package-operators with parent option does not work
> --
>
> Key: APEXCORE-644
> URL: https://issues.apache.org/jira/browse/APEXCORE-644
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Yatin Chaubal
>Assignee: Sergey Golovko
>Priority: Minor
>
> Issue: get-app-package-operators with -parent option doesnot work
>
> Steps:
> 1) Start dtcli/apex
> 2) Run get-app-package-operators -parent com.datatorrent.demos.pi
> /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> Expected out output: valid JSON
> Actual output: com.datatorrent.stram.cli.ApexCli$CliException:
> /home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
> at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
> at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
> Reference:
> Without -parent option this work fine
> apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> {
> "operatorClasses": [
> {
> "name": "com.datatorrent.common.util.DefaultDelayOperator",
> "properties": [],
> "portTypeInfo": [
> {
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (APEXCORE-644) get-app-package-operators with parent option does not work
[
https://issues.apache.org/jira/browse/APEXCORE-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-644:
Affects Version/s: (was: 3.5.0)
Description:
Issue: get-app-package-operators with -parent option doesnot work
Steps:
1) Start dtcli/apex
2) Run get-app-package-operators -parent com.datatorrent.demos.pi
/home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
Expected out output: valid JSON
Actual output:
{noformat}
com.datatorrent.stram.cli.ApexCli$CliException:
/home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
at
com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
{noformat}
Reference:
Without -parent option this work fine
apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
{
"operatorClasses": [
{
"name": "com.datatorrent.common.util.DefaultDelayOperator",
"properties": [],
"portTypeInfo": [
{
was:
Issue: get-app-package-operators with -parent option doesnot work
Steps:
1) Start dtcli/apex
2) Run get-app-package-operators -parent com.datatorrent.demos.pi
/home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
Expected out output: valid JSON
Actual output: com.datatorrent.stram.cli.ApexCli$CliException:
/home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
at
com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
Reference:
Without -parent option this work fine
apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
{
"operatorClasses": [
{
"name": "com.datatorrent.common.util.DefaultDelayOperator",
"properties": [],
"portTypeInfo": [
{
> get-app-package-operators with parent option does not work
> --
>
> Key: APEXCORE-644
> URL: https://issues.apache.org/jira/browse/APEXCORE-644
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Yatin Chaubal
>Assignee: Sergey Golovko
>Priority: Minor
>
> Issue: get-app-package-operators with -parent option doesnot work
>
> Steps:
> 1) Start dtcli/apex
> 2) Run get-app-package-operators -parent com.datatorrent.demos.pi
> /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> Expected out output: valid JSON
> Actual output:
> {noformat}
> com.datatorrent.stram.cli.ApexCli$CliException:
> /home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
> at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
> at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
> {noformat}
> Reference:
> Without -parent option this work fine
> apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> {
> "operatorClasses": [
> {
> "name": "com.datatorrent.common.util.DefaultDelayOperator",
> "properties": [],
> "portTypeInfo": [
> {
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (APEXCORE-640) BufferServer - purge crashes when the window purged is the last available
[
https://issues.apache.org/jira/browse/APEXCORE-640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov updated APEXCORE-640:
Priority: Minor (was: Major)
Description:
While working on the unit test of BufferServer saw this behavior,
purge on the last window in the buffer server crashes it.
{noformat}
java.lang.IllegalArgumentException: Invalid slice: offset=2118, length=0
array.length=4096
at com.datatorrent.netlet.util.Slice.(Slice.java:43)
at
com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
at
com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
at
com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
at
com.datatorrent.bufferserver.server.Server.handlePurgeRequest(Server.java:199)
at
com.datatorrent.bufferserver.server.Server.access$1100(Server.java:65)
at
com.datatorrent.bufferserver.server.Server$UnidentifiedClient.onMessage(Server.java:526)
at
com.datatorrent.netlet.AbstractLengthPrependerClient.read(AbstractLengthPrependerClient.java:149)
at com.datatorrent.netlet.AbstractClient.read(AbstractClient.java:104)
at
com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:364)
at
com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
at
com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)
at
com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157)
at
com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156)
at java.lang.Thread.run(Thread.java:745)
{noformat}
was:
While working on the unit test of BufferServer saw this behavior,
purge on the last window in the buffer server crashes it.
java.lang.IllegalArgumentException: Invalid slice: offset=2118, length=0
array.length=4096
at com.datatorrent.netlet.util.Slice.(Slice.java:43)
at
com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
at
com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
at
com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
at
com.datatorrent.bufferserver.server.Server.handlePurgeRequest(Server.java:199)
at
com.datatorrent.bufferserver.server.Server.access$1100(Server.java:65)
at
com.datatorrent.bufferserver.server.Server$UnidentifiedClient.onMessage(Server.java:526)
at
com.datatorrent.netlet.AbstractLengthPrependerClient.read(AbstractLengthPrependerClient.java:149)
at com.datatorrent.netlet.AbstractClient.read(AbstractClient.java:104)
at
com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:364)
at
com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
at
com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)
at
com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157)
at
com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156)
at java.lang.Thread.run(Thread.java:745)
> BufferServer - purge crashes when the window purged is the last available
> -
>
> Key: APEXCORE-640
> URL: https://issues.apache.org/jira/browse/APEXCORE-640
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Sandesh
>Assignee: Sandesh
>Priority: Minor
>
> While working on the unit test of BufferServer saw this behavior,
> purge on the last window in the buffer server crashes it.
> {noformat}
> java.lang.IllegalArgumentException: Invalid slice: offset=2118, length=0
> array.length=4096
> at com.datatorrent.netlet.util.Slice.(Slice.java:43)
> at
> com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
> at
> com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
> at
> com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
> at
> com.datatorrent.bufferserver.server.Server.handlePurgeRequest(Server.java:199)
> at
> com.datatorrent.bufferserver.server.Server.access$1100(Server.java:65)
> at
> com.datatorrent.bufferserver.server.Server$UnidentifiedClient.onMessage(Server.java:526)
> at
> com.datatorrent.netlet.AbstractLengthPrependerClient.read(AbstractLengthPrependerClient.java:149)
> at com.datatorrent.netlet.AbstractClient.read(AbstractClient.java:104)
> at
> com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:364)
> at
>
[jira] [Commented] (APEXCORE-640) BufferServer - purge crashes when the window purged is the last available
[
https://issues.apache.org/jira/browse/APEXCORE-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859652#comment-15859652
]
Vlad Rozov commented on APEXCORE-640:
-
The code in the BufferServer that handles the case when all data published so
far is purged needs to be re-written. The purpose of SerializedData object is
not clear. The exception thrown by the Slice initialization is on purpose, as
SerialziedData object is invalid.
> BufferServer - purge crashes when the window purged is the last available
> -
>
> Key: APEXCORE-640
> URL: https://issues.apache.org/jira/browse/APEXCORE-640
> Project: Apache Apex Core
> Issue Type: Bug
>Reporter: Sandesh
>Assignee: Sandesh
>Priority: Minor
>
> While working on the unit test of BufferServer saw this behavior,
> purge on the last window in the buffer server crashes it.
> {noformat}
> java.lang.IllegalArgumentException: Invalid slice: offset=2118, length=0
> array.length=4096
> at com.datatorrent.netlet.util.Slice.(Slice.java:43)
> at
> com.datatorrent.bufferserver.util.SerializedData.(SerializedData.java:40)
> at
> com.datatorrent.bufferserver.internal.DataList$Block.purge(DataList.java:712)
> at
> com.datatorrent.bufferserver.internal.DataList.purge(DataList.java:192)
> at
> com.datatorrent.bufferserver.server.Server.handlePurgeRequest(Server.java:199)
> at
> com.datatorrent.bufferserver.server.Server.access$1100(Server.java:65)
> at
> com.datatorrent.bufferserver.server.Server$UnidentifiedClient.onMessage(Server.java:526)
> at
> com.datatorrent.netlet.AbstractLengthPrependerClient.read(AbstractLengthPrependerClient.java:149)
> at com.datatorrent.netlet.AbstractClient.read(AbstractClient.java:104)
> at
> com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:364)
> at
> com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
> at
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)
> at
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157)
> at
> com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (APEXCORE-635) Proposal: Manage memory to avoid memory copy and garbage collection
[
https://issues.apache.org/jira/browse/APEXCORE-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859910#comment-15859910
]
Vlad Rozov commented on APEXCORE-635:
-
The end goal of the override BufferServerPublisher.write() in is not clear. Why
is the override necessary? It is also not clear why Kryo (that is a library
designed to do efficient serialization and de-serialization) will not match the
proposed mechanism. If the goal is to serialize Collections (list in the
example) using Kryo, Kryo has CollectionsSerializer and will (should?) do
something similar to what is proposed. IMO, it will be much better not to even
use Kryo when Tuples are strongly typed and implement StreamCodec without using
Kryo. Default StreamCodec is provided to handle cases when Tuples are not
strongly typed, in which case it is required to use reflection. The same will
apply to unchecked Collections. (n) till all those issues are resolved.
> Proposal: Manage memory to avoid memory copy and garbage collection
> ---
>
> Key: APEXCORE-635
> URL: https://issues.apache.org/jira/browse/APEXCORE-635
> Project: Apache Apex Core
> Issue Type: Wish
>Reporter: bright chen
>Assignee: bright chen
>
> Manage memory to avoid memory copy and garbage collection
> The aim of this proposal is to reuse the memory to avoid the garbage
> collection and avoid unnecessary memory copy to increase the performance. In
> this proposal the term serde means serialization and deserialization. It’s
> same as codec.
> Currently, apex by default use DefaultStatefulStreamCodec for serde, which
> extends Kryo and optimize it by replace class by class id. And application
> developer can optimize serializer by implement interface StreamCodec.
> First, let’s look into the default codec DefaultStatefulStreamCodec. It
> basically optimize serde by replace class name by class id as my
> understanding. And the state information only send before sending first
> tuple, it’s kind like configuration for serde. So I suggest to separate this
> feature from serde. The benefit is the customized serde can still use this
> feature. And the kryo have some limitation which I’ll state later.
> Second, Let’s look at the customized serde. Let’s stand from application
> developer point of view and look at how to implement StreamCodec. I take a
> simple tuple List as example.
> The first solution is use kryo. This is basically same as apex default codec.
> The second solution is implement StreamCodec for String and List, and
> ListSerde delegate String to StringSerde. The benefit of this solution is the
> StringSerde ListSerde can be reused. The problem is there need a lot of
> temporary memory and memory copy. Following is the sample implement.
> Class StringSerde {
> Slice toByteArray(String o) {
> byte[] b = o.getBytes(“UTF8”); // new bytes
> byte[] b1 = new byte[b1.length + 4]; // new bytes
> set the length of the string at the first 4 bytes
> System.arrayCopy(b, 0, b1, 4, b.length); //copy bytes
> return new Slice(b1);
> }
> class ListSerde {
> StreamCodec itemSerde; //the serde for serialize/deserialize item
> Slice toByteArray(List list) {
> Slice[] itemSlices = new Slice[list.size()];
> int size = 0;
> int index = 0;
> for(T item : list) {
> Slice slice = itemSerde.toByteArray(item);
> size += slice.length;
> itemSlices[index++] = slice;
> }
> byte[] b = new byte[size+4]; //allocated the memory
> set the length of the list at the first 4 bytes
> copy the data from itemSlices
> return new Slice(b);
> }
> }
>
> from above code, we can see that around 2 times of required memory were
> allocated and data copied twice( one copy maybe special to string, but
> another copy is mandatory). And when bytes written to the socket, all
> allocated memory can’t be reused but need to be garbage collected.
> The above tuple only have two levels, if the tuple have n level, n times of
> required memory would be allocated and n-1 time of data copy is required.
> The third solution could be allocate memory and then pass the memory and
> offset to item serde. There are some problems for this solution:
> How to pass the memory from caller? As our previous interface only pass the
> object but no way to pass memory. So the pass of memory will depends on
> implementation.
> Another big problem of this solution is it hard to reallocate proper
> memory(For this special case, it probably can allocate 2 times of all string
> length. ). And the memory allocated more than required would be wasted until
> data send to the socket(or allocate exact memory and copy the data to avoid
> waste memory). And the code also need to handle the case if
[jira] [Assigned] (APEXCORE-644) get-app-package-operators with parent option does not work
[
https://issues.apache.org/jira/browse/APEXCORE-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vlad Rozov reassigned APEXCORE-644:
---
Assignee: Sergey Golovko
> get-app-package-operators with parent option does not work
> --
>
> Key: APEXCORE-644
> URL: https://issues.apache.org/jira/browse/APEXCORE-644
> Project: Apache Apex Core
> Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Yatin Chaubal
>Assignee: Sergey Golovko
>
> Issue: get-app-package-operators with -parent option doesnot work
>
> Steps:
> 1) Start dtcli/apex
> 2) Run get-app-package-operators -parent com.datatorrent.demos.pi
> /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> Expected out output: valid JSON
> Actual output: com.datatorrent.stram.cli.ApexCli$CliException:
> /home/hduser/tf2jan/com.datatorrent.demos.pi does not match any file
> at com.datatorrent.stram.cli.ApexCli.expandFileName(ApexCli.java:918)
> at com.datatorrent.stram.cli.ApexCli.access$000(ApexCli.java:152)
> at
> com.datatorrent.stram.cli.ApexCli$GetAppPackageOperatorsCommand.execute(ApexCli.java:3827)
> at com.datatorrent.stram.cli.ApexCli$3.run(ApexCli.java:1492)
> Reference:
> Without -parent option this work fine
> apex> get-app-package-operators /home/hduser/tf2jan/apa/pi-demo-3.4.0.apa
> {
> "operatorClasses": [
> {
> "name": "com.datatorrent.common.util.DefaultDelayOperator",
> "properties": [],
> "portTypeInfo": [
> {
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
Vlad Rozov created APEXCORE-510: --- Summary: Enforce DefaultOutputPort.emit() or Sink.put() thread affinity Key: APEXCORE-510 URL: https://issues.apache.org/jira/browse/APEXCORE-510 Project: Apache Apex Core Issue Type: Improvement Reporter: Vlad Rozov Apex platform assumes that an operator interacts with the platform using the dedicated operator thread. Currently, operators may create worker threads and emit tuples from a worker thread. This leads to undefined behavior and hard to find bugs, so it should be possible to enforce that DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
[ https://issues.apache.org/jira/browse/APEXCORE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426932#comment-15426932 ] Vlad Rozov commented on APEXCORE-510: - - Changing emit() to be final breaks backward compatibility. - It should be possible to use DefaultOutputPort.setup() to initialize Thread object instead of relying on reflection. It may be necessary to see if DefaultOutputPort.setup() is called on the correct thread and if not, why. - It is important to benchmark the change. > Enforce DefaultOutputPort.emit() or Sink.put() thread affinity > -- > > Key: APEXCORE-510 > URL: https://issues.apache.org/jira/browse/APEXCORE-510 > Project: Apache Apex Core > Issue Type: Improvement >Reporter: Vlad Rozov > > Apex platform assumes that an operator interacts with the platform using the > dedicated operator thread. Currently, operators may create worker threads and > emit tuples from a worker thread. This leads to undefined behavior and hard > to find bugs, so it should be possible to enforce that > DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
