[jira] [Updated] (APEXCORE-494) Window id of downstream operator is not moving after dynamic partition of upstream operator.

[Chaitanya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaitanya updated APEXCORE-494:
---
Fix Version/s: 3.5.0

> Window id of downstream operator is not moving after dynamic partition of 
> upstream operator.
> 
>
> Key: APEXCORE-494
> URL: https://issues.apache.org/jira/browse/APEXCORE-494
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chaitanya
>Assignee: Chaitanya
> Fix For: 3.5.0
>
> Attachments: PhysicalDAGAfterKafkaRepartition.png, 
> PhysicalDAGBeforeKafkaRepartition.png, 
> PhysicalOperatorAfterKafkaRepartition.png
>
>
> Created the following application to reproduce this issue.
> Application has the following DAG:
>KafkaSinglePortStringInputOperator(Input) -> ConsoleOutputOperator(Output)
>
>I launched the application with below configuration:
> Kafka topic created with single partition and replication factor as 1.
> Partition Strategy: ONE_TO_ONE
>Launched the application successfully. After some time, I increased the 
> topic partitions to 2. After re-partition, the window of down stream operator 
> is not moving. By looking into the app Physical DAG, it looks like there is 
> an issue in construction of Physical DAG after re-partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXCORE-494) Window id of downstream operator is not moving after dynamic partition of upstream operator.

[Chaitanya (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chaitanya resolved APEXCORE-494.

Resolution: Fixed

> Window id of downstream operator is not moving after dynamic partition of 
> upstream operator.
> 
>
> Key: APEXCORE-494
> URL: https://issues.apache.org/jira/browse/APEXCORE-494
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chaitanya
>Assignee: Chaitanya
> Fix For: 3.5.0
>
> Attachments: PhysicalDAGAfterKafkaRepartition.png, 
> PhysicalDAGBeforeKafkaRepartition.png, 
> PhysicalOperatorAfterKafkaRepartition.png
>
>
> Created the following application to reproduce this issue.
> Application has the following DAG:
>KafkaSinglePortStringInputOperator(Input) -> ConsoleOutputOperator(Output)
>
>I launched the application with below configuration:
> Kafka topic created with single partition and replication factor as 1.
> Partition Strategy: ONE_TO_ONE
>Launched the application successfully. After some time, I increased the 
> topic partitions to 2. After re-partition, the window of down stream operator 
> is not moving. By looking into the app Physical DAG, it looks like there is 
> an issue in construction of Physical DAG after re-partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-core pull request #362: APEXCORE-494 Fixed the dynamic partition issue ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/362


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXCORE-494) Window id of downstream operator is not moving after dynamic partition of upstream operator.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425930#comment-15425930
 ] 

ASF GitHub Bot commented on APEXCORE-494:
-

Github user asfgit closed the pull request at:

https://github.com/apache/apex-core/pull/362


> Window id of downstream operator is not moving after dynamic partition of 
> upstream operator.
> 
>
> Key: APEXCORE-494
> URL: https://issues.apache.org/jira/browse/APEXCORE-494
> Project: Apache Apex Core
>  Issue Type: Bug
>Reporter: Chaitanya
>Assignee: Chaitanya
> Attachments: PhysicalDAGAfterKafkaRepartition.png, 
> PhysicalDAGBeforeKafkaRepartition.png, 
> PhysicalOperatorAfterKafkaRepartition.png
>
>
> Created the following application to reproduce this issue.
> Application has the following DAG:
>KafkaSinglePortStringInputOperator(Input) -> ConsoleOutputOperator(Output)
>
>I launched the application with below configuration:
> Kafka topic created with single partition and replication factor as 1.
> Partition Strategy: ONE_TO_ONE
>Launched the application successfully. After some time, I increased the 
> topic partitions to 2. After re-partition, the window of down stream operator 
> is not moving. By looking into the app Physical DAG, it looks like there is 
> an issue in construction of Physical DAG after re-partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXCORE-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425917#comment-15425917
 ] 

Sanjay M Pujare commented on APEXCORE-510:
--

One way to do this as follows:

- In DefaultOutputPort we add a data field 'operatorThread' to save a Thread 
object.
- In DefaultOutputPort.emit() we verify that Thread.currentThread() == 
operatorThread else throw an exception. Either make emit() final or split emit 
into 2 functions - one final one (having the check) calling the other 
overridable method.
- In the engine in Node.setup(), add a call to operator.setupThreadObject() 
after operator.setup()
- Operator.setupThreadObject() uses Java reflection to find all data member 
objects in the current class of type DefaultOutputPort and sets the 
operatorThread object in each of those objects to Thread.currentThread()

> Enforce DefaultOutputPort.emit() or Sink.put() thread affinity
> --
>
> Key: APEXCORE-510
> URL: https://issues.apache.org/jira/browse/APEXCORE-510
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Vlad Rozov
>
> Apex platform assumes that an operator interacts with the platform using the 
> dedicated operator thread. Currently, operators may create worker threads and 
> emit tuples from a worker thread. This leads to undefined behavior and hard 
> to find bugs, so it should be possible to enforce that 
> DefaultOutputPort.emit() and/or Sink.put() are called on the operator thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2142) High-level API window support

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2142:
-
Labels: roadmap  (was: )

> High-level API window support
> -
>
> Key: APEXMALHAR-2142
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2142
> Project: Apache Apex Malhar
>  Issue Type: New Feature
>Reporter: Siyuan Hua
>Assignee: Siyuan Hua
>  Labels: roadmap
> Fix For: 3.5.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXCORE-510) Enforce DefaultOutputPort.emit() or Sink.put() thread affinity

[Vlad Rozov (JIRA)]

Vlad Rozov created APEXCORE-510:
---

 Summary: Enforce DefaultOutputPort.emit() or Sink.put() thread 
affinity
 Key: APEXCORE-510
 URL: https://issues.apache.org/jira/browse/APEXCORE-510
 Project: Apache Apex Core
  Issue Type: Improvement
Reporter: Vlad Rozov


Apex platform assumes that an operator interacts with the platform using the 
dedicated operator thread. Currently, operators may create worker threads and 
emit tuples from a worker thread. This leads to undefined behavior and hard to 
find bugs, so it should be possible to enforce that DefaultOutputPort.emit() 
and/or Sink.put() are called on the operator thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXCORE-509) OperatorContext.JVM_OPTIONS should preserve order

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-509:

Labels: newbie  (was: )

> OperatorContext.JVM_OPTIONS should preserve order
> -
>
> Key: APEXCORE-509
> URL: https://issues.apache.org/jira/browse/APEXCORE-509
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Vlad Rozov
>Priority: Minor
>  Labels: newbie
>
> While majority of JVM options don't require any particular order, few -XX 
> options are order sensitive. For example +UnlockCommercialFeatures must 
> precede +FlightRecorder or JVM will fail to start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Malhar Newbie / Beginner tasks

[Thomas Weise]

Hi Brendan,

Welcome to the community! I took this as opportunity to put the ticket link
onto our community page and also added a few more tickets that I thought
would be good fit:

https://issues.apache.org/jira/issues/?jql=project%20in%20(APEXCORE%2C%20APEXMALHAR)%20and%20labels%20%3D%20newbie

You can browse the full list and find any other ticket that isn't assigned
of course.

Thanks,
Thomas



On Tue, Aug 16, 2016 at 2:52 PM, Herger, Brendan <
brendan.her...@capitalone.com> wrote:

> Hey,
>
> There is only one Newbie / Beginner task for Malhar. Are there any other
> tasks that could be labeled Newbie / Beginner?
>
>
> Thanks,
> Brendan Herger
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

[jira] [Updated] (APEXCORE-508) Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov updated APEXCORE-508:

Labels: documentation  (was: )

> Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS
> -
>
> Key: APEXCORE-508
> URL: https://issues.apache.org/jira/browse/APEXCORE-508
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Vlad Rozov
>Priority: Minor
>  Labels: documentation
>
> Provide better documentation on DAGContext.CONTAINER_JVM_OPTIONS and 
> OperatorContext.JVM_OPTIONS. Currently, it is not clear which one to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (APEXCORE-508) Document difference between JVM_OPTIONS and CONTAINER_JVM_OPTIONS

[Vlad Rozov (JIRA)]

Vlad Rozov created APEXCORE-508:
---

 Summary: Document difference between JVM_OPTIONS and 
CONTAINER_JVM_OPTIONS
 Key: APEXCORE-508
 URL: https://issues.apache.org/jira/browse/APEXCORE-508
 Project: Apache Apex Core
  Issue Type: Improvement
Reporter: Vlad Rozov
Priority: Minor


Provide better documentation on DAGContext.CONTAINER_JVM_OPTIONS and 
OperatorContext.JVM_OPTIONS. Currently, it is not clear which one to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-site pull request #47: Add candidate JIRA list link to contributing sec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/apex-site/pull/47


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (APEXCORE-448) Make operator name available in OperatorContext

[Vlad Rozov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXCORE-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vlad Rozov resolved APEXCORE-448.
-
   Resolution: Fixed
Fix Version/s: 3.5.0

> Make operator name available in OperatorContext
> ---
>
> Key: APEXCORE-448
> URL: https://issues.apache.org/jira/browse/APEXCORE-448
> Project: Apache Apex Core
>  Issue Type: Improvement
>Reporter: Chandni Singh
>Assignee: Chandni Singh
> Fix For: 3.5.0
>
>
> Need name of the logical operator in the OperatorContext which can be used by 
> WindowDataManager to create a unique path per logical operator .



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2149) Add unittests for FunctionOperator

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved APEXMALHAR-2149.
--
Resolution: Fixed

> Add unittests for FunctionOperator
> --
>
> Key: APEXMALHAR-2149
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2149
> Project: Apache Apex Malhar
>  Issue Type: Test
>Reporter: Shunxin Lu
>Assignee: Shunxin Lu
>
> Add unittests for FunctionOperator (Map, FlatMap, Filter).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2162) Enable unit tests of 0.9.0 Kafka Output operator

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2162:
-
Labels: newbie  (was: )

> Enable unit tests of 0.9.0 Kafka Output operator
> 
>
> Key: APEXMALHAR-2162
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2162
> Project: Apache Apex Malhar
>  Issue Type: Test
>Reporter: Sandesh
>  Labels: newbie
>
> Kafka output operator(0.9)'s unit tests are ignored. 
> Reason: Running the Kafka test server creates a lot of logs, which exhausts 
> the limit of Travis's 4mb limit.
> Tests pass locally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2159) Add Enum Support in CSV Parser

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2159:
-
Labels: newbie  (was: )

> Add Enum Support in CSV Parser
> --
>
> Key: APEXMALHAR-2159
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2159
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: shubham pathak
>Priority: Minor
>  Labels: newbie
>
> Support for Enum should be added in CSV Parser to convert String to Enum in 
> POJO. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2178) Unnecessary byte array copy in KryoSerializableStreamCodec

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2178:
-
Labels: newbie  (was: )

> Unnecessary byte array copy in KryoSerializableStreamCodec
> --
>
> Key: APEXMALHAR-2178
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2178
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Vlad Rozov
>  Labels: newbie
>
> {noformat}
>   public Slice toByteArray(T info)
>   {
> ByteArrayOutputStream os = new ByteArrayOutputStream();
> Output output = new Output(os);
> kryo.writeClassAndObject(output, info);
> output.flush();
> return new Slice(os.toByteArray(), 0, os.toByteArray().length);
>   }
> {noformat}
> It is not necessary to call os.toByteArray().length as it will result in 
> duplicate copy of the byte array.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2175) POJOUtils is considering java reserved words as fields in the expression

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2175:
-
Labels: newbie  (was: )

> POJOUtils is considering java reserved words as fields in the expression
> 
>
> Key: APEXMALHAR-2175
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2175
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Yogi Devendra
>Priority: Minor
>  Labels: newbie
>
> For example, if expression given is 
> {code}
> true
> {code}
> or
> {code}
> toBoolean("1") == true
> {code}
> it treats the above expression as a field in the object.
> As a test case you might use examples/tutorial/filter application with 
> condition true.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2173) add a constraint validation for subject in JMSBase

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2173:
-
Labels: newbie  (was: )

> add a constraint validation for subject in JMSBase
> --
>
> Key: APEXMALHAR-2173
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2173
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Sanjay M Pujare
>Priority: Minor
>  Labels: newbie
>
> In com/datatorrent/lib/io/jms/JMSBase add NotNull constraint check for the 
> member "subject" 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2183) Add user document for CsvFormatter operator

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2183:
-
Labels: newbie  (was: )

> Add user document for CsvFormatter operator
> ---
>
> Key: APEXMALHAR-2183
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2183
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: Venkatesh Kottapalli
>  Labels: newbie
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Add user documentation for CsvFormatter operator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2134) Catch NullPointerException if some Kafka partition has no leader broker

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2134:
-
Labels: newbie  (was: )

> Catch NullPointerException if some Kafka partition has no leader broker
> ---
>
> Key: APEXMALHAR-2134
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2134
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>  Labels: newbie
> Fix For: 3.5.0, 3.4.1
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Kafka partition could have no leader broker some time and we need to catch 
> exception and skip that partition for the time until new leader is elected
> Here is the exception we see in the stacktrace
> 2016-07-05 14:00:46,087 ERROR kafka.SimpleKafkaConsumer 
> (SimpleKafkaConsumer.java:run(481)) - Exception {}
> java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.monitorMetadata(SimpleKafkaConsumer.java:511)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.run(SimpleKafkaConsumer.java:477)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-05 14:01:15,999 ERROR kafka.SimpleKafkaConsumer 
> (SimpleKafkaConsumer.java:run(481)) - Exception {}
> java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.monitorMetadata(SimpleKafkaConsumer.java:511)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.run(SimpleKafkaConsumer.java:477)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2134) Catch NullPointerException if some Kafka partition has no leader broker

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated APEXMALHAR-2134:
-
Assignee: (was: Siyuan Hua)

> Catch NullPointerException if some Kafka partition has no leader broker
> ---
>
> Key: APEXMALHAR-2134
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2134
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
> Fix For: 3.5.0, 3.4.1
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Kafka partition could have no leader broker some time and we need to catch 
> exception and skip that partition for the time until new leader is elected
> Here is the exception we see in the stacktrace
> 2016-07-05 14:00:46,087 ERROR kafka.SimpleKafkaConsumer 
> (SimpleKafkaConsumer.java:run(481)) - Exception {}
> java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.monitorMetadata(SimpleKafkaConsumer.java:511)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.run(SimpleKafkaConsumer.java:477)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-07-05 14:01:15,999 ERROR kafka.SimpleKafkaConsumer 
> (SimpleKafkaConsumer.java:run(481)) - Exception {}
> java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
> at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.monitorMetadata(SimpleKafkaConsumer.java:511)
> at 
> com.datatorrent.contrib.kafka.SimpleKafkaConsumer$MetaDataMonitorTask.run(SimpleKafkaConsumer.java:477)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Malhar 3.5.0 release

[Thomas Weise]

Friendly reminder.. There are still a few unresolved tickets for 3.5:

https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%203.5.0%20AND%20project%20%3D%20APEXMALHAR%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC

Please remove the fix version unless you expect to complete the work in the
next few days.

And we need to complete the PR for high level API:

https://issues.apache.org/jira/browse/APEXMALHAR-2142

Thanks,
Thomas


On Fri, Aug 5, 2016 at 2:12 AM, Chaitanya Chebolu  wrote:

> APEXMALHAR-2100. PR is open: https://github.com/apache/
> apex-malhar/pull/330
>
> Regards,
> Chaitanya
>
> On Fri, Aug 5, 2016 at 1:10 PM, Priyanka Gugale 
> wrote:
>
> > APEXMALHAR-2171
> > PR is open: https://github.com/apache/apex-malhar/pull/358
> >
> > -Priyanka
> >
> > On Fri, Aug 5, 2016 at 12:32 PM, Yogi Devendra <
> > devendra.vyavah...@gmail.com
> > > wrote:
> >
> > > Fix for APEXMALHAR-2176
> > > PR open: https://github.com/apache/apex-malhar/pull/361
> > >
> > > ~ Yogi
> > >
> > > On 1 August 2016 at 11:59, Chinmay Kolhatkar 
> > > wrote:
> > >
> > > > APEXMALHAR-2128 is already done. Somehow the status did not go to
> > > resolved.
> > > > Changed it now.
> > > >
> > > > On Fri, Jul 29, 2016 at 1:53 PM, Yogi Devendra <
> > > > devendra.vyavah...@gmail.com
> > > > > wrote:
> > > >
> > > > > APEXMALHAR-2116
> > > > > https://github.com/apache/apex-malhar/pull/326
> > > > >
> > > > > @Ram
> > > > > Your review comments are incorporated. Could you please have a look
> > at
> > > > > this?
> > > > >
> > > > > ~ Yogi
> > > > >
> > > > > On 29 July 2016 at 13:49, Thomas Weise 
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I would suggest to target the 3.5.0 release with an RC out
> perhaps
> > 2
> > > > > weeks
> > > > > > from now, as there is a good amount of work that we should make
> > > > available
> > > > > > to the users:
> > > > > >
> > > > > >
> > > > > >
> > > > > https://issues.apache.org/jira/browse/APEXMALHAR/
> > > > fixforversion/12335815/?selectedTab=com.atlassian.
> > > > jira.jira-projects-plugin:version-issues-panel
> > > > > >
> > > > > > I also think that the following should go in:
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2128
> > > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2142
> > > > > > https://issues.apache.org/jira/browse/APEXMALHAR-2063
> > > > > >
> > > > > > Any other candidates?
> > > > > >
> > > > > > There are quite a few other PRs open:
> > > > > >
> > > > > > https://github.com/apache/apex-malhar/pulls
> > > > > >
> > > > > > Would be good to complete some more reviews for this release.
> > > > > >
> > > > > > Thanks,
> > > > > > Thomas
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Updated] (APEXMALHAR-2154) Update kafka 0.9 input operator to use new CheckpointNotificationListener

[Siyuan Hua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyuan Hua updated APEXMALHAR-2154:
---
Fix Version/s: (was: 3.5.0)

> Update kafka 0.9 input operator to use new CheckpointNotificationListener
> -
>
> Key: APEXMALHAR-2154
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2154
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Siyuan Hua
>Assignee: Siyuan Hua
>
> CheckpointListener interface has been deprecated. We should upgrade the 
> operator to use the new one



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (APEXMALHAR-2154) Update kafka 0.9 input operator to use new CheckpointNotificationListener

[Siyuan Hua (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siyuan Hua updated APEXMALHAR-2154:
---
Assignee: Chaitanya  (was: Siyuan Hua)

> Update kafka 0.9 input operator to use new CheckpointNotificationListener
> -
>
> Key: APEXMALHAR-2154
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2154
> Project: Apache Apex Malhar
>  Issue Type: Improvement
>Reporter: Siyuan Hua
>Assignee: Chaitanya
>
> CheckpointListener interface has been deprecated. We should upgrade the 
> operator to use the new one



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (APEXMALHAR-2135) Upgrade Kafka 0.8 input operator to support 0.8.2 client

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/APEXMALHAR-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise resolved APEXMALHAR-2135.
--
Resolution: Fixed

> Upgrade Kafka 0.8 input operator to support 0.8.2 client
> 
>
> Key: APEXMALHAR-2135
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2135
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: Siyuan Hua
>Assignee: Thomas Weise
> Fix For: 3.5.0
>
>
> Right now, if you are using 0.8.2 client with 0.8 kafka inputoperator.
> You will get exception:
> *java.lang.NoSuchMethodError:
> > kafka.cluster.Broker.getConnectionString()Ljava/lang/String;*
> >
> >
> >
> > *at
> > com.datatorrent.contrib.kafka.KafkaMetadataUtil.getBrokers(KafkaMetadataUtil.java:114)*
> >
> > *at
> > com.datatorrent.contrib.kafka.KafkaConsumer.initBrokers(KafkaConsumer.java:131)*
> >
> > *at
> > com.datatorrent.contrib.kafka.AbstractKafkaInputOperator.definePartitions(AbstractKafkaInputOperator.java:488)*
> >
> > 
> We should support both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2182) benchmark for spillable data structure and storage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424947#comment-15424947
 ] 

ASF GitHub Bot commented on APEXMALHAR-2182:


Github user brightchen closed the pull request at:

https://github.com/apache/apex-malhar/pull/370


> benchmark for spillable data structure  and storage
> ---
>
> Key: APEXMALHAR-2182
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2182
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: bright chen
>Assignee: bright chen
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (APEXMALHAR-2182) benchmark for spillable data structure and storage

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424948#comment-15424948
 ] 

ASF GitHub Bot commented on APEXMALHAR-2182:


GitHub user brightchen reopened a pull request:

https://github.com/apache/apex-malhar/pull/370

APEXMALHAR-2182 #resolve #comment benchmark for spillable data struct…

…ure and storage

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2182

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/370.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #370


commit ea4f7a434ee97c1079bbf9432e9443cb203aa502
Author: brightchen 
Date:   2016-08-12T23:43:45Z

APEXMALHAR-2182 #resolve #comment benchmark for spillable data structure 
and storage




> benchmark for spillable data structure  and storage
> ---
>
> Key: APEXMALHAR-2182
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2182
> Project: Apache Apex Malhar
>  Issue Type: Bug
>Reporter: bright chen
>Assignee: bright chen
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #370: APEXMALHAR-2182 #resolve #comment benchmark f...

[brightchen]

GitHub user brightchen reopened a pull request:

https://github.com/apache/apex-malhar/pull/370

APEXMALHAR-2182 #resolve #comment benchmark for spillable data struct…

…ure and storage

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2182

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/370.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #370


commit ea4f7a434ee97c1079bbf9432e9443cb203aa502
Author: brightchen 
Date:   2016-08-12T23:43:45Z

APEXMALHAR-2182 #resolve #comment benchmark for spillable data structure 
and storage




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXMALHAR-2100) Development of Inner Join Operator using Spillable Datastructures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424013#comment-15424013
 ] 

ASF GitHub Bot commented on APEXMALHAR-2100:


GitHub user chaithu14 reopened a pull request:

https://github.com/apache/apex-malhar/pull/330

APEXMALHAR-2100 Implementation of Inner Join operator



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chaithu14/incubator-apex-malhar 
APEXMALHAR-2100-InnerJoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #330


commit 363287a2cca2a0a5dbb9096b4d971719e71d5342
Author: Chaitanya 
Date:   2016-08-17T07:08:23Z

APEXMALHAR-2100 Implementation of Inner Join operator




> Development of Inner Join Operator using Spillable Datastructures
> -
>
> Key: APEXMALHAR-2100
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2100
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: Chaitanya
>Assignee: Chaitanya
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #330: APEXMALHAR-2100 Implementation of Inner Join ...

[chaithu14]

GitHub user chaithu14 reopened a pull request:

https://github.com/apache/apex-malhar/pull/330

APEXMALHAR-2100 Implementation of Inner Join operator



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chaithu14/incubator-apex-malhar 
APEXMALHAR-2100-InnerJoin

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/apex-malhar/pull/330.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #330


commit 363287a2cca2a0a5dbb9096b4d971719e71d5342
Author: Chaitanya 
Date:   2016-08-17T07:08:23Z

APEXMALHAR-2100 Implementation of Inner Join operator




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (APEXMALHAR-2100) Development of Inner Join Operator using Spillable Datastructures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424008#comment-15424008
 ] 

ASF GitHub Bot commented on APEXMALHAR-2100:


Github user chaithu14 closed the pull request at:

https://github.com/apache/apex-malhar/pull/330


> Development of Inner Join Operator using Spillable Datastructures
> -
>
> Key: APEXMALHAR-2100
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2100
> Project: Apache Apex Malhar
>  Issue Type: Task
>Reporter: Chaitanya
>Assignee: Chaitanya
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] apex-malhar pull request #330: APEXMALHAR-2100 Implementation of Inner Join ...

[chaithu14]

Github user chaithu14 closed the pull request at:

https://github.com/apache/apex-malhar/pull/330


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---