Re: [jira] [Commented] (APEXMALHAR-2537) POM bundling issue in apex-malhar/kafka/kafka010
Hi Srikanth,
Below dependency also doesn't work. Because the abstract operator is
under kafka-common module.
org.apache.apex
malhar-kafka
3.8.0-SNAPSHOT
*
*
You need to include the dependency as follows:
org.apache.apex
malhar-kafka
3.8.0-SNAPSHOT
org.apache.apex
malhar-library
(or)
org.apache.apex
malhar-kafka
3.8.0-SNAPSHOT
Regards,
On Fri, Aug 11, 2017 at 6:26 AM, Vinay Bangalore Srikanth (JIRA) <
j...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/APEXMALHAR-2537?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&
> focusedCommentId=16122624#comment-16122624 ]
>
> Vinay Bangalore Srikanth commented on APEXMALHAR-2537:
> --
>
> Update:
>
> org.apache.apex
> malhar-kafka010
>3.8.0-SNAPSHOT
>works.
>
>
> org.apache.apex
> malhar-kafka010
> 3.8.0-SNAPSHOT
>
>
> *
> *
>
>
> does not work
>
>
>
> org.apache.apex
> malhar-kafka
> 3.8.0-SNAPSHOT
>
>
> *
> *
>
>
>works!
>
> Just wondering why this difference between two kafka versions.
>
> > POM bundling issue in apex-malhar/kafka/kafka010
> >
> >
> > Key: APEXMALHAR-2537
> > URL: https://issues.apache.org/
> jira/browse/APEXMALHAR-2537
> > Project: Apache Apex Malhar
> > Issue Type: Bug
> >Reporter: Vinay Bangalore Srikanth
> >
> > This error is seen in the file -
> > *_org.apache.apex.malhar.kafka.AbstractKafkaInputOperator
> cannot be resolved. It is indirectly referenced from required .class files_*
> > Even though, I have included this in the pom.xml
> >
> > org.apache.apex
> > malhar-kafka010
> > 3.8.0-SNAPSHOT
> >
> >
> > *
> > *
> >
> >
> >
> > I had to add this explicitly to resolve -
> >
> > org.apache.apex
> > malhar-kafka-common
> > 3.8.0-SNAPSHOT
> >
> >
> > org.apache.kafka
> > kafka_2.11
> >
> >
> >
> > However, I don't see such issue with -
> >
> > org.apache.apex
> > malhar-kafka
> > ${malhar.version}
> >
> >
> > *
> > *
> >
> >
> >
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>
--
*Chaitanya*
Software Engineer
E: chaita...@datatorrent.com | Twitter: @chaithu1403
www.datatorrent.com | apex.apache.org
[jira] [Commented] (APEXMALHAR-2034) Avro File Input Operator
[ https://issues.apache.org/jira/browse/APEXMALHAR-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124191#comment-16124191 ] Saumya Mohan commented on APEXMALHAR-2034: -- Hi Thomas, this JIRA is just the first step toward improving the Avro input functionality. As part of this JIRA we're creating a module to encapsulate Avro Container File -> Generic Record -> POJO transformation which user can use directly. Further enhancements will take place as part of separate JIRAs. > Avro File Input Operator > > > Key: APEXMALHAR-2034 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2034 > Project: Apache Apex Malhar > Issue Type: New Feature >Reporter: devendra tagare >Assignee: Saumya Mohan > > This operator would extend the AbstractFileInputOperator to read Avro > Container files. > Input would be an Avro Container File. > Output would be a GenericRecord. > There would be 2 additional optional ports, > 1.FilePort - for completed files. > 2.FailedRecordsPort - this will capture fileName,Offset & error message as a > string for handling by a downstream operator. > This operator can be used in isolation or with the AvroToPojo operator to > read an Avro record and convert it to a POJO. > - > This JIRA is used to create a Module on top of AvroFileInputOperator and > AvroToPojo operator. The stream between the two operators will be set to > CONTAINER_LOCAL which is required as Avro objects are not serialized by Kryo. > This will help users to directly use the module which has the locality set to > CONTAINER_LOCAL. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (APEXCORE-737) AppMaster does not shut down because numRequestedContainers becomes negative
[ https://issues.apache.org/jira/browse/APEXCORE-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124245#comment-16124245 ] ASF GitHub Bot commented on APEXCORE-737: - vrozov closed pull request #535: APEXCORE-737 Increment numRequestedContainers instead of decrementing for an already allocated container URL: https://github.com/apache/apex-core/pull/535 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > AppMaster does not shut down because numRequestedContainers becomes negative > > > Key: APEXCORE-737 > URL: https://issues.apache.org/jira/browse/APEXCORE-737 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare > Original Estimate: 24h > Remaining Estimate: 24h > > A previous fix (for APEXCORE-624) has introduced a regression where > numRequestedContainers inside StreamingAppMasterService.execute() becomes > negative because instead of incrementing numRequestedContainers it is being > decremented under certain conditions as follows: > While processing the YARN response of container allocation/release request, > numRequestContainers is decremented by newAllocatedContainers.size() > (correctly). However in case of (alreadyAllocated == true) the decrement > should be undone by incrementing numRequestContainers (i.e. > numRequestContainers++). However the current code is decrementing again (i.e. > numRequestContainers--) on line 886 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (APEXCORE-777) Application Master may not shutdown due to incorrect numRequesteContainers counting
[ https://issues.apache.org/jira/browse/APEXCORE-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-777: Description: Consider a scenario where App master requests a container from Yarn (numRequestedContainers = 1). There is not enough resources and the request timeouts. My understanding is that App master will re-request it again but the number of requested containers will not change (one newly requested, one removed). Let's assume that App master, by the time Yarn responds back decides that it does not need any. If Yarn responds with one allocated containers, numRequestedContainers will go to 0 (correct), but Yarn may respond back with 2 allocated containers if by the time App Master sends the second request it already allocated a container requested in the original request (the one that timeouted) as Yarn does not guarantee that removed request is fullfilled (see Yarn doc). Will not in this case numRequestedContainers be -1 due to the bulk decrement? > Application Master may not shutdown due to incorrect numRequesteContainers > counting > --- > > Key: APEXCORE-777 > URL: https://issues.apache.org/jira/browse/APEXCORE-777 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Vlad Rozov >Priority: Minor > > Consider a scenario where App master requests a container from Yarn > (numRequestedContainers = 1). There is not enough resources and the request > timeouts. My understanding is that App master will re-request it again but > the number of requested containers will not change (one newly requested, one > removed). Let's assume that App master, by the time Yarn responds back > decides that it does not need any. If Yarn responds with one allocated > containers, numRequestedContainers will go to 0 (correct), but Yarn may > respond back with 2 allocated containers if by the time App Master sends the > second request it already allocated a container requested in the original > request (the one that timeouted) as Yarn does not guarantee that removed > request is fullfilled (see Yarn doc). Will not in this case > numRequestedContainers be -1 due to the bulk decrement? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (APEXCORE-737) AppMaster does not shut down because numRequestedContainers becomes negative
[ https://issues.apache.org/jira/browse/APEXCORE-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov resolved APEXCORE-737. - Resolution: Fixed Fix Version/s: 3.7.0 > AppMaster does not shut down because numRequestedContainers becomes negative > > > Key: APEXCORE-737 > URL: https://issues.apache.org/jira/browse/APEXCORE-737 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.7.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > A previous fix (for APEXCORE-624) has introduced a regression where > numRequestedContainers inside StreamingAppMasterService.execute() becomes > negative because instead of incrementing numRequestedContainers it is being > decremented under certain conditions as follows: > While processing the YARN response of container allocation/release request, > numRequestContainers is decremented by newAllocatedContainers.size() > (correctly). However in case of (alreadyAllocated == true) the decrement > should be undone by incrementing numRequestContainers (i.e. > numRequestContainers++). However the current code is decrementing again (i.e. > numRequestContainers--) on line 886 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (APEXCORE-737) AppMaster does not shut down because numRequestedContainers becomes negative
[ https://issues.apache.org/jira/browse/APEXCORE-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Rozov updated APEXCORE-737: Priority: Minor (was: Major) > AppMaster does not shut down because numRequestedContainers becomes negative > > > Key: APEXCORE-737 > URL: https://issues.apache.org/jira/browse/APEXCORE-737 > Project: Apache Apex Core > Issue Type: Bug >Reporter: Sanjay M Pujare >Assignee: Sanjay M Pujare >Priority: Minor > Fix For: 3.7.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > A previous fix (for APEXCORE-624) has introduced a regression where > numRequestedContainers inside StreamingAppMasterService.execute() becomes > negative because instead of incrementing numRequestedContainers it is being > decremented under certain conditions as follows: > While processing the YARN response of container allocation/release request, > numRequestContainers is decremented by newAllocatedContainers.size() > (correctly). However in case of (alreadyAllocated == true) the decrement > should be undone by incrementing numRequestContainers (i.e. > numRequestContainers++). However the current code is decrementing again (i.e. > numRequestContainers--) on line 886 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (APEXCORE-777) Application Master may not shutdown due to incorrect numRequesteContainers counting
Vlad Rozov created APEXCORE-777: --- Summary: Application Master may not shutdown due to incorrect numRequesteContainers counting Key: APEXCORE-777 URL: https://issues.apache.org/jira/browse/APEXCORE-777 Project: Apache Apex Core Issue Type: Bug Reporter: Vlad Rozov Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
