Hi Vivek,
I tried out the BoundedDedupOperator and processed ~ 50k records, seems to
be running okay. Here is the example I tried out: https://github.com/apache
/apex-malhar/tree/master/examples/dedup.
Make sure to replace TimeBasedDedupOperator with BoundedDedupOperator. Also
note that this uses the latest code 3.8.0-SNAPSHOT (master).
Can you check if this helps?
~ Bhupesh
_______________________________________________________
Bhupesh Chawda
E: bhup...@datatorrent.com | Twitter: @bhupeshsc
www.datatorrent.com | apex.apache.org
On Fri, Jun 9, 2017 at 11:01 PM, Vivek Bhide <bhide.vi...@gmail.com> wrote:
> Hi Bhupesh,
>
> I even tried using the TimeBoundedDedupe instead of BoundedDedup and even
> that one fails with exception. In this case, the container starts properly
> but as soon as it tries to process the tuples it fails.
>
> Below are configurations
> ================
>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.port.input.
> attr.TUPLE_CLASS</name>
>
> <value>com.tgt.dqs.datausageingest.object.DataSetAttributeWithChecksum</
> value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> keyExpression</name>
> <value>checksum</value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> timeExpression</name>
> <value>date.getTime()</value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> bucketSpan</name>
> <value>1800</value>
> </property>
> <property>
>
> <name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.
> expireBefore</name>
> <value>180000</value>
> </property>
>
> Below are the container logs
> ===================
>
> 2017-06-08 18:59:53,569 INFO util.LoggerUtil
> (LoggerUtil.java:changeLoggersLevel(274)) - changing level of
> com.datatorrent.stram.util.LoggerUtil to INFO
> 2017-06-08 18:59:53,590 INFO engine.StreamingContainer
> (StreamingContainer.java:main(291)) - Child starting with classpath:
> ./kafka-clients-0.9.0.0.jar:./jetty-io-8.1.10.v20130312.jar:
> ./jetty-http-8.1.10.v20130312.jar:./jetty-server-8.1.10.
> v20130312.jar:./activemq-client-5.8.0.jar:./bval-core-
> 0.5.jar:./joda-time-2.9.1.jar:./jetty-servlet-8.1.10.
> v20130312.jar:./mbassador-1.1.9.jar:./jersey-core-1.9.jar:./
> httpcore-4.3.2.jar:./avro-1.7.4.jar:./validation-api-1.1.0.
> Final.jar:./malhar-contrib-3.7.0.jar:./named-regexp-0.2.3.
> jar:./jetty-security-8.1.10.v20130312.jar:./jackson-core-
> asl-1.9.13.jar:./commons-lang3-3.1.jar:./httpcore-4.3.
> 3.jar:./jcip-annotations-1.0.jar:./malhar-hive-3.7.0.jar:./
> rhino-1.7R4.jar:./httpclient-4.3.5.jar:./commons-compress-
> 1.4.1.jar:./jersey-client-1.9.jar:./xbean-asm5-shaded-4.3.
> jar:./jetty-websocket-8.1.10.v20130312.jar:./apex-common-3.
> 7.0-SNAPSHOT.jar:./jackson-annotations-2.7.0.jar:./
> mailapi-1.4.3.jar:./kafka_2.12-0.10.2.0.jar:./bval-jsr303-
> 0.5.jar:./jctools-core-1.1.jar:./netlet-1.3.0.jar:./
> commons-collections-3.2.1.jar:./libthrift-0.9.3.jar:./
> malhar-kafka-3.7.0.jar:./aws-java-sdk-s3-1.10.73.jar:./
> janino-3.0.7.jar:./malhar-library-3.7.0.jar:./minlog-1.
> 2.jar:./hawtbuf-1.9.jar:./datausageingest-1.0-SNAPSHOT.
> jar:./jetty-continuation-8.1.10.v20130312.jar:./jetty-util-
> 8.1.10.v20130312.jar:./jsr305-1.3.9.jar:./lz4-1.2.0.jar:./
> httpclient-4.3.6.jar:./apex-api-3.7.0-SNAPSHOT.jar:./
> log4j-1.2.17.jar:./activation-1.1.jar:./json-schema-core-1.
> 0.2.jar:./scala-library-2.12.1.jar:./jopt-simple-5.0.3.jar:
> ./kryo-2.24.0.jar:./snappy-java-1.0.4.1.jar:./fastutil-7.
> 0.6.jar:./apex-engine.jar:./guava-11.0.2.jar:./adaptor-
> commons-0.0.2-SNAPSHOT.jar:./geronimo-j2ee-management_1.1_
> spec-1.0.1.jar:./jersey-apache-client4-1.9.jar:./
> zookeeper-3.4.9.jar:./jooq-3.6.4.jar:./slf4j-api-1.7.5.jar:
> ./hive-jdbc-2.0.0.jar:./metrics-core-2.2.0.jar:./
> commons-beanutils-1.9.2.jar:./slf4j-log4j12-1.7.21.jar:./
> apex-shaded-ning19-1.0.0.jar:./libphonenumber-5.3.jar:./aws-
> java-sdk-kms-1.10.73.jar:./aws-java-sdk-core-1.10.73.jar:
> ./hive-service-2.0.0.jar:./commons-logging-1.1.1.jar:./
> zkclient-0.10.jar:./jackson-core-2.7.0.jar:./scala-parser-
> combinators_2.12-1.0.4.jar:./jackson-databind-2.5.4.jar:./
> jms-api-1.1-rev-1.jar:./paranamer-2.3.jar:./apex-
> bufferserver-3.7.0-SNAPSHOT.jar:./hive-exec-0.13.1.jar:./
> json-schema-validator-2.0.1.jar:./commons-compiler-3.0.7.
> jar:./javax.mail-1.5.0.jar:./geronimo-jms_1.1_spec-1.1.1.
> jar:./jackson-mapper-asl-1.9.13.jar:./jackson-dataformat-
> cbor-2.5.3.jar:./xz-1.0.jar:/usr/hdp/current/hadoop-client/
> conf:/usr/hdp/current/hadoop-client/hadoop-azure.jar:/usr/
> hdp/current/hadoop-client/hadoop-annotations.jar:/usr/
> hdp/current/hadoop-client/hadoop-nfs-2.7.3.2.5.3.0-37.
> jar:/usr/hdp/current/hadoop-client/hadoop-nfs.jar:/usr/
> hdp/current/hadoop-client/hadoop-annotations-2.7.3.2.5.
> 3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-azure-2.
> 7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-
> auth.jar:/usr/hdp/current/hadoop-client/hadoop-auth-2.7.
> 3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/hadoop-
> common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/
> hadoop-aws-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> client/hadoop-common-tests.jar:/usr/hdp/current/hadoop-
> client/hadoop-aws.jar:/usr/hdp/current/hadoop-client/
> hadoop-common-2.7.3.2.5.3.0-37-tests.jar:/usr/hdp/current/
> hadoop-client/hadoop-common.jar:/usr/hdp/current/hadoop-
> client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/
> hadoop-client/lib/ojdbc6.jar:/usr/hdp/current/hadoop-client/
> lib/asm-3.2.jar:/usr/hdp/current/hadoop-client/lib/
> ranger-yarn-plugin-shim-0.6.0.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-client/lib/snappy-java-1.0.4.1.jar:/usr/
> hdp/current/hadoop-client/lib/nimbus-jose-jwt-3.9.jar:/usr/
> hdp/current/hadoop-client/lib/curator-framework-2.7.1.jar:/
> usr/hdp/current/hadoop-client/lib/jaxb-impl-2.2.3-1.jar:/
> usr/hdp/current/hadoop-client/lib/commons-codec-1.4.jar:/
> usr/hdp/current/hadoop-client/lib/jsp-api-2.1.jar:/usr/hdp/
> current/hadoop-client/lib/slf4j-log4j12-1.7.10.jar:/usr/
> hdp/current/hadoop-client/lib/stax-api-1.0-2.jar:/usr/hdp/
> current/hadoop-client/lib/jackson-jaxrs-1.9.13.jar:/usr/
> hdp/current/hadoop-client/lib/junit-4.11.jar:/usr/hdp/
> current/hadoop-client/lib/paranamer-2.3.jar:/usr/hdp/
> current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/
> hdp/current/hadoop-client/lib/java-xmlbuilder-0.4.jar:/usr/
> hdp/current/hadoop-client/lib/commons-net-3.1.jar:/usr/hdp/
> current/hadoop-client/lib/jsch-0.1.42.jar:/usr/hdp/
> current/hadoop-client/lib/jackson-databind-2.2.3.jar:/
> usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar:/usr/
> hdp/current/hadoop-client/lib/apacheds-i18n-2.0.0-M15.jar:/
> usr/hdp/current/hadoop-client/lib/ranger-hdfs-plugin-shim-0.
> 6.0.2.5.3.0-37.jar:/usr/hdp/current/hadoop-client/lib/
> commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-client/
> lib/jackson-core-2.2.3.jar:/usr/hdp/current/hadoop-client/
> lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-
> client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/
> hadoop-client/lib/commons-configuration-1.6.jar:/usr/
> hdp/current/hadoop-client/lib/jackson-annotations-2.2.3.jar:
> /usr/hdp/current/hadoop-client/lib/xz-1.0.jar:/usr/
> hdp/current/hadoop-client/lib/guava-11.0.2.jar:/usr/hdp/
> current/hadoop-client/lib/commons-beanutils-core-1.8.0.
> jar:/usr/hdp/current/hadoop-client/lib/gson-2.2.4.jar:/
> usr/hdp/current/hadoop-client/lib/htrace-core-3.1.0-
> incubating.jar:/usr/hdp/current/hadoop-client/lib/
> commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-client/
> lib/jettison-1.1.jar:/usr/hdp/current/hadoop-client/lib/
> ranger-plugin-classloader-0.6.0.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/
> current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/
> hdp/current/hadoop-client/lib/httpcore-4.4.4.jar:/usr/hdp/
> current/hadoop-client/lib/netty-3.6.2.Final.jar:/usr/
> hdp/current/hadoop-client/lib/jersey-json-1.9.jar:/usr/hdp/
> current/hadoop-client/lib/jcip-annotations-1.0.jar:/usr/
> hdp/current/hadoop-client/lib/httpclient-4.5.2.jar:/usr/hdp/
> current/hadoop-client/lib/hamcrest-core-1.3.jar:/usr/
> hdp/current/hadoop-client/lib/curator-recipes-2.7.1.jar:/
> usr/hdp/current/hadoop-client/lib/commons-io-2.4.jar:/usr/
> hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar:/
> usr/hdp/current/hadoop-client/lib/apacheds-kerberos-codec-2.
> 0.0-M15.jar:/usr/hdp/current/hadoop-client/lib/protobuf-
> java-2.5.0.jar:/usr/hdp/current/hadoop-client/lib/
> jetty-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/lib/
> jersey-core-1.9.jar:/usr/hdp/current/hadoop-client/lib/api-
> asn1-api-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/
> jsr305-3.0.0.jar:/usr/hdp/current/hadoop-client/lib/
> xmlenc-0.52.jar:/usr/hdp/current/hadoop-client/lib/
> curator-client-2.7.1.jar:/usr/hdp/current/hadoop-client/lib/
> commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-client/lib/
> jets3t-0.9.0.jar:/usr/hdp/current/hadoop-client/lib/
> jackson-xc-1.9.13.jar:/usr/hdp/current/hadoop-client/lib/
> activation-1.1.jar:/usr/hdp/current/hadoop-client/lib/api-
> util-1.0.0-M20.jar:/usr/hdp/current/hadoop-client/lib/
> azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-
> client/lib/slf4j-api-1.7.10.jar:/usr/hdp/current/hadoop-
> client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-client/
> lib/commons-lang3-3.4.jar:/usr/hdp/current/hadoop-client/
> lib/commons-cli-1.2.jar:/usr/hdp/current/hadoop-client/lib/
> joda-time-2.8.1.jar:/usr/hdp/current/hadoop-client/lib/
> jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-client/
> lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-client/lib/
> log4j-1.2.17.jar:/usr/hdp/current/hadoop-client/lib/
> commons-digester-1.8.jar:/usr/hdp/current/hadoop-client/lib/
> jersey-server-1.9.jar:/usr/hdp/current/hadoop-client/lib/
> zookeeper-3.4.6.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> client/lib/azure-storage-4.2.0.jar:/usr/hdp/current/hadoop-
> client/lib/commons-collections-3.2.2.jar:/usr/
> hdp/current/hadoop-client/lib/mockito-all-1.8.5.jar:/usr/
> hdp/current/hadoop-client/lib/json-smart-1.1.1.jar:/usr/hdp/
> current/hadoop-hdfs-client/hadoop-hdfs.jar:/usr/hdp/
> current/hadoop-hdfs-client/hadoop-hdfs-2.7.3.2.5.3.0-37-
> tests.jar:/usr/hdp/current/hadoop-hdfs-client/hadoop-
> hdfs-tests.jar:/usr/hdp/current/hadoop-hdfs-client/
> hadoop-hdfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> hdfs-client/hadoop-hdfs-nfs-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-hdfs-client/hadoop-hdfs-nfs.jar:/usr/hdp/
> current/hadoop-hdfs-client/lib/asm-3.2.jar:/usr/hdp/
> current/hadoop-hdfs-client/lib/commons-codec-1.4.jar:/
> usr/hdp/current/hadoop-hdfs-client/lib/commons-lang-2.6.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/xercesImpl-2.
> 9.1.jar:/usr/hdp/current/hadoop-hdfs-client/lib/
> commons-logging-1.1.3.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/
> hadoop-hdfs-client/lib/jackson-core-asl-1.9.13.jar:/
> usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar:/
> usr/hdp/current/hadoop-hdfs-client/lib/netty-all-4.0.23.
> Final.jar:/usr/hdp/current/hadoop-hdfs-client/lib/guava-
> 11.0.2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/htrace-
> core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/hadoop-
> hdfs-client/lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-
> hdfs-client/lib/protobuf-java-2.5.0.jar:/usr/hdp/current/
> hadoop-hdfs-client/lib/leveldbjni-all-1.8.jar:/usr/
> hdp/current/hadoop-hdfs-client/lib/jetty-6.1.26.hwx.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/jersey-core-1.
> 9.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jsr305-3.0.0.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/xmlenc-0.52.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/okhttp-2.4.0.
> jar:/usr/hdp/current/hadoop-hdfs-client/lib/commons-cli-1.
> 2.jar:/usr/hdp/current/hadoop-hdfs-client/lib/xml-apis-1.3.
> 04.jar:/usr/hdp/current/hadoop-hdfs-client/lib/jetty-
> util-6.1.26.hwx.jar:/usr/hdp/current/hadoop-hdfs-client/
> lib/servlet-api-2.5.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/log4j-1.2.17.jar:/usr/hdp/current/hadoop-hdfs-
> client/lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-
> hdfs-client/lib/commons-daemon-1.0.13.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-server-
> nodemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-
> yarn-client/hadoop-yarn-server-tests.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-common-2.7.3.2.5.
> 3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-
> applicationhistoryservice.jar:/usr/hdp/current/hadoop-yarn-
> client/hadoop-yarn-applications-distributedshell.
> jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-
> server-sharedcachemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-client-2.7.3.2.5.
> 3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-server-resourcemanager-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-registry-2.7.3.2.
> 5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-server-common-2.7.3.2.5.3.0-37.jar:/usr/hdp/current/
> hadoop-yarn-client/hadoop-yarn-server-web-proxy.jar:/
> usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-
> timeline-pluginstorage-2.7.3.2.5.3.0-37.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-api-2.7.3.2.5.3.0-
> 37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-common.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-server-timeline-pluginstorage.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-server-common.jar:
> /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-server-
> nodemanager.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-api.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/current/hadoop-yarn-
> client/hadoop-yarn-registry.jar:/usr/hdp/current/hadoop-
> yarn-client/hadoop-yarn-server-tests-2.7.3.2.5.3.0-37.
> jar:/usr/hdp/current/hadoop-yarn-client/hadoop-yarn-
> applications-unmanaged-am-launcher.jar:/usr/hdp/current/
> hadoop-yarn-client/hadoop-yarn-client.jar:/usr/hdp/
> current/hadoop-yarn-client/hadoop-yarn-server-web-proxy-
> 2.7.3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-applications-distributedshell-2.7.3.2.5.3.
> 0-37.jar:/usr/hdp/current/hadoop-yarn-client/hadoop-
> yarn-server-resourcemanager.jar:/usr/hdp/current/hadoop-
> yarn-client/hadoop-yarn-server-applicationhistoryservice-2.7.
> 3.2.5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/
> hadoop-yarn-applications-unmanaged-am-launcher-2.7.3.2.
> 5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/lib/asm-3.
> 2.jar:/usr/hdp/current/hadoop-yarn-client/lib/snappy-java-1.
> 0.4.1.jar:/usr/hdp/current/hadoop-yarn-client/lib/nimbus-
> jose-jwt-3.9.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/guice-3.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/curator-framework-2.7.1.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jersey-client-1.9.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/commons-codec-1.4.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/jsp-api-2.1.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/stax-api-1.0-2.jar:
> /usr/hdp/current/hadoop-yarn-client/lib/jackson-jaxrs-1.9.
> 13.jar:/usr/hdp/current/hadoop-yarn-client/lib/paranamer-2.3.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/java-xmlbuilder-0.4.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/commons-net-3.1.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/jsch-0.1.42.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-
> databind-2.2.3.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-lang-2.6.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/commons-logging-1.1.3.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/jackson-core-2.2.3.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/jackson-
> mapper-asl-1.9.13.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/metrics-core-3.0.1.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/commons-configuration-1.6.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/jackson-annotations-2.2.3.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/xz-1.0.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/guava-11.0.2.jar:/
> usr/hdp/current/hadoop-yarn-client/lib/commons-beanutils-
> core-1.8.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/gson-2.2.4.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jettison-1.1.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/httpcore-4.4.4.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/netty-3.6.2.Final.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/jersey-json-1.9.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/jcip-annotations-1.0.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/javax.inject-1.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/httpclient-4.5.2.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/curator-
> recipes-2.7.1.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-io-2.4.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/objenesis-2.1.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/commons-compress-1.4.1.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/apacheds-kerberos-codec-2.0.0-
> M15.jar:/usr/hdp/current/hadoop-yarn-client/lib/
> protobuf-java-2.5.0.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/javassist-3.18.1-GA.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/jersey-core-1.9.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/api-asn1-api-
> 1.0.0-M20.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/jsr305-3.0.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/xmlenc-0.52.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/curator-client-2.7.1.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/commons-math3-3.1.1.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jets3t-0.9.0.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/jackson-xc-1.9.13.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/guice-servlet-3.0.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/activation-1.1.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/api-util-1.0.0-M20.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-
> keyvault-core-0.8.0.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/avro-1.7.4.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/jersey-guice-1.9.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/commons-lang3-3.4.jar:/usr/hdp/current/
> hadoop-yarn-client/lib/commons-cli-1.2.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/fst-2.24.jar:/usr/hdp/
> current/hadoop-yarn-client/lib/jetty-util-6.1.26.hwx.jar:
> /usr/hdp/current/hadoop-yarn-client/lib/servlet-api-2.5.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/log4j-1.2.17.
> jar:/usr/hdp/current/hadoop-yarn-client/lib/commons-
> digester-1.8.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/jersey-server-1.9.jar:/usr/hdp/current/hadoop-yarn-
> client/lib/aopalliance-1.0.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/zookeeper-3.4.6.2.5.3.0-37-tests.jar:/usr/
> hdp/current/hadoop-yarn-client/lib/zookeeper-3.4.6.2.
> 5.3.0-37.jar:/usr/hdp/current/hadoop-yarn-client/lib/azure-
> storage-4.2.0.jar:/usr/hdp/current/hadoop-yarn-client/
> lib/commons-collections-3.2.2.jar:/usr/hdp/current/hadoop-
> yarn-client/lib/json-smart-1.1.1.jar:.
> 2017-06-08 18:59:54,005 WARN util.NativeCodeLoader
> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
> library
> for your platform... using builtin-java classes where applicable
> 2017-06-08 18:59:55,009 WARN shortcircuit.DomainSocketFactory
> (DomainSocketFactory.java:<init>(117)) - The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.
> 2017-06-08 18:59:55,435 INFO storage.DiskStorage
> (DiskStorage.java:<init>(53)) - using
> /grid/10/hadoop/yarn/local/usercache/SVDATHDP/appcache/
> application_1496931225841_2096/container_e3093_
> 1496931225841_2096_01_000002/tmp
> as the basepath for spooling.
> 2017-06-08 18:59:55,438 INFO server.Server (Server.java:registered(112)) -
> Server started listening at /0.0.0.0:41023
> 2017-06-08 18:59:56,471 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:56,976 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:57,482 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:57,987 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:58,491 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:58,996 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 18:59:59,500 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 19:00:00,004 INFO engine.StreamingContainer
> (StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending
> request.
> 2017-06-08 19:00:00,414 INFO server.Server (Server.java:onMessage(599)) -
> Received subscriber request: SubscribeRequestTuple{version=1.0,
> identifier=tcp://d-2vwlw12.target.com:41023/5.unique.1,
> windowId=ffffffffffffffff, type=uniquMessages/6.inputPort,
> upstreamIdentifier=5.unique.1, mask=0, partitions=null, bufferSize=1024}
> 2017-06-08 19:00:00,528 INFO engine.StreamingContainer
> (StreamingContainer.java:processHeartbeatResponse(825)) - Deploy request:
> [OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,
> checkpoint={ffffffffffffffff,
> 0,
> 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=
> checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],
> outputs=[OperatorDeployInfo.OutputDeployInfo[portName=
> unique,streamId=uniquMessages,bufferServer=d-2vwlw12.target.com]]]]
> 2017-06-08 19:00:00,664 INFO server.Server (Server.java:onMessage(555)) -
> Received publisher request: PublishRequestTuple{version=1.0,
> identifier=5.unique.1, windowId=ffffffffffffffff}
> 2017-06-08 19:00:03,105 INFO util.AsyncFSStorageAgent
> (AsyncFSStorageAgent.java:save(91)) - using
> /grid/10/hadoop/yarn/local/usercache/SVDATHDP/appcache/
> application_1496931225841_2096/container_e3093_
> 1496931225841_2096_01_000002/tmp/chkp4165334308239559126
> as the basepath for checkpointing.
> 2017-06-08 19:05:57,806 ERROR engine.StreamingContainer
> (StreamingContainer.java:run(1456)) - Operator set
> [OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,
> checkpoint={ffffffffffffffff,
> 0,
> 0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=
> checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,
> locality=<null>,partitionMask=0,partitionKeys=<null>]],
> outputs=[OperatorDeployInfo.OutputDeployInfo[portName=
> unique,streamId=uniquMessages,bufferServer=d-2vwlw12.target.com]]]]
> stopped running due to an exception.
> java.lang.IllegalArgumentException: Invalid slice: offset=0, length=0
> array.length=0
> at com.datatorrent.netlet.util.Slice.<init>(Slice.java:43)
> at
> org.apache.apex.malhar.lib.utils.serde.BufferSlice.<init>
> (BufferSlice.java:48)
> at
> org.apache.apex.malhar.lib.utils.serde.BufferSlice.<init>
> (BufferSlice.java:58)
> at
> org.apache.apex.malhar.lib.utils.serde.SliceUtils.
> toBufferSlice(SliceUtils.java:111)
> at
> org.apache.apex.malhar.lib.state.managed.Bucket$
> DefaultBucket.put(Bucket.java:421)
> at
> org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.
> putInBucket(AbstractManagedStateImpl.java:286)
> at
> org.apache.apex.malhar.lib.state.managed.ManagedTimeUnifiedStateImpl.put(
> ManagedTimeUnifiedStateImpl.java:72)
> at
> org.apache.apex.malhar.lib.dedup.TimeBasedDedupOperator.putManagedState(
> TimeBasedDedupOperator.java:189)
> at
> org.apache.apex.malhar.lib.dedup.AbstractDeduper.processAuxiliary(
> AbstractDeduper.java:316)
> at
> org.apache.apex.malhar.lib.dedup.AbstractDeduper.
> endWindow(AbstractDeduper.java:337)
> at
> com.datatorrent.stram.engine.GenericNode.processEndWindow(
> GenericNode.java:153)
> at com.datatorrent.stram.engine.GenericNode.run(GenericNode.
> java:397)
> at
> com.datatorrent.stram.engine.StreamingContainer$2.run(
> StreamingContainer.java:1428)
> 2017-06-08 19:05:58,082 INFO engine.StreamingContainer
> (StreamingContainer.java:processHeartbeatResponse(808)) - Undeploy
> request:
> [5]
> 2017-06-08 19:05:58,084 INFO engine.StreamingContainer
> (StreamingContainer.java:undeploy(561)) - Undeploy complete.
> 2017-06-08 19:05:58,085 INFO server.Server (Server.java:run(414)) -
> Removing ln
> LogicalNode@411ca26bidentifier=tcp://d-2vwlw12.target.com:41023/5.unique.1
> ,
> upstream=5.unique.1, group=uniquMessages/6.inputPort, partitions=[],
> iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@
> 64b843d2{da=com.datatorrent.bufferserver.internal.DataList$Block@2c9a38f1{
> identifier=5.unique.1,
> data=67108864, readingOffset=5661, writingOffset=6062,
> starting_window=5939e4e6000002cf, ending_window=5939e4e6000002ff,
> refCount=2, uniqueIdentifier=0, next=null, future=null}}} from dl
> com.datatorrent.bufferserver.internal.DataList@e9ca021 {5.unique.1}
>
> Regards
> Vivek
>
>
>
> --
> View this message in context: http://apache-apex-users-list.
> 78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-
> IllegalArgumentException-bucket-conflict-tp1698p1703.html
> Sent from the Apache Apex Users list mailing list archive at Nabble.com.
>
