Re: Async StreamMultipleTupleExtractor
https://ignite.apache.org/docs/latest/extensions-and-integrations/streaming/kafka-streamer чт, 2 дек. 2021 г. в 05:10, yonghua : > How does ignite integrate with kafka? do you have a guide doc? thanks. > > On 2021/12/1 11:06, Maxim Volkomorov wrote: > > I want to Ignite KafkaStreamer commits kafka without waiting for > > execution of extract() method. >
Async StreamMultipleTupleExtractor
Hello, Can @IgniteAsyncCallback make StreamMultipleTupleExtractor asynchronous? I want to Ignite KafkaStreamer commits kafka without waiting for execution of extract() method.
restart service via service proxy
Hello! I used a custom bean with properties for kafka streamer service at ignite config.xml: localhost:9092 latest true ignite-consumer-group org.apache.kafka.common.serialization.StringDeserializer org.apache.kafka.common.serialization.StringDeserializer Injected this bean using: @SpringResource(resourceName = "kafkaConsumerProperties") private transient Properties kafkaConsumerProperties; I have an ignite service with a custom REST API (Jersey). I am trying to implement an ignite service method that can restart another ignite service via ignite serviceProxy. Calling this method at REST API: class RESTController { // some REST method signature here Ignite ignite = Ignition.ignite(); RESTService restService = ignite.services().serviceProxy(RESTService.SERVICE_NAME, RESTService.class, false); restService.restartMyService("myServiceName"); } } public class RESTServiceImpl implements RESTService { public void restartMyService(String serviceName) { ignite.services().cancel(serviceName); ignite.services().deployNodeSingleton(serviceName, new KafkaConsumerService()); } } My error at method call: [2020-07-28 14:58:32,123][ERROR][services-deployment-worker-#70][IgniteServiceProcessor] Failed to initialize service (service will not be deployed): KafkaCDRConsumerServiceorg.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'kafkaConsumerProperties' available I believe that deployNodeSingleton() can't see @SpringResource injection. I guess that proxyService calls operated at Ignite context. What are the reasons? Is there a way to restart my service with injections?
custom ignite-log4j.xml when using stock docker ignite image
Hi! Is there a simple way to use a custom ignite-log4.xml config, nested in URI ignite-config.xml, when ignite is running like a stock docker image? Same question for custom app.properties beans, nested in config.xml. Our docker run cmd: docker run -it --net=localhost -e "CONFIG_URI=http://host/ignite-config.xml; apacheignite/ignite
Re: enum behavior in REST
Hi! http://localhost:8080/ignite?cmd=get=eventors=string=12345 {"successStatus":0,"affinityNodeId":"a34539ec-9c1a-44f9-b400-5d9b63a8a72c","error":null,"sessionToken":null,"response":{"date":"Jul 14, 2020 9:34:35 AM","score":1,"eventorStatus":{"platformType":false},"eventorUuid":"bd71b9c5-445a-4cf7-8ad1-8b7f27817412","events":{"":{"recordTimestamp":"Jul 14, 2020 8:56:53 AM","deliveryTimestamp":"Jul 14, 2020 9:54:13 AM","score":1}},"targetScore":10}} I mentioned "eventorStatus":{"platformType":false} in previous message. I tried to specify a getter explicitly for eventorStatus property with no effect. пт, 10 июл. 2020 г. в 15:15, Ilya Kasnacheev : > Hello! > > Can you provide an example of REST response? I don't understand how this > issue manifests. > > Regards, > -- > Ilya Kasnacheev > > > ср, 8 июл. 2020 г. в 01:25, Maxim Volkomorov <2201...@gmail.com>: > >> I have "type":{"platformType":false}" in REST response for my >> enum object property. >> >> Object property: >> public class Organization { >> //... >> private OrganizationType type; >> //... >>public OrganizationType type() { >> return type; >> } >> //... >> } >> >> OrganizationType : >> public enum OrganizationType { >> /** Non-profit organization. */ >> NON_PROFIT, >> >> /** Private organization. */ >> PRIVATE, >> >> /** Government organization. */ >> GOVERNMENT >> } >> >> I have correct deserializing at log: >> >> ... type=PRIVATE ... >> >> Should I make custom deserialization for REST requests? Could I make a >> custom REST method for retrieving some fields of object? >> >
Ignite node top ram limit
Is there a RAM limit for single node (hardcoded or practical)? What practical top limit is comfortable in terms of node failures?
enum behavior in REST
I have "type":{"platformType":false}" in REST response for my enum object property. Object property: public class Organization { //... private OrganizationType type; //... public OrganizationType type() { return type; } //... } OrganizationType : public enum OrganizationType { /** Non-profit organization. */ NON_PROFIT, /** Private organization. */ PRIVATE, /** Government organization. */ GOVERNMENT } I have correct deserializing at log: ... type=PRIVATE ... Should I make custom deserialization for REST requests? Could I make a custom REST method for retrieving some fields of object?
Re: KafkaStream marshaller error
Ok, if i put an assignment to init(), how will i start it in execute() method, and stop in canсel()? This example was taken here http://apache-ignite-users.70518.x6.nabble.com/Kindly-tell-me-where-to-find-these-jar-files-td12649.html чт, 2 июл. 2020 г. в 18:57, Ilya Kasnacheev : > Hello! > > You should do the assignment in init() method. > > Don't create KafkaStreamer before service is sent over and is ready for > initialization. > > Regards, > -- > Ilya Kasnacheev > > > чт, 2 июл. 2020 г. в 18:54, Maxim Volkomorov <2201...@gmail.com>: > >> public class KafkaStreamerService implements Service { >> >> public static final String SERVICE_NAME = "KafkaStreamerService"; >> private static final long serialVersionUID = 1L; >> >> @IgniteInstanceResource >> private Ignite ignite; >> private KafkaStreamer kafkaStreamer = new >> KafkaStreamer<>(); >> private IgniteLogger log; >> >> чт, 2 июл. 2020 г. в 18:51, Ilya Kasnacheev : >> >>> Hello! >>> >>> Where do you assign kafkaStreamer field? >>> >>> Regards, >>> -- >>> Ilya Kasnacheev >>> >>> >>> чт, 2 июл. 2020 г. в 18:46, Maxim Volkomorov <2201...@gmail.com>: >>> >>>> Now i disabled node Filtering. You mean i started KafkaStreamer in >>>> init()? >>>> I started KafkaStreamer like: >>>> >>>> @Override >>>> public void execute(ServiceContext ctx) throws Exception { >>>> log.info("KafkaStreamerService starting ..."); >>>> kafkaStreamer.start(); >>>> log.info("KafkaStreamerService started OK"); >>>> >>>> In init() i only configure KafkaStreamer parameters. >>>> >>>> @Override >>>> public void init(ServiceContext ctx) throws Exception { >>>> log = ignite.log(); >>>> >>>> IgniteDataStreamer stmr = >>>> ignite.dataStreamer("kafkaCache"); >>>> stmr.allowOverwrite(true); >>>> stmr.autoFlushFrequency(1000); >>>> >>>> kafkaStreamer.setIgnite(ignite); >>>> kafkaStreamer.setStreamer(stmr); >>>> kafkaStreamer.setThreads(4); >>>> ... >>>> >>>> >>>> If i have to avoid fat instances, how i can share kafkaStreamer >>>> instance between init(), execute() and cancel()? >>>> >>>> >>>> чт, 2 июл. 2020 г. в 16:53, Ilya Kasnacheev >>> >: >>>> >>>>> Hello! >>>>> >>>>> You should probably start KafkaStreamer on remote node when the >>>>> service is initialized (init()), instead of starting it in e.g. >>>>> constructor >>>>> and trying to send it to remote node. >>>>> >>>>> Avoid putting fat instances in the fields of service/compute/predicate >>>>> classes. >>>>> >>>>> Regards, >>>>> -- >>>>> Ilya Kasnacheev >>>>> >>>>> >>>>> чт, 2 июл. 2020 г. в 10:42, Maxim Volkomorov <2201...@gmail.com>: >>>>> >>>>>> I have 1 DataNod and 1 Service with streaming. I have a filter for >>>>>> service: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> public boolean apply(ClusterNode node) { >>>>>> Boolean dataNode = node.attribute("kafkastreamer.service.node"); >>>>>> >>>>>> return dataNode != null && dataNode; >>>>>> } >>>>>> >>>>>> >>>>>> I have a marshalling error java.io.NotSerializableException: >>>>>> org.apache.ignite.stream.kafka.KafkaStreamer using KafkaStreamer in my >>>>>> Service: >>>>>> >>>>>> private KafkaStreamer kafkaStreamer = new >>>>>> KafkaStreamer<>(); >>>>>> >>>>>> I only can start service with: >>>>>> >>>>>> private static KafkaStreamer kafkaStreamer = new >>>>>> KafkaStreamer<>(); >>>>>> >>>>>> Is it because Ignite trying data transfer KafkaStreamer instance >>>>>> between nodes? >>>>>>
Re: KafkaStream marshaller error
public class KafkaStreamerService implements Service { public static final String SERVICE_NAME = "KafkaStreamerService"; private static final long serialVersionUID = 1L; @IgniteInstanceResource private Ignite ignite; private KafkaStreamer kafkaStreamer = new KafkaStreamer<>(); private IgniteLogger log; чт, 2 июл. 2020 г. в 18:51, Ilya Kasnacheev : > Hello! > > Where do you assign kafkaStreamer field? > > Regards, > -- > Ilya Kasnacheev > > > чт, 2 июл. 2020 г. в 18:46, Maxim Volkomorov <2201...@gmail.com>: > >> Now i disabled node Filtering. You mean i started KafkaStreamer in >> init()? >> I started KafkaStreamer like: >> >> @Override >> public void execute(ServiceContext ctx) throws Exception { >> log.info("KafkaStreamerService starting ..."); >> kafkaStreamer.start(); >> log.info("KafkaStreamerService started OK"); >> >> In init() i only configure KafkaStreamer parameters. >> >> @Override >> public void init(ServiceContext ctx) throws Exception { >> log = ignite.log(); >> >> IgniteDataStreamer stmr = >> ignite.dataStreamer("kafkaCache"); >> stmr.allowOverwrite(true); >> stmr.autoFlushFrequency(1000); >> >> kafkaStreamer.setIgnite(ignite); >> kafkaStreamer.setStreamer(stmr); >> kafkaStreamer.setThreads(4); >> ... >> >> >> If i have to avoid fat instances, how i can share kafkaStreamer instance >> between init(), execute() and cancel()? >> >> >> чт, 2 июл. 2020 г. в 16:53, Ilya Kasnacheev : >> >>> Hello! >>> >>> You should probably start KafkaStreamer on remote node when the service >>> is initialized (init()), instead of starting it in e.g. constructor and >>> trying to send it to remote node. >>> >>> Avoid putting fat instances in the fields of service/compute/predicate >>> classes. >>> >>> Regards, >>> -- >>> Ilya Kasnacheev >>> >>> >>> чт, 2 июл. 2020 г. в 10:42, Maxim Volkomorov <2201...@gmail.com>: >>> >>>> I have 1 DataNod and 1 Service with streaming. I have a filter for >>>> service: >>>> >>>> >>>> >>>> >>>> >>>> public boolean apply(ClusterNode node) { >>>> Boolean dataNode = node.attribute("kafkastreamer.service.node"); >>>> >>>> return dataNode != null && dataNode; >>>> } >>>> >>>> >>>> I have a marshalling error java.io.NotSerializableException: >>>> org.apache.ignite.stream.kafka.KafkaStreamer using KafkaStreamer in my >>>> Service: >>>> >>>> private KafkaStreamer kafkaStreamer = new >>>> KafkaStreamer<>(); >>>> >>>> I only can start service with: >>>> >>>> private static KafkaStreamer kafkaStreamer = new >>>> KafkaStreamer<>(); >>>> >>>> Is it because Ignite trying data transfer KafkaStreamer instance >>>> between nodes? >>>> >>>> Log: >>>> >>>> [2020-07-02 10:27:47,552][ERROR][main][IgniteServiceProcessor] Failed >>>> to marshal service with configured marshaller [name=KafkaStreamerService, >>>> srvc=services.kafkastreamer.KafkaStreamerService@3dedb4a6, >>>> marsh=JdkMarshaller [clsFilter=null]] >>>> class org.apache.ignite.IgniteCheckedException: Failed to serialize >>>> object: services.kafkastreamer.KafkaStreamerService@3dedb4a6 >>>> at >>>> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:102) >>>> at >>>> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109) >>>> at >>>> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:57) >>>> at >>>> org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10386) >>>> at >>>> org.apache.ignite.internal.processors.service.IgniteServiceProcessor.prepareServiceConfigurations(IgniteServiceProcessor.java:583) >>>> at >>>> org.apache.ignite.internal.processors.service.IgniteServiceProcessor.staticallyConfiguredServices(IgniteServiceProcessor.java:1541) >>>> at >>>> org.apache.ignite.internal.processors.service.IgniteServic
Re: KafkaStream marshaller error
Now i disabled node Filtering. You mean i started KafkaStreamer in init()? I started KafkaStreamer like: @Override public void execute(ServiceContext ctx) throws Exception { log.info("KafkaStreamerService starting ..."); kafkaStreamer.start(); log.info("KafkaStreamerService started OK"); In init() i only configure KafkaStreamer parameters. @Override public void init(ServiceContext ctx) throws Exception { log = ignite.log(); IgniteDataStreamer stmr = ignite.dataStreamer("kafkaCache"); stmr.allowOverwrite(true); stmr.autoFlushFrequency(1000); kafkaStreamer.setIgnite(ignite); kafkaStreamer.setStreamer(stmr); kafkaStreamer.setThreads(4); ... If i have to avoid fat instances, how i can share kafkaStreamer instance between init(), execute() and cancel()? чт, 2 июл. 2020 г. в 16:53, Ilya Kasnacheev : > Hello! > > You should probably start KafkaStreamer on remote node when the service is > initialized (init()), instead of starting it in e.g. constructor and trying > to send it to remote node. > > Avoid putting fat instances in the fields of service/compute/predicate > classes. > > Regards, > -- > Ilya Kasnacheev > > > чт, 2 июл. 2020 г. в 10:42, Maxim Volkomorov <2201...@gmail.com>: > >> I have 1 DataNod and 1 Service with streaming. I have a filter for >> service: >> >> >> >> >> >> public boolean apply(ClusterNode node) { >> Boolean dataNode = node.attribute("kafkastreamer.service.node"); >> >> return dataNode != null && dataNode; >> } >> >> >> I have a marshalling error java.io.NotSerializableException: >> org.apache.ignite.stream.kafka.KafkaStreamer using KafkaStreamer in my >> Service: >> >> private KafkaStreamer kafkaStreamer = new >> KafkaStreamer<>(); >> >> I only can start service with: >> >> private static KafkaStreamer kafkaStreamer = new >> KafkaStreamer<>(); >> >> Is it because Ignite trying data transfer KafkaStreamer instance between >> nodes? >> >> Log: >> >> [2020-07-02 10:27:47,552][ERROR][main][IgniteServiceProcessor] Failed to >> marshal service with configured marshaller [name=KafkaStreamerService, >> srvc=services.kafkastreamer.KafkaStreamerService@3dedb4a6, >> marsh=JdkMarshaller [clsFilter=null]] >> class org.apache.ignite.IgniteCheckedException: Failed to serialize >> object: services.kafkastreamer.KafkaStreamerService@3dedb4a6 >> at >> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:102) >> at >> org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109) >> at >> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:57) >> at >> org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10386) >> at >> org.apache.ignite.internal.processors.service.IgniteServiceProcessor.prepareServiceConfigurations(IgniteServiceProcessor.java:583) >> at >> org.apache.ignite.internal.processors.service.IgniteServiceProcessor.staticallyConfiguredServices(IgniteServiceProcessor.java:1541) >> at >> org.apache.ignite.internal.processors.service.IgniteServiceProcessor.collectJoiningNodeData(IgniteServiceProcessor.java:354) >> at >> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.collect(GridDiscoveryManager.java:861) >> at >> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.collectExchangeData(TcpDiscoverySpi.java:2032) >> at >> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1029) >> at >> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:427) >> at >> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2099) >> at >> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:299) >> at >> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:943) >> at >> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1960) >> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1276) >> at >> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2045) >> at >> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703) >> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117) >> at >> org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1035) >> a
KafkaStream marshaller error
I have 1 DataNod and 1 Service with streaming. I have a filter for service: public boolean apply(ClusterNode node) { Boolean dataNode = node.attribute("kafkastreamer.service.node"); return dataNode != null && dataNode; } I have a marshalling error java.io.NotSerializableException: org.apache.ignite.stream.kafka.KafkaStreamer using KafkaStreamer in my Service: private KafkaStreamer kafkaStreamer = new KafkaStreamer<>(); I only can start service with: private static KafkaStreamer kafkaStreamer = new KafkaStreamer<>(); Is it because Ignite trying data transfer KafkaStreamer instance between nodes? Log: [2020-07-02 10:27:47,552][ERROR][main][IgniteServiceProcessor] Failed to marshal service with configured marshaller [name=KafkaStreamerService, srvc=services.kafkastreamer.KafkaStreamerService@3dedb4a6, marsh=JdkMarshaller [clsFilter=null]] class org.apache.ignite.IgniteCheckedException: Failed to serialize object: services.kafkastreamer.KafkaStreamerService@3dedb4a6 at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:102) at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:57) at org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10386) at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.prepareServiceConfigurations(IgniteServiceProcessor.java:583) at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.staticallyConfiguredServices(IgniteServiceProcessor.java:1541) at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.collectJoiningNodeData(IgniteServiceProcessor.java:354) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.collect(GridDiscoveryManager.java:861) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.collectExchangeData(TcpDiscoverySpi.java:2032) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1029) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:427) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2099) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:299) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:943) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1960) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1276) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2045) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1035) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:921) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:820) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:659) at org.apache.ignite.Ignition.start(Ignition.java:346) at app.KafkaStreamerServiceNodeStartup.main(KafkaStreamerServiceNodeStartup.java:40) Caused by: java.io.NotSerializableException: org.apache.ignite.stream.kafka.KafkaStreamer at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:97) ... 25 more