Re: Streaming test
Jessie, If it's a preprocessing done on client side, then I think it should happen on application level. Do required preprocessing, convert to Ignite's data model and then do cache operations. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16349.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Hi Val, thank you getting back. The source type will be different, and the client will deal with the difference in a preprocess that's unique to that data source. Once that's done, it'll be ready for a standard transformation. So we can encapsulate it in a single service or callable. Thanks for explaining the difference between client and servers. It clarifies things and make sense. We wanted to build different type of clients to accommodate the differences in the data sources. Does this make sense? what kind of solutions would you suggest? Jessie On Fri, Aug 18, 2017 at 3:34 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > If data sources are different, then transformations are different as well, > no? If so, how are you going to encapsulate them in a single service? I > would really appreciate a concrete example, it might make this much easier. > I'm probably missing something now. > > Difference between clients and servers is the same is in other systems. > Servers store data, run services and computations, etc. while clients are > only used to interact with the cluster. > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p16310.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
If data sources are different, then transformations are different as well, no? If so, how are you going to encapsulate them in a single service? I would really appreciate a concrete example, it might make this much easier. I'm probably missing something now. Difference between clients and servers is the same is in other systems. Servers store data, run services and computations, etc. while clients are only used to interact with the cluster. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16310.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
We'd like to take either input from streaming or a cache to do some transformation and insert data into another cache. Because the input could be different types, I thought I could encapsulate the transformation and just let a client to call it. The difference between client and server in Ignite isn't very clear to me, to be honest :) One thing I can think of is client doesn't hold data. But I don't know the difference in terms of computation. Jessie On Thu, Aug 17, 2017 at 3:56 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > What does business logic do exactly? Is it supposed to be executed on > server > or client side? > > Can you give an example? > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p16270.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
What does business logic do exactly? Is it supposed to be executed on server or client side? Can you give an example? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16270.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Val, I'd like to build a piece of logic ( could be service, callable, etc) and encapsulate business logic there. And build different clients to for different type of sources, for example file, streaming, jdbc, etc, the client will invoke the business logic by calling the service or callables. What architecture/components would you recommend to use in this type of situation? Thank you very much! Jessie On Tue, Aug 15, 2017 at 5:28 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > Can you describe the use case and what you want to achieve, step by step? > I'll try to suggest a solution. > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p16213.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
Can you describe the use case and what you want to achieve, step by step? I'll try to suggest a solution. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16213.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Btw, I tried two ways to get service proxy on a client node, the first one worked fine, but the one with Stream Visitor failed. 1) I stream the file and put them into a cache, and then loop through the cache entries and call Service 2) I stream the file and define a stream visitor, in which I get the key(String) and value(a defined class) from the Map<K,V> and call the service. The service interface is defined as String, the defined class. Kinda interesting... Thank you for helping out! On Tue, Aug 15, 2017 at 10:07 AM, Jessie Lin <jessie.jianwei@gmail.com> wrote: > The class gets a service proxy and invokes the service. > I thought service could be invoked on client as well...Looks like it's not > designed to be used that way. > > If I wrap the Service in a Callable, can I call it from a client node? > The client node would stream from a local file and call the Callable, I > think? > Does that sound right? > > Jessie > > On Mon, Aug 14, 2017 at 5:30 PM, vkulichenko < > valentin.kuliche...@gmail.com> wrote: > >> What is this class used for? I'm a bit confused - service is usually >> invoked >> on the server, so it will obviously need all classes it works with. Am I >> missing something? >> >> Can you show the whole trace? >> >> -Val >> >> >> >> -- >> View this message in context: http://apache-ignite-users.705 >> 18.x6.nabble.com/Streaming-test-tp14039p16189.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >
Re: Streaming test
The class gets a service proxy and invokes the service. I thought service could be invoked on client as well...Looks like it's not designed to be used that way. If I wrap the Service in a Callable, can I call it from a client node? The client node would stream from a local file and call the Callable, I think? Does that sound right? Jessie On Mon, Aug 14, 2017 at 5:30 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > What is this class used for? I'm a bit confused - service is usually > invoked > on the server, so it will obviously need all classes it works with. Am I > missing something? > > Can you show the whole trace? > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p16189.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
What is this class used for? I'm a bit confused - service is usually invoked on the server, so it will obviously need all classes it works with. Am I missing something? Can you show the whole trace? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16189.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Hello Val, I now put the stream visitor code in a different model and packaged it in a different jar with dependencies. Interestingly, when I ran the program again, the server node has weird error messages. The FileConnector is a class in the client jar. Why would the service needs the caller's signature? Btw, I turned enablePeerClassLoader on, but I still have the same error. Jessie Caused by: java.lang.ClassNotFoundException: com.sample.client.FileConnector at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_ 91] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_91] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8 .0_91] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_91] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_91] at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_91] at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java: 8478) ~[sample-client-0.1-SNAPSHOT-jar-with-dependencies.jar:?] at org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerC ontextImpl.java:340) ~[sample-client-0.1-SNAPSHOT-jar-with-dependencies.jar: ?] at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUt ils.classDescriptor(OptimizedMarshallerUtils.java:268) ~[sample-client:?] -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16188.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Val, thanks for pointing it out. Now I call AtomicLong Function from service#execute() and it's working. Thank you very much! Jessie On Thu, Aug 10, 2017 at 3:08 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > Jessie, > > You still call atomicLong() method from Service#init(). As I already > mentioned, this is causing the startup hang. You should move > IgniteAtomicLong creation out of init() method to avoid it. > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p16113.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
Val, please see thread print attached. This is take after a server is run by "bin\ignite.bat config\ignite-writebehind.xml" and the service initialization didn't complete. Thank you very much for helping out! "srvc-deploy-#33%null%" #59 prio=5 os_prio=0 tid=0x577b8000 nid=0x1ef8 waiting on condition [0x608fe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xc09fbdc8> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.ignite.internal.util.IgniteUtils.await(IgniteUtils.java:7419) at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.awaitInitialization(DataStructuresProcessor.java:1112) at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.atomicLong(DataStructuresProcessor.java:517) at org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3436) at com.sample.SampleServiceImpl.init(SampleServiceImpl.java:63) Jessie On Wed, Aug 9, 2017 at 2:29 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > I can't reproduce it, your project works fine for me. Can you attach thread > dumps? > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p16088.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > 12484: 2017-08-10 12:50:26 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode): "sys-#56%null%" #82 prio=5 os_prio=0 tid=0x58ad8000 nid=0x33dc waiting on condition [0x5b49f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xc0277e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "sys-#55%null%" #81 prio=5 os_prio=0 tid=0x58ad7800 nid=0x1c50 waiting on condition [0x5ab9e000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xc0277e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "sys-#54%null%" #80 prio=5 os_prio=0 tid=0x58ad6800 nid=0x2a40 waiting on condition [0x5b5ff000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xc0277e70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:74
Re: Streaming test
Jessie, What do you mean by "stuck"? Did you check thread dumps? Is it possible you have memory/GC issues? -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p16066.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Val, I replace atomicSequence with atomicLong and used the method below. I need to specify an initial value, start, according to requirement. return ignite.atomicLong( "id", // name. start, // Initial value true // Create if it does not exist. ); But the process got stuck as well. Shall I do something differently? Appreciate your advice. Best, Jessie On Tue, Jul 25, 2017 at 7:55 PM, Jessie Lin <jessie.jianwei@gmail.com> wrote: > Thank you Val. I'll give it a try. > > On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko < > valentin.kuliche...@gmail.com> wrote: > >> Jessie, >> >> atomicLong() method will do this for you automatically. It either creates >> new instance or returns existing one. There is no need for separate >> initialization step. >> >> -Val >> >> >> >> -- >> View this message in context: http://apache-ignite-users.705 >> 18.x6.nabble.com/Streaming-test-tp14039p15646.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >
Re: Streaming test
Thank you Val. I'll give it a try. On Tue, Jul 25, 2017 at 5:16 PM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > Jessie, > > atomicLong() method will do this for you automatically. It either creates > new instance or returns existing one. There is no need for separate > initialization step. > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p15646.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
Jessie, atomicLong() method will do this for you automatically. It either creates new instance or returns existing one. There is no need for separate initialization step. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15646.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Jessie, Ignite#atomicLong method will return existing instance if it is already initialized, so you actually don't need to manage this manually. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15525.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Thank you for running the test. If I need to initialize Atomic Seq, where shall I do it? Jessie On Fri, Jul 21, 2017 at 12:37 PM, vkulichenko <valentin.kuliche...@gmail.com > wrote: > I ran the test. Atomic sequence initialization should be moved out of > init() > because it is called synchronously with node start, and they wait for each > other. > > After I do this, everything works fine. Although CSV file has only 348 and > it gets loaded multiple times in a loop. > > Is there anything else I need to do to reproduce the hang? In addition, if > you do have the hang, take thread dumps from all nodes and attach them > here. > > -Val > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Streaming-test-tp14039p15263.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Streaming test
I ran the test. Atomic sequence initialization should be moved out of init() because it is called synchronously with node start, and they wait for each other. After I do this, everything works fine. Although CSV file has only 348 and it gets loaded multiple times in a loop. Is there anything else I need to do to reproduce the hang? In addition, if you do have the hang, take thread dumps from all nodes and attach them here. -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p15263.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Hello there, please my test project. You can see 1) The initialization seem didn't finish 2) the streaming client stopped after a while. If you take out the Atomic sequence part, the stream start to work. Thank you for helping. Jessie -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p14168.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Streaming test
Hi, Could you please share your test project? Thanks! -Dmitry. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039p14041.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Streaming test
Hello all, I'm testing streaming input data and call a service deployed. Two things happened quite confusing and could someone in the community help? 1. I print a '.' in StreamVisitor.from method to show it's running, and call service via proxy. It actually ran for over 17K but then stopped/stuck. The file I used to stream data in has over 1M records in it. 2. I also print a '.' in the service method to show it's being called. It only printed couple dots and seemed to stuck as well. First I thought, the message queue was too small, but increasing it to 10240 didn't change the behavior. What shall I do next to see where it might have got stuck? Btw, the memory policy looks like this and cache has a expiry policy looks like this. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Streaming-test-tp14039.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
