Hive (IGFS + IgniteMR) vs Hive (Tez)

Hi I am doing a POC on HDP 2.5 Cluster with Ignite as Hadoop Accelerator. we have 3 node cluster each with 8 core and 60G RAM. I was able to run hive on Tez query on a sample data set and finished in 32 sec. The same query took 94 sec in Hive + IGFS + Ignite-MR. I followed most of the instructio

Ignite File System (igfs) spillover to disk

Is it possible to have the IGFS component write to disk once heap/off-heap consumption hits a certain threshold? I have a custom cache store for one component of the app, but a different component requires temporary storage of raw data; we're using igfs for this, but what happens if the file size i

Re: Deadlock during cache loading

I have a fairly similar setup. What type of EC2 instances are you using? Just for compare my setup. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: OutOfMemoryError while streaming

Thanks again. I have one more question. After pushing some data, I see only two nodes having entries out of 8. All the others have 0 entries. My keys are 9 char long strings with a common prefix (all have the same first 3 chars). Wondering why the keys are not evenly distributed. -- Sent fro

Re: OutOfMemoryError while streaming

Non-heap memory is different than off-heap memory. Non-heap is (roughly speaking) memory that the JVM itself uses. Off-heap is what Ignite is using for storage off the heap. So you're probably not looking at what you think you're looking at. -- Sent from: http://apache-ignite-users.70518.x6.nab

Re: OutOfMemoryError while streaming

breischl, Thanks for the quick response. I'm not sure why the node stats show Non-heap memory maximum | 744mb Each server node has 32GB and I've assigned 3g for heap. Shouldn't this be 16GB since that's what I set the data region size to? -

Re: OutOfMemoryError while streaming

You're probably just running out of memory, though if you examine the stacktrace it may tell you if you're running out of heap or off-heap memory. If there's a call to Unsafe.something() in there, it's probably off-heap. Otherwise it's probably on-heap. You do seem to be configuring only a 3 GB h

OutOfMemoryError while streaming

I'm seeing OutOfMemoryError when I have multiple data streamers pushing data into the cluster. My cluster consists of 8 servers running one node each. Each server has 32GB RAM and 4 Cores. All the nodes are started like this = bin/ignite.sh -J-Xmx3g config/dev-cluster-config.xml ===

Re: Deadlock during cache loading

StreamTransformer does an invoke() pretty much exactly like what I'm doing, so that would not seem to change anything. https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/stream/StreamTransformer.java#L50-L53 I may try using a put(), but since I need to co

Re: Deadlock during cache loading

Entries, that are provided to the *receive()* method are immutable. But you can either do *cache.put() *inside the *receive() *method, just like *DataStreamerCacheUpdaters#Individual

Re: Deadlock during cache loading

Hi Denis, It was not clear to me that we could do the update from within the StreamReceiver without some sort of cache operation. Would we just use the CacheEntry.setValue() method to do that? Something roughly like the following? Thanks! public void receive(IgniteCache cache, Collection> newE

Re: Deadlock during cache loading

Hi Denis, It was not clear to me that we could do the update from within the StreamReceiver without some sort of cache operation. Would we just use the CacheEntry.setValue() method to do that? Something roughly like the following? Thanks! public void receive(IgniteCache cache, Collection> newE

Re: What is the difference between PRIMARY_SYNC and FULL_ASYNC

Yep, you understood the difference correctly. I'm not sure, if we are waiting for local updates in FULL_ASYNC mode, though. But anyways, updating local entries is much faster than the remote ones. I'm not aware of any available public benchmark results with comparison of different write synchroniz

What is the difference between PRIMARY_SYNC and FULL_ASYNC

Hi, Does PRIMARY_SYNC means waiting for only primary copies even if the primary copies are on remote nodes, while FULL_ASYNC means waiting for only local copies no matter the local copies are primary or not? Could you please give me an example case to show different performance results with th

Re: Deadlock during cache loading

Hi! Why do you do this inside an invoke()? All of this can be done just inside a receiver. Can you get rid of the invoke and check, that deadlocks disappear? Denis пт, 29 июн. 2018 г. в 17:24, breischl : > That does seem to be what's happening, but we're only invoke()'ing on keys > that were pa

Re: Information regarding Ignite Web Console

Sriveena, You should configure corresponding query entities to be able to query data in cache. Annotation driven configuration is also available. See more: https://apacheignite.readme.io/docs/cache-queries#section-query-configuration-by-annotations Denis пт, 29 июн. 2018 г. в 12:43, Sriveena Ma

RE: Deadlock during cache loading

That does seem to be what's happening, but we're only invoke()'ing on keys that were passed into receive(), so that should not require going off-box. Right? Here's the relevant code... @Override public void receive(IgniteCache cache, Collection> newEntries) throws IgniteException { for (va

Re: Questions on setting up firewall for multicast cluster discovery.

Hi Jon, First of all, you don't have to use multicast for discovery. Using static IP configuration or one other shared IP finder might simplify the setup: https://apacheignite.readme.io/docs/tcpip-discovery Second of all, I'm not sure I fully understand what you're trying to achieve. Are both nod

Questions on setting up firewall for multicast cluster discovery.

Am trying to set up a couple of 2.5.0 nodes on CentOS boxes. I have opened the recommended ports: firewall-cmd --add-port=47500-47502/tcp firewall-cmd --add-port=47100-47200/tcp firewall-cmd --add-port=47400/udp I see an initial UDP packet, to the ignite multicast group address, received corre

RE: Information regarding Ignite Web Console

Hi Denis, I am trying to use the below code to query the binary object IgniteCache cache = start.getOrCreateCache(cfg).withKeepBinary(); BinaryObjectBuilder builder = start.binary().builder("BinaryTest"); builder.setField("name", "Test"); cache.put(1, builder.build()); Query

Re: And again... Failed to get page IO instance (page content is corrupted)

Hi Oleg, Yes, page corruption issues shouldn't happened when persistence is disabled. Please, let us know it you will face one. On Fri, Jun 29, 2018 at 1:56 AM Olexandr K wrote: > Hi Andrey, > > Thanks for clarifying this. > We have just a single persistent cache and I reworked the code to get

RE: Best practice for class versioning: marshaller error

Then I am confused with the exception I got. Please bear with me. I believe what happened was a client (with a new version of user class) trying to deserialize an old object (an old version of user class) from Continuous Query. This was what happened base on the log: 1. GridCacheIoManager unmar