Parent/Child relationships

Hi Ignite-users, We are looking for some advice on how best to approach a problem using Ignite. We’d be grateful if someone who has encountered this type of thing could share their experience. We are looking for a recommended way of modelling a Parent->Child type relationship between caches such

Re: Lag before records are visible after transaction commit

Hi is anyone able to help look into/reproduce this with the example code given? thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

Thanks for looking into it. Is this expected? Just wondering how another node can ever be transactionally notified of an update in an event driven way if continuous queries don't support transactions? using getAll isn't a practical workaround unfortunately as we want to get the B records based on

Re: Lag before records are visible after transaction commit

unfortunately the 's' on B here can't be derived from a number 0..n - e.g. it isn't a numeric id. E.g. in practice lets say: A is a "Location" it has properties: "city", "street" etc B is a "Person" with key: p = city q = street r = social security number E.g. an A and associated B's are

Re: Lag before records are visible after transaction commit

Thanks Evgenii, Could you please elaborate a bit on how the get would work here. E.g. parent object A has properties p, q, r child object B has properties q, r, s, t {q, r, s} are the primary key of B (as defined in backing SQL table DDL which is how the cache was created) When an A update

Re: Lag before records are visible after transaction commit

Thanks Evgenii, Sorry to keep revisiting this - maybe I am misunderstanding, but don't we also need 's' to be able to query B by key. E.g. the key of B consists of {q, r, s} We only have q and r from the parent A. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ContinuousQuery Batch updates

Hi, We have an app that writes N records to the cluster (REPLICATED) - e.g. 10,000 records, in one transaction. We also have an app that issues a continuous query against the cluster, listening for updates to this cache. We'd like the app to receive all 10,000 records in one call into the

Re: Client App Object Allocation Rate

Thanks! I will have a read through -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Client App Object Allocation Rate

Hi, we are moving away from a legacy (in house) distributed cache solution to ignite, and are doing some profiling of new app vs old app behaviour before we go live. One thing we have noticed, is that the object allocation rate is higher in the version of the app running with the ignite client

Re: ContinuousQuery Batch updates

Thanks, this is what I have ended up doing. However, it looks like AffinityKeyMapper is deprecated? I am adding an implementation of this (which returns the binary typename of the key BinaryObject) - and this does seem to have the desired effect (e.g. all keys with the same typename are marked as

Re: ContinuousQuery Batch updates

Hi, thanks for the reply again! 1. @AffinityKeyMapped is not deprecated as you mentioned, but AffinityKeyMapper is (it seems the AffinityKeyMapper is used in places where the annotation cannot be - e.g. our case). if we use the AFFINITY_KEY clause on the table definition, we don't want to select

Re: ContinuousQuery Batch updates

Hi, RE: the custom affinity function, this is what we have: public class CacheLevelAffinityKeyMapper implements AffinityKeyMapper { private final Logger LOGGER = LoggerFactory.getLogger(CacheLevelAffinityKeyMapper.class); @Override public Object affinityKey(Object key) {

Lag before records are visible after transaction commit

Hi, I am performing the following operation on node 1 of my 3 node cluster (All caches use CacheRebalanceMode.SYNC, CacheWriteSynchronizationMode.FULL_SYNC, CacheAtomicityMode.TRANSACTIONAL): try (Transaction tx = ignite.transactions().txStart(

Re: Lag before records are visible after transaction commit

Thanks for the reply but we are already using that setting which is the strange thing (All caches use CacheRebalanceMode.SYNC, CacheWriteSynchronizationMode.FULL_SYNC, CacheAtomicityMode.TRANSACTIONAL): As an update, the same behaviour is observed if B is retrieved using a ScanQuery rather than

Re: Lag before records are visible after transaction commit

As an update, if I update the printBs method to also try a cache.getAll(keys), it still exhibits the same problem (missing records): private int printBs(long id) { IgniteCache cacheB = ignite.cache("B").withKeepBinary(); ScanQuery scanQuery = new ScanQuery<>(

Re: Block until partition map exchange is complete

Still seeing the same issue in 2.8.1 unfortunately. I have a related question however. Assuming I perform the following operation on node 1 of my 3 node cluster (All caches use CacheRebalanceMode.SYNC, CacheWriteSynchronizationMode.FULL_SYNC, CacheAtomicityMode.TRANSACTIONAL):

Re: Lag before records are visible after transaction commit

Here is a reproducer for this btw. Run the mainclass with program argument READER and again with argument WRITER. In the console for WRITER press a key (this will generate an A and 100 associated Bs) READER subscribes to A and gets the associated B's with a scan query. However, it takes some

Re: Lag before records are visible after transaction commit

Hi, thanks - that is broadly what we are doing: 1. A continuous query on A (annotated with @IgniteAsyncCallback). 2. When an update comes in, fetch all the associated keys of B. We were doing step 2 with an SQL fields query - and the issue was that not all the B's were available yet because the

Re: Lag before records are visible after transaction commit

Yeah sadly that is the only option we can think of too Not very nice though - and we'd still need to somehow transmit information in the callback about how many B's to wait for - and what to do if they never arrive etc Is there some other blocking operation on the B cache perhaps we can call

Re: Lag before records are visible after transaction commit

Are there any other ways we can model this to make this problem easier to solve with Ignite (95% of our other caches don't have this requirement, but we need to solve the 5% to migrate to ignite from our legacy solution) The original thread on this is here btw:

Workaround for getting ContinuousQuery to support transactions

Following on from: http://apache-ignite-users.70518.x6.nabble.com/ContinuousQuery-Batch-updates-td34198.html The takeaway from there is that the continuous queries do not honour transactions, so if a writer writes 100 records (e.g. CalculationParameters) in a transaction, the continuous query

Re: Client App Object Allocation Rate

This doesn't seem to help unfortunately. Re-examining the allocation stats, it seems the app is actually allocating around 1.5mb per second with ignite (vs only 0.15mb per second without ignite in the app). I've read about past issues with IGNITE_EXCHANGE_HISTORY_SIZE causing a lot of allocations,

Large Heap with lots of BinaryMetaDataHolders

Hi, could anyone please help understand why the heap of a client app has such large amounts of data pertaining to binary meta data? Here it takes up 30mb but in our UAT environment we have approx 50 caches. The binary meta data that gets added to the client's heap equats to around 220mb (even for

Re: Workaround for getting ContinuousQuery to support transactions

Hi thanks for the reply. Appreciate the suggestion - and if creating a new solution around this, we would likely take that tact. Unfortunately the entire platform we are looking to migrate over to ignite has dependencies in places for updates to come in as a complete batch (e.g. whatever was in an

Re: Client App Object Allocation Rate

Hi, here's an example (using YourKit rather than JFR). Apologies, I had to obfuscate some of the company specific information. This shows a window of about 10 seconds of allocations Looks like these come from

Re: Block until partition map exchange is complete

Hi, could the behaviour I have observed be captured by this bug: https://issues.apache.org/jira/browse/IGNITE-9841 "Note, ScanQuery exhibits the same behavior - returns partial results when some partitions are lost. Not sure if solution would be related or needs to be tracked and fixed under a

Re: Block until partition map exchange is complete

Hi, the following setup should reproduce the issue: A server class starts up a server node with the config in my original mail (eg 3 servers, partitioned with 1 backup). In that class, at the end do something like: ignite.events(ignite.cluster().forServers()).localListen(ignitePredicate,

Re: Block until partition map exchange is complete

By the way, just referring back to the original question - is there such a callback that can be used to wait for the partition exchange to complete, in any version of ignite? We are using ignite 2.7.6 (which I acknowledge is slightly behind - but we planning to upgrade) -- Sent from:

Block until partition map exchange is complete

Hi Ignite users, I have 3 nodes running, with a cache with the following configuration: cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); cacheConfiguration.setBackups(1); cacheConfiguration.setRebalanceMode(CacheRebalanceMode.SYNC);

Re: Block until partition map exchange is complete

Hi Ilya, thanks for the quick help! Within the local listen, I am adding a task to an executor - so the cache operations happen in a different thread. However, is the key thing here that the local listen handler metho needs to have returned? E.g. the local listen may not have fully completed by

Re: Block until partition map exchange is complete

Thanks - the issue I have now is how can I confirm that the local listen has returned before executing my code? e.g. in the local listen I can set a flag, and then the local listen returns - but the thread that detects this flag and runs the task could still be scheduled to run before the local

Re: Large Heap with lots of BinaryMetaDataHolders

Hi did anyone have any possible suggestions for this? Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client App Object Allocation Rate

Hi did you get a response for this out of interest? also is there a ticket we can follow? We really need to understand this - and ideally turn it off -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client App Object Allocation Rate

Yep theyre the ones we'd like to turn off... is that possible with IGNITE_DISCOVERY_DISABLE_CACHE_METRICS_UPDATE=true? it doesn't seem to have an effect -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Workaround for getting ContinuousQuery to support transactions

Thanks for this, We considering this approach - writing all the entries to some table V, and then updating a separate token cache T with a signal, picked up by the continuous query, which then filters the underlying V records, transforms them and sends them to the client. However, one problem we

Re: Client App Object Allocation Rate

hi, 6k/sec was a misreading on my part - its more 1mb+ ! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client App Object Allocation Rate

Also according to the heap dump they aren't cache statistic messages, but rather, TcpDiscoveryClientMetricsUpdateMessage -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Workaround for getting ContinuousQuery to support transactions

interesting! might just work. We will try it out. E.g. A chance of 500 V's. V has fields a, b, c, (b=foo on all records) and some client app wants to run a continuous query on all V where b=foo, or was =foo but now is not following the update. The writer updates 100 V's, by setting b=bar on all

Adding fields without adding to sql table DDL

Hi, we define our caches via a create table statement, and specify our various columns/fields. We would like to add some additional fields, that are not exposed as part of the DDL, so not visible in a select statement. Is this possible? If I try and get a BinaryObjectBuilder for my type, and add

Re: Large Heap with lots of BinaryMetaDataHolders

Hi, there are approx 6 server nodes, and only around 10 client nodes at the moment (there will eventually be several hundred client nodes once this is enabled in production if we can overcome these issues - and the cluster metric messages which are causing over 1mb of allocations per sec). Each

Re: Adding fields without adding to sql table DDL

Last question would this work with datastreamer? e.g. adding a field inside the transformation? Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Workaround for getting ContinuousQuery to support transactions

Thanks, How is this different to multiple puts inside a transaction? By using the data streamer to write the records, does that mean the continuous query will receive all 10,000 records in one go in the local listen? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client App Object Allocation Rate

That's great! seems to eliminate a lot of the traffic. Are there any other optimizations like this you know of that we can make to reduce this any further? We also have the issue documented in this thread:

Re: Client App Object Allocation Rate

Apologies I may have spoken to soon (I was looking at the wrong process). It looks like we can't turn EVT_NODE_METRICS_UPDATED off as it is designated as an internal event GridEventStorageManager.disableEvents line 441 (ignite 2.8.1), this checks to see if the event that is being disabled is

Re: Large Heap with lots of BinaryMetaDataHolders

Hi, Here is the client config: "> "> "

Re: Large Heap with lots of BinaryMetaDataHolders

P.S. here's a screenshot of where this is coming from: Note, some binary meta data seems to be about remote filters and transformers that have been issued by other client apps connected to the cluster (we do not want this stuff propagated to this client app either). The rest of the binary meta

Re: Client App Object Allocation Rate

Thanks, please do keep us posted - perhaps with a ticket number or something we can track as this is currently a blocker for putting ignite into production. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Large Heap with lots of BinaryMetaDataHolders

Hi Andrew any thoughts on this? thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client App Object Allocation Rate

Hi was there any update on this? thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Workaround for getting ContinuousQuery to support transactions

Ah ok so this wouldn't help solve our problem? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Workaround for getting ContinuousQuery to support transactions

Yeah the key thing is to be able to be notified when all records have been updated in the cache. We've tried using IgniteLock or this too by the way (e.g. the writer locks, writes the records, unlocks). Then the client app, internally queues all updates as they arrive from the continuous query.

Java Thin Client: Continuous Queries

Hi, I saw this earlier, which seems to imply continuous queries will be supported in the .Net client in 2.10 https://issues.apache.org/jira/browse/IGNITE-13148 Will the java client also get this functionality? Is there an expected date/release for that as we are very interested in it. If there

Re: Java Thin Client: Continuous Queries

Thats great thanks for confirming, is there a Jira or something we can keep an eye on to get up to date with progress etc? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Java Thin Client: Continuous Queries

Hi, I saw the changes for this were in the master branch? Is there a rough release date for this by any chance? https://issues.apache.org/jira/browse/IGNITE-14402 Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/