Hi Denis, I am trying to test Streaming data with Ignite Kafka Streamer Module.
Cache name :
IKAFKA
Topic messages
{"schema":{"type":"string","optional":false},"payload":"hello world 6"}
{"schema":{"type":"string","optional":false},"payload":"hello world 7"}
Tuple extractor:
kafkaStreamer.setSingleTupleExtractor(msg -> {
return new AbstractMap.SimpleEntry<String, String>(new String(msg.key()),
new String(msg.message()));
})
But If I say ignite.cache(‘IKAFKA’), cache is empty with no message.
Can you please correct and help?
Please find the code for reference.
Thanks & Regards,
Sriveena
From: Denis Mekhanikov [mailto:[email protected]]
Sent: Monday, July 02, 2018 8:39 PM
To: [email protected]
Subject: Re: Information regarding Ignite Web Console
You should specify
QueryEntity.valueType<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fcache%2FQueryEntity.html%23setValueType-java.lang.String-&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cdba57179c28741cc91a908d5e02dbcd2%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636661409431739415&sdata=FcyKO1Sy8oTG%2BOXLXGzg2XS4E9T2R%2FEHlH2LDvloTbs%3D&reserved=0>
corresponding to the type name, that you use when constructing a binary object.
Please find attached example, that shows how to insert BinaryObjects in a way,
that will let you query them from SQL.
I made a named constant PERSON_TYPE_NAME to emphasise, that
QueryEntity#valueType<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fcache%2FQueryEntity.html%23setValueType-java.lang.String-&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cdba57179c28741cc91a908d5e02dbcd2%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636661409431739415&sdata=FcyKO1Sy8oTG%2BOXLXGzg2XS4E9T2R%2FEHlH2LDvloTbs%3D&reserved=0>
should match BinaryObject's type name.
If you want different binary objects with the same name to have different
fields, you should disable
BinaryConfiguration#compactFooter<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fconfiguration%2FBinaryConfiguration.html%23setCompactFooter-boolean-&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cdba57179c28741cc91a908d5e02dbcd2%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636661409431749411&sdata=%2FH%2FailQQWResSh1we9dAMVUu9idd57yITlI5opUXXHQ%3D&reserved=0>.
It will let different BinaryObjects with the same name have different schemas.
See the following thread for more information about compactFooter:
http://apache-ignite-users.70518.x6.nabble.com/Best-practice-for-class-versioning-marshaller-error-td22294.html<https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-ignite-users.70518.x6.nabble.com%2FBest-practice-for-class-versioning-marshaller-error-td22294.html&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cdba57179c28741cc91a908d5e02dbcd2%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636661409431749411&sdata=1qEv4ftuaWvuss20ksZZhyjCnFJbxD2lA7MNdNCnjnQ%3D&reserved=0>
But all fields, that you want to access from SQL, should be specified in the
QueryEntity, so you should think about it in advance.
Denis
пн, 2 июл. 2018 г. в 17:54, Denis Mekhanikov
<[email protected]<mailto:[email protected]>>:
> Does all the tables created in ignite gets converted to binary objects
> internally?
Yes, unless you specify a different
IgniteConfiguration#marshaller<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fconfiguration%2FIgniteConfiguration.html%23setMarshaller-org.apache.ignite.marshaller.Marshaller-&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cdba57179c28741cc91a908d5e02dbcd2%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636661409431759416&sdata=GrF7BYIXRQy%2F29%2Fv01XB5M6MRFXEifYYCl7eJVM%2FGV8%3D&reserved=0>.
But if you want to query data with Ignite SQL, only BinaryMarshaller is
applicable.
> Does all the cache entities like Person gets converted to binary objects
> internally?
All entries are serialized with a configured marshaller. It is binary
marshaller by default.
> Is using binary objects better than entity cache objects?
Using POJOs is usually more convenient. But BinaryObject lets you operate over
objects without having the corresponding POJOs on your class path.
Also by using BinaryObject you skip (de)serialization step, when performing
put/get operations, so you may get better performance.
> Is the a way to deserialize AvroFormat messages from kafka to ignite sink?
> Examples are available for String and JSON converters.
You can deserialize any data, coming from Kafka.
All you need is to implement a proper
StreamSingleTupleExtractor<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fstream%2FStreamSingleTupleExtractor.html&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cdba57179c28741cc91a908d5e02dbcd2%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636661409431759416&sdata=Czc1yxhVUceks5FoR058kA%2BctUnuEV8r80SaixTDGL0%3D&reserved=0>
Denis
пн, 2 июл. 2018 г. в 14:03, Sriveena Mattaparthi
<[email protected]<mailto:[email protected]>>:
Denis,
Have some open questions in ignite
1. Does all the tables created in ignite gets converted to binary objects
internally?
2. Does all the cache entities like Person gets converted to binary
objects internally?
3. Is using binary objects better than entity cache objects?
4. Is the a way to deserialize AvroFormat messages from kafka to ignite
sink? Examples are available for String and JSON converters.
Please help.
Thanks & Regards,
Sriveena
From: Denis Mekhanikov
[mailto:[email protected]<mailto:[email protected]>]
Sent: Friday, June 29, 2018 8:09 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Information regarding Ignite Web Console
Sriveena,
You should configure corresponding query entities to be able to query data in
cache.
Annotation driven configuration is also available.
See more:
https://apacheignite.readme.io/docs/cache-queries#section-query-configuration-by-annotations<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fdocs%2Fcache-queries%23section-query-configuration-by-annotations&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C54bd9746eb084b48efc108d5ddce2906%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636658799914056234&sdata=jOJs8Be4hdwLNY1AYdwCHU0sisEzcdSkwGEQY4v9%2F3Y%3D&reserved=0>
Denis
пт, 29 июн. 2018 г. в 12:43, Sriveena Mattaparthi
<[email protected]<mailto:[email protected]>>:
Hi Denis,
I am trying to use the below code to query the binary object
IgniteCache<Integer, BinaryObject> cache =
start.getOrCreateCache(cfg).withKeepBinary();
BinaryObjectBuilder builder = start.binary().builder("BinaryTest");
builder.setField("name", "Test");
cache.put(1, builder.build());
QueryCursor<List<?>> query = cache.query(new SqlFieldsQuery("select name
from BinaryTest"));
But it is failing in latest 2.5 version saying BinaryTest Table does not exist.
How do we query the binary objects in the above example?
Please help.
Thanks & Regards,
Sriveena
From: Denis Mekhanikov
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, June 27, 2018 6:37 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Information regarding Ignite Web Console
Sriveena,
You can have objects of different types in one cache, but querying it will be
tricky.
You will have to configure
QueryEntities<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite-sql.readme.io%2Fdocs%2Fschema-and-indexes%23section-queryentity-based-configuration&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C6e1e06d9d5c0452da8da08d5dc2ee46a%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636657016326903455&sdata=Z093el7e%2Fj%2BngjoDdJjUwWNUWfPu4Yb0OZgKv3ZEiis%3D&reserved=0>
for your data, that will describe, which fields are available for querying.
Annotation based
configuration<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite-sql.readme.io%2Fdocs%2Fschema-and-indexes%23section-queryentity-based-configuration&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C6e1e06d9d5c0452da8da08d5dc2ee46a%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636657016326903455&sdata=Z093el7e%2Fj%2BngjoDdJjUwWNUWfPu4Yb0OZgKv3ZEiis%3D&reserved=0>
is also available.
Querying nested object is also possible, if you configure the query entities
properly:
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-indexing-nested-objects<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite-sql.readme.io%2Fdocs%2Fschema-and-indexes%23section-indexing-nested-objects&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C6e1e06d9d5c0452da8da08d5dc2ee46a%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636657016326903455&sdata=1amkW4husY1fLSN%2FKtEBx8app5u%2FkUJhqiA4JCYgQ0o%3D&reserved=0>
So, if you want to run SQL queries over your data, it should have some concrete
schema.
Denis
ср, 27 июн. 2018 г. в 14:08, Sriveena Mattaparthi
<[email protected]<mailto:[email protected]>>:
Thank you so much for the quicker responses unlike any other forums..I really
appreciate that.
One last question Denis, we have plan to load all the mongodb collections to
ignite cache and perform complex aggregations and join in memory.
But Unlike any RDBMS data stores we cannot have fixed model objects for each
collections as each document in the collection may have its own columns and
datatypes.
Could you please suggest, if ignite is the choice for this kind of scenario
where same mongo collection have different type of data.
Please note that we have tried using BinaryObject, but we are stuck that ignite
doesn’t support querying on the inner binaryobject.( binaryobject inside a
binaryobject – sub documents, array inside a mongo document)
Thanks & Regards,
Sriveena
From: Denis Mekhanikov
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, June 27, 2018 4:02 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Information regarding Ignite Web Console
Sriveena,
CacheStore<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fcache%2Fstore%2FCacheStore.html&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260911744&sdata=7LEcZ6l%2FFMHm7ZZ518w4WH%2FBm8ylC3yhTDTl05AcZ0M%3D&reserved=0>
extends the
CacheWriter<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstatic.javadoc.io%2Fjavax.cache%2Fcache-api%2F1.0.0%2Fjavax%2Fcache%2Fintegration%2FCacheWriter.html&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260911744&sdata=UBNLjs2J4E4IZG6JGhToO6QzOr7QXkNo7OFZycVYbq0%3D&reserved=0>
interface, which has
delete<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstatic.javadoc.io%2Fjavax.cache%2Fcache-api%2F1.0.0%2Fjavax%2Fcache%2Fintegration%2FCacheWriter.html%23delete(java.lang.Object)&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260921745&sdata=2q1Mutj83625oVhVIEiawuiarAsK8UNmyZvrbk9J7nE%3D&reserved=0>
and
deleteAll<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstatic.javadoc.io%2Fjavax.cache%2Fcache-api%2F1.0.0%2Fjavax%2Fcache%2Fintegration%2FCacheWriter.html%23deleteAll(java.util.Collection)&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260921745&sdata=NIJFLEwPbXOgpgkja9m48c8hENGNg1dTOyghSzaIiJ4%3D&reserved=0>
methods, which will be called by Ignite, when you remove entries from Ignite
caches.
write<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstatic.javadoc.io%2Fjavax.cache%2Fcache-api%2F1.0.0%2Fjavax%2Fcache%2Fintegration%2FCacheWriter.html%23write(javax.cache.Cache.Entry)&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260931759&sdata=tBdc%2FPmMicFBKc82w7s10TvP5r9Sl7%2B7FLPJidwsFsw%3D&reserved=0>
and
writeAll<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstatic.javadoc.io%2Fjavax.cache%2Fcache-api%2F1.0.0%2Fjavax%2Fcache%2Fintegration%2FCacheWriter.html%23writeAll(java.util.Collection)&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260931759&sdata=Sw5W5BSEgyrBRfoIUpx7pwIfYl5gXSNXtYefE4ghmBk%3D&reserved=0>
methods will be called, when you put some data into cache.
It works the same way with MySQL or any other relational DB.
So, CacheStore gives you a possibility to implement synchronization with any
external data source. You can do it for MongoDB, if you need.
Denis
ср, 27 июн. 2018 г. в 13:18, Sriveena Mattaparthi
<[email protected]<mailto:[email protected]>>:
Thanks Denis for the pointers.
But the case explained in
https://www.youtube.com/watch?v=XBtI2Z01kAA<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXBtI2Z01kAA&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf08122ce8c394e2c7a1e08d5dc193829%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656923260941765&sdata=HLmtH02Dr4h455PggnRBQxe0eZJxENxp7nDlPXb5xek%3D&reserved=0>
- Using Ignite Web Console for Automatic RDBMS Integration with Apache Ignite
- Part 3.
shows that any deletions happening on ignite cache are getting reflected in the
mysql database.
Which I thought is ignite feature of autosyncing data to and from RDBMS
databases.
Please correct my understanding.
Thanks & Regards,
Sriveena
From: Denis Mekhanikov
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, June 27, 2018 3:28 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Information regarding Ignite Web Console
Sriveena,
Web Console can generate schema in Ignite based on an existing schema in a
relational database.
But you can do it yourself, without use of Web Console.
Web Console only makes it easier to wire-up stuff, i.e. generate required data
classes and configuration.
Unfortunately, this feature is not available for NoSQL databases. But you still
can configure everything yourself.
You will need to implement a
CacheStore<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fignite.apache.org%2Freleases%2Flatest%2Fjavadoc%2Forg%2Fapache%2Fignite%2Fcache%2Fstore%2FCacheStore.html&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf4918a96e28644d507b908d5dc1491ba%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656903279754570&sdata=V09AYASdhDUexLS4ZNs44zvEWjJ1lmZFqjVRwQft2xs%3D&reserved=0>,
that will load data from your database into Ignite.
Here is documentation on this feature:
https://apacheignite.readme.io/docs/3rd-party-store<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapacheignite.readme.io%2Fdocs%2F3rd-party-store&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7Cf4918a96e28644d507b908d5dc1491ba%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656903279754570&sdata=9w94GuXNlswyUb9XshSsZtol3Qp8ib7GIk9dpwbZiYQ%3D&reserved=0>
Denis
ср, 27 июн. 2018 г. в 12:41, Sriveena Mattaparthi
<[email protected]<mailto:[email protected]>>:
Hi Denis,
Thank you so much for the quick turn around
Could you also confirm if there is a similar integration from ignite to NoSQL
database like MongoDB ?
Our requirement also has mongo db changes to be automatically synced up to
Ignite?
Please help.
Thanks & Regards,
Sriveena
From: Denis Mekhanikov
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, June 27, 2018 2:58 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Information regarding Ignite Web Console
Sriveena,
You can find sources of Web Console in the Ignite Git repository:
https://github.com/apache/ignite/tree/master/modules/web-console<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fignite%2Ftree%2Fmaster%2Fmodules%2Fweb-console&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C1bd39c9b865f43eca6d408d5dc104bee%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656884921777570&sdata=J6LSkoURVCvHJDnKyWrjpFRj02E9T0njpXSxFJPJAvU%3D&reserved=0>
If you want to run Web Console by yourself, you can use the following Docker
image:
https://hub.docker.com/r/apacheignite/web-console-standalone/<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhub.docker.com%2Fr%2Fapacheignite%2Fweb-console-standalone%2F&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C1bd39c9b865f43eca6d408d5dc104bee%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656884921777570&sdata=xQKYRGlEn9UQQ2EDM9nc6RTZLEbHH7eCl7vGCf2UbLM%3D&reserved=0>
There is also Web Console, hosted by GridGain, available at
https://console.gridgain.com/<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconsole.gridgain.com%2F&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C1bd39c9b865f43eca6d408d5dc104bee%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656884921787575&sdata=yyxqlC%2BcnR9yLRwgGxD6aCvy0t9eh2J4djvNsztr2YM%3D&reserved=0>
Denis
ср, 27 июн. 2018 г. в 10:14, Sriveena Mattaparthi
<[email protected]<mailto:[email protected]>>:
Hi,
I am interested to know if Ignite Webconsole is opensource as the document says
Ignite can integrate with any relational database (RDBMS) that supports a JDBC
driver - Oracle, PostgreSQL, Microsoft SQL Server, and MySQL
Ignite supports automatic RDBMS integration via Ignite Web
Console<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fconsole.gridgain.com%2F&data=02%7C01%7CSriveena.Mattaparthi%40ekaplus.com%7C1bd39c9b865f43eca6d408d5dc104bee%7C2a5b4e9716be4be4b2d40f3fcb3d373c%7C1%7C0%7C636656884921797580&sdata=CorLQYS%2FCirN%2BOxK72fvwKFGL5E%2BSQ%2FY4Qn0hPzNvW0%3D&reserved=0>.
Please confirm.
Thanks & Regards,
Sriveena
App.java
Description: App.java
