Re: Enable Kerberos on Ignite Kafka streamer / support for new consumer config

[Andrey Mashenkov]

Hi Jackson,

I can't figure out how to switch to new style config.
Seem, "bootstrap.servers" option should be used instead of
"zookeeper.connect".
Do you mean using "bootstrap.server" option is not working for you?



On Wed, Feb 7, 2018 at 1:18 PM, jackson.dickson 
wrote:

> Thanks Andrew for your response.
>
> As per this documentation (
> https://docs.confluent.io/3.0.0/kafka/sasl.html#sasl-
> configuration-for-kafka-clients
>  configuration-for-kafka-clients>
> ), SASL auth is supported only for new Kafka consumers. The new kafka
> consumers connect to the Kafka brokers directly instead of zookeeper
>
>
> Below two security config are required to be specified in the consumers
> configs which are only supported in the new Kafka consumer config.
> security.protocol
> sasl.kerberos.service.name
>
> Refer -  https://kafka.apache.org/0100/documentation.html#
> oldconsumerconfigs
> 
>
> The ignite kafka streamer uses old consumer config
> (kafka.consumer.ConsumerConfig). I get below errors in the log file if I
> specify these new consumer configs.
>
> 2018-02-07 05:11:42.432|myapp|ip-x-x-x-x.myserver|[main]|INFO
> |||kafka.utils.VerifiableProperties:info|68|Verifying properties
> 2018-02-07 05:11:42.454|myapp|ip-x-x-x-x.myserver|[main]|WARN
> |||kafka.utils.VerifiableProperties:warn|83|Property auto.reset.offset
> is not valid
> 2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|INFO
> |||kafka.utils.VerifiableProperties:info|68|Property group.id is
> overridden to ignio-kafka-streamer
> 2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|WARN
> |||kafka.utils.VerifiableProperties:warn|83|Property
> sasl.kerberos.service.name is not valid
> 2018-02-07 05:11:42.455|myapp|ip-x-x-x-x.myserver|[main]|WARN
> |||kafka.utils.VerifiableProperties:warn|83|Property security.protocol
> is not valid
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Andrey V. Mashenkov

Re: 20 minute 12x throughput drop using data streamer and Ignite persistence

[slava.koptilin]

Hi Dave,

Could you please provide more details about that use-case.
Is it possible to reproduce the issue and gather JFR and log files from all
participated nodes?
It would be very helpful in order to understand the cause of that behavior.

Thanks!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 20 minute 12x throughput drop using data streamer and Ignite persistence

[slava.koptilin]

Hi Dave,

Is possible to share a code snippet which illustrates DataStreamer settings
and stream receiver code?

Best regards,
Slava.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 20 minute 12x throughput drop using data streamer and Ignite persistence

[David Harvey]

We are pulling in a large number of records from an RDB, and reorganizing
the data so that our analytics will be much faster.

I'm running Sumo, and have looked at all of the log files from all the
nodes, and the only things are checkpoints and GC logs.  The checkpoints
are fast, and occur at a lower rate during the slowdowns.   GC is not a
problem at all.
(I see a project in the future where the number of messages/bytes per TOPIC
are counted.)

The average packet size goes to 6KB from a normal ~ 400 bytes.   I'm going
to add  a rebalance throttle.


On Tue, Feb 13, 2018 at 5:41 AM, slava.koptilin 
wrote:

> Hi Dave,
>
> Could you please provide more details about that use-case.
> Is it possible to reproduce the issue and gather JFR and log files from all
> participated nodes?
> It would be very helpful in order to understand the cause of that behavior.
>
> Thanks!
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: Logging using Log4Net

[Alexey Popov]

Hi,

Hm, actually, you've got several log4net misconfigurations in your solution.
Everything was fine with Ignite config itself ).
Please see updated minified solution attached. Just "build a solution" and
it will restore all required packages.

The following log4net misconfigurations fixed:

1. Your app.config file section  was ignored, an explicit call
XmlConfigurator.Configure() is required for log4net.

2. you should always have  logger in  section.

3.  name attribute could not be abstract. It should have some
package name.

4. %property{LogName} from xml should be initialized in global context
before usage. Please see call 
log4net.GlobalContext.Properties["LogName"] =
System.Reflection.Assembly.GetExecutingAssembly().GetName().Name;

Please find more info about log4net at [1]

Thank you,
Alexey

[1] https://logging.apache.org/log4net/release/manual/configuration.html




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: slow query performance against berkley db

[Stanislav Lukyanov]

Hi Rajesh, 

While I don't have - and, probably, no one has - any benchmarks comparing
Ignite vs Berkeley in a single node configuration (as others have said, this
is not really a common use case for Ignite), I can say that performance
problems you see are likely to be caused by your query structure.

Rule of thumb for Ignite's SQL - avoid nested SELECTs. Also make sure you
have proper indexes for the fields you use in conditions. Usually you also
need to make sure that your data is efficiently collocated, but that only
applies to cases when you have multiple nodes.

I've attempted to optimize the SELECT you've posted - here it is:
SELECT st.entryID, st.attrName, st.attrValue, st.attrsType 
FROM "objectclass".Ignite_ObjectClass as t
JOIN "Ignite_DSAttributeStore".IGNITE_DSATTRIBUTESTORE AS st 
ON st.entryID = t.entryID 
JOIN "dn".Ignite_DN AS dnt 
ON st.entryID = dnt.entry
WHERE t.attrValue= ?
AND (st.attrKind = 'u' OR st.attrKind = 'o')
AND dnt.parentDN LIKE ?

I can't really verify its correctness, but I guess it can be a decent place
to start.

Thanks,
Stan



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Kubernetes discovery with readinessProbe

[Bryan Rosander]

We're using Apache Ignite in Kubernetes as a caching layer right now and
it's working well for us so far.

One thing that's been problematic is that when defining a readinessProbe,
the TcpDiscoveryKubernetesIpFinder only sees pods already in a ready state.

This means that if you start all your pods at once (as part of deployment
creation) you'll wind up with each pod being in its own grid.

It seems like others are experiencing this problem and I think I have a
workaround:
https://stackoverflow.com/questions/45176143/ignite-readinessprobe/48773865#48773865

The basic premise is to kill pods after they connect to a grid that doesn't
contain the alphabetical first ip in the service list.

I was wondering if there is a better solution to this segmentation problem
or if this seems workable.

Thanks,
Bryan

Re: Ignite with Spring Cache on K8S, eviction problem

[vkulichenko]

Pavel,

Can you please elaborate on this? What is generated by Spring and why is it
not suitable for Ignite?

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Write ahead log and early eviction of new elements

[Raymond Wilson]

I have a scenario I would like to validate when using Ignite Persistence.



I understand when I add an element to a cache that element is serialized,
placed into the local memory for the cache on that server and then placed
into the WAL pending checkpointing (merging into the persistence store).



What happens if the newly added element is evicted and then re-read from
the cache by the client before the next checkpoint occurs?



Thanks,

Raymond.

RE: Dates before epoch - java.sql.Date

[Williams, Michael]

Nevermind, I made an error, works fine.

Mike

From: Williams, Michael
Sent: Tuesday, February 13, 2018 3:24 PM
To: user@ignite.apache.org
Subject: Dates before epoch - java.sql.Date

If I need to store dates before the Unix epoch, should I use timestamp, as 
java.sql.Date doesn't store anything before 1970? Do date functions support 
timestamp as well as date?

See: https://docs.oracle.com/javase/7/docs/api/java/sql/Date.html
See also: https://apacheignite-sql.readme.io/docs/data-types#section-date


Mike Williams

Dates before epoch - java.sql.Date

[Williams, Michael]

If I need to store dates before the Unix epoch, should I use timestamp, as 
java.sql.Date doesn't store anything before 1970? Do date functions support 
timestamp as well as date?

See: https://docs.oracle.com/javase/7/docs/api/java/sql/Date.html
See also: https://apacheignite-sql.readme.io/docs/data-types#section-date


Mike Williams

MultiMap

[Michael André Pearce]

Hi 

I just filled out the survey from the website and it reminded me of one short 
coming of Apache ignite still that we some times hit that means we occasionally 
have to use Hazelcast still.

And that is lack of a multimap.

We saw last year this JIRA but it seems to have died.

https://issues.apache.org/jira/plugins/servlet/mobile#issue/IGNITE-640

Is there any life in this? It be great to close this api/functionality gap.

Cheers
Mike




Sent from my iPhone

Re: Dates before epoch - java.sql.Date

[vkulichenko]

Mike,

Looks like you already have a solution, right?

http://apache-ignite-users.70518.x6.nabble.com/RE-Dates-before-epoch-java-sql-Date-tp20034.html

-Val



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Dates before epoch - java.sql.Date

[Williams, Michael]

Yep, all set. Sorry. Getting better at this, honestly.

-Original Message-
From: vkulichenko [mailto:valentin.kuliche...@gmail.com] 
Sent: Tuesday, February 13, 2018 6:07 PM
To: user@ignite.apache.org
Subject: Re: Dates before epoch - java.sql.Date

Mike,

Looks like you already have a solution, right?

https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_RE-2DDates-2Dbefore-2Depoch-2Djava-2Dsql-2DDate-2Dtp20034.html&d=DwICAg&c=9g4MJkl2VjLjS6R4ei18BA&r=ipRRuqPnuP3BWnXGSOR_sLoARpltax56uFYU6n57c3GFvMdyEV-dz2ez2lZZpYl0&m=qzHn3Eb4QZ_ZLBqfAELVQJ2by68li402eOFp9eVeMOk&s=9CtXiWxVKTUnQl0DrOdoywNcZn2aDtQZLjzLKLJudwE&e=

-Val



--
Sent from: 
https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dignite-2Dusers.70518.x6.nabble.com_&d=DwICAg&c=9g4MJkl2VjLjS6R4ei18BA&r=ipRRuqPnuP3BWnXGSOR_sLoARpltax56uFYU6n57c3GFvMdyEV-dz2ez2lZZpYl0&m=qzHn3Eb4QZ_ZLBqfAELVQJ2by68li402eOFp9eVeMOk&s=SBYXsRrSDdGJBj8axC5kWs8coxQ8tfwr-EhTVuswbi4&e=

Re: MultiMap

[Amir Akhmedov]

I'm planning to start working on this ticket in couple of days, hopefully
will be implemented soon.

On Feb 13, 2018 6:06 PM, "Michael André Pearce" 
wrote:

> Hi
>
> I just filled out the survey from the website and it reminded me of one
> short coming of Apache ignite still that we some times hit that means we
> occasionally have to use Hazelcast still.
>
> And that is lack of a multimap.
>
> We saw last year this JIRA but it seems to have died.
>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/IGNITE-640
>
> Is there any life in this? It be great to close this api/functionality gap.
>
> Cheers
> Mike
>
>
>
>
> Sent from my iPhone
>

Re: Write ahead log and early eviction of new elements

[Denis Magda]

My guess before the entry (element) gets evicted it will be synced to a 
respective partition file on disk forcibly so that you can grab it from there 
later. 

Ignite persistence experts please confirm my understanding.

—
Denis

> On Feb 13, 2018, at 1:28 PM, Raymond Wilson  
> wrote:
> 
> I have a scenario I would like to validate when using Ignite Persistence.
>  
> I understand when I add an element to a cache that element is serialized, 
> placed into the local memory for the cache on that server and then placed 
> into the WAL pending checkpointing (merging into the persistence store).
>  
> What happens if the newly added element is evicted and then re-read from the 
> cache by the client before the next checkpoint occurs?
>  
> Thanks,
> Raymond.
>  

How to connect to kubernetes pods from ignite client

[Vishwas Bm]

Hi,

I have  setup 3 kuberentes pods each running a ignite server as per the
link.

My question is how to connect to these from an ignite client which is
outside the kubernetes environment. I have the client running in a
contianer.
How does the client discover the ignite pods. Should we use the same
TcpDiscoveryKubernetesIpFinder when starting the client.


*Thanks & Regards,*

*Vishwas *

Re: 20 minute 12x throughput drop using data streamer and Ignite persistence

[Dave Harvey]

I made improvements to the statistics collection in the stream receiver, and
I'm finding an excessive number of retry's of the optimistic transactions we
are using.   I will understand that and retry.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Logging using Log4Net

[Alexey Popov]

Igg.zip   



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/