Re: [ANNOUNCE] New committer: Mikhail Pochatkin

[Stephen Darlington]

Congratulations!

On Wed, 8 May 2024 at 12:08, Mirza Aliev  wrote:

> Congratulations, Mikhail!
>
> ср, 8 мая 2024 г. в 10:53, Roman Puchkovskiy  >:
>
> > Congratulations!
> >
> > ср, 8 мая 2024 г. в 10:28, Zhenya Stanilovsky  > >:
> > >
> > >
> > > Great ! Good luck Mikhail !
> > >
> > >
> > >
> > > >Congratulations, keep it going!
> > > >
> > > >Thanks,
> > > >S.
> > > >
> > > >ср, 8 мая 2024г. в 07:27, Pavel Tupitsyn < ptupit...@apache.org >:
> > > >
> > > >> Dear Igniters,
> > > >>
> > > >> The Project Management Committee (PMC) for Apache Ignite
> > > >> has invited Mikhail Pochatkin to become a committer and we are
> pleased
> > > >> to announce that they have accepted.
> > > >>
> > > >> Being a committer enables easier contribution to the
> > > >> project since there is no need to go via the patch
> > > >> submission process. This should enable better productivity.
> > > >>
> > > >> Mikhail, congratulations on your new role!
> > > >>
> > > >> Pavel Tupitsyn
> > > >> on behalf of Apache Ignite PMC
> > > >>
> > >
> > >
> > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Roman Puchkovskiy

[Stephen Darlington]

Congratulations!

On Wed, 8 May 2024 at 12:07, Mirza Aliev  wrote:

> Congratulations, Roman!
>
> ср, 8 мая 2024 г. в 10:27, Zhenya Stanilovsky  >:
>
> >
> > Great ! My congrats too !
> >
> >
> >
> > >Congrats, Roman! Well deserved!
> > >
> > >Thanks,
> > >S.
> > >
> > >ср, 8 мая 2024г. в 07:28, Pavel Tupitsyn < ptupit...@apache.org >:
> > >
> > >> Dear Igniters,
> > >>
> > >> The Project Management Committee (PMC) for Apache Ignite
> > >> has invited Roman Puchkovskiy to become a committer and we are pleased
> > >> to announce that they have accepted.
> > >>
> > >> Being a committer enables easier contribution to the
> > >> project since there is no need to go via the patch
> > >> submission process. This should enable better productivity.
> > >>
> > >> Roman, congratulations on your new role!
> > >>
> > >> Pavel Tupitsyn
> > >> on behalf of Apache Ignite PMC
> > >>
> >
> >
> >
> >
>

Re: Questions about functions bit_and and bit_or

[Stephen Darlington]

This is really a question for the user mailing list. Would you mind joining
and posting there?

On Fri, 2 Feb 2024 at 06:07, Jian Chen  wrote:

> Dear Igniters,
>
> I have questions about the ignite sql function bit_and and bit_or.
>
> https://ignite.apache.org/docs/latest/sql-reference/aggregate-functions#bit_and
> The official document gives the description about those two functions, but
> when I use it I always meet the error :
> "Error: Unsupported expression: BIT_AND(ID) [type=Aggregate]
> (state=5,code=1)"
> I am use the sql like : "select bitwise_and(id) from city;"
>
> Does the Ignite really supports these twos functions? If yes, how to use
> it? Thanks!
> Regards
>

Re: Ignite Extensions

[Stephen Darlington]

Excellent news! Thanks for the update.

On Mon, 22 Jan 2024 at 16:20, Ivan Daschinsky  wrote:

> Also, I have updated dependencies and fixed native BLAS integration (tested
> on ubuntu 22.04 with Intel MKL)
>
> пн, 22 янв. 2024 г. в 19:18, Ivan Daschinsky :
>
> > Hi, Steven. Currently I am preparing an initial release of the ML
> > extension. Just thinking about creating a new thread and here you are :)
> > The release branch is ready, I just need to prepare artifacts and upload
> > them to staging. Then I am going to start the vote thread.
> >
> >
> >
> https://github.com/apache/ignite-extensions/tree/release/ignite-ml-ext-1.0.0
> >
> > пн, 22 янв. 2024 г. в 18:15, Stephen Darlington  >:
> >
> >> Hi all,
> >>
> >> With Ignite 2.16 now out, is there any movement towards releasing some
> of
> >> the Ignite Extensions?
> >>
> >> We have a question in the user mailing list about the ML extensions,
> which
> >> were moved from core to extensions. And I'd like to see the SpringBoot
> >> extensions released so that we can support current versions (3.x does
> not
> >> work with Ignite out of the box).
> >>
> >> Regards,
> >> Stephen
> >>
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Ignite Extensions

[Stephen Darlington]

Hi all,

With Ignite 2.16 now out, is there any movement towards releasing some of
the Ignite Extensions?

We have a question in the user mailing list about the ML extensions, which
were moved from core to extensions. And I'd like to see the SpringBoot
extensions released so that we can support current versions (3.x does not
work with Ignite out of the box).

Regards,
Stephen

Re: graalvm native images work on recent 2.16.0-SNAPSHOT builds with jdk21

[Stephen Darlington]

That's great news, thanks for sharing!

On Sun, 8 Oct 2023 at 19:30, Scott Feldstein  wrote:

> Hi All, I just wanted to let the community know in case folks weren't aware
> - it looks like Graalvm native images work on Ignite 2.16.0-SNAPSHOT builds
> with jdk21.  The fixes that appear responsible are IGNITE-19652
>  and IGNITE-20532
> .  Unfortunately I
> can't get jdk17 to work properly as graalvm on that version appears to have
> limitations.  Graalvm native-image associated with jdk21 seems to alleviate
> those issues.
>
> If anyone is interested in my poc see
> https://github.com/scottmf/graalvm-ignite
>
> Scott
>

Re: How to achive Ignite cache counter atomic operation

[Stephen Darlington]

I think this question would be better suited to this user mailing list.

On Thu, 21 Sept 2023 at 10:09, Neeraj Mukati 
wrote:

> I am creating a site visitor counter.
>
> site visitor counter per day by geographic location
> previous day counter to be expired automatically and new counter
> initialised at 00 hour or first call for the day.
> values to be store in the bucket where geo location is the key and counter
> is the value.
> counters are to be automatic integers using an increment and get operation.
> this bucket should be persisted in RDBMS.
> Memcached provide all other option expect persistence.
> I want to use only IgniteCache not ClientAtomicLong.
>
> is there any way to do the above use case with ignite cache?
>

Re: [ANNOUNCE] Welcome Stephen Darlington as a new committer

[Stephen Darlington]

Thanks, everyone!

On Mon, 4 Sept 2023 at 13:30, Pavel Tupitsyn  wrote:

> Congrats, Stephen!
>
> On Mon, Sep 4, 2023 at 12:52 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> wrote:
>
> > Well deserved! Three cheers for you, Stephen!
> >
> > Thanks,
> > S.
> >
> > пн, 4 сент. 2023 г. в 12:38, Wei Cheng <29608336...@gmail.com>:
> >
> > >
> > > Congratulations
> > >
> > > Thanks
> > > Wei Cheng
> > >
> > > > 在 2023年9月4日，17:28，ZhangJian He  写道：
> > > >
> > > > Configurations!
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> > > >
> > > >> On Mon, 4 Sept 2023 at 17:24, Kseniya Romanova <
> ksroman...@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >> Igniters!
> > > >>
> > > >> The Project Management Committee (PMC) for Apache Ignite has invited
> > > >> Stephen
> > > >> Darlington to become a committer and we are pleased to announce that
> > he
> > > has
> > > >> accepted.
> > > >>
> > > >> Stephen made several valuable code contributions and he participates
> > > >> actively in discussions about Apache  Ignite development, but he’s
> > best
> > > >> known as a dedicated mentor for those developers who are just
> starting
> > > >> their exciting Ignite journey! Stephen answers their questions on
> the
> > > user
> > > >> list and SOF, supports free Ignite learning sessions as a trainer
> and
> > > >> several times was a host of the community main events - Ignite
> Summit.
> > > >>
> > > >> And I'm sure you agree with me that all of the above is an
> invaluable
> > > >> contribution to the Apache Ignite community!
> > > >>
> > > >> Please join me in welcoming Stephen, and congratulating him on the
> new
> > > role
> > > >> in the Apache Ignite Community!
> > > >>
> > > >> Kseniya Romanova
> > > >>
> > > >> on behalf of Apache Ignite PMC
> > > >>
> > >
> >
>

Re: Apache ignite as Apache druids cache layer

[Stephen Darlington]

This question would be better addressed to the user mailing list or Stack 
Overflow.

> On 1 Aug 2023, at 08:31, Jithin AM  wrote:
> 
> Hi, My name is Jithin. I,m working in  a data management company. We are
> using Apache druid as our database. and now we heard about apache ignite
> and its capability in processing data. Im writing this mail to enquire that
> can we use ignite as a cache layer for druids to increase querying. is this
> possible could u please  give me a guidance for it with proper steps or
> code sample

Re: [IGNITE-19899] SpringBoot doesn't register autoconfigures - ASF JIRA

[Stephen Darlington]

I just created a pull request for something related:

https://issues.apache.org/jira/browse/IGNITE-20059
https://github.com/apache/ignite-extensions/pull/223

In short, the version of SpringBoot we’re using has not been supported for 
nearly three years. This bumps the version used to the current oldest supported 
version.

These are both potential blockers in getting Ignite added to the Spring 
Initializr (https://github.com/spring-io/start.spring.io/issues/960).

Regards,
Stephen

> On 3 Jul 2023, at 11:41, Stephen Darlington  
> wrote:
> 
> Hi all,
> 
> The way that SpringBoot “autoconfigures” itself changed in SpringBoot 2.7 and 
> the old method was removed entirely in 3.x. Our SpringBoot extensions appear 
> not to work with newer versions.
> 
> Luckily the changes required are pretty small. I added the new method, but 
> didn’t remove the old one in this ticket:
> 
>> https://issues.apache.org/jira/browse/IGNITE-19899
> 
> Not sure who is the right person to review/merge?
> 
> Regards,
> Stephen

[IGNITE-19899] SpringBoot doesn't register autoconfigures - ASF JIRA

[Stephen Darlington]

Hi all,

The way that SpringBoot “autoconfigures” itself changed in SpringBoot 2.7 and 
the old method was removed entirely in 3.x. Our SpringBoot extensions appear 
not to work with newer versions.

Luckily the changes required are pretty small. I added the new method, but 
didn’t remove the old one in this ticket:

> https://issues.apache.org/jira/browse/IGNITE-19899

Not sure who is the right person to review/merge?

Regards,
Stephen

Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution

[Stephen Darlington]

Right, it doesn’t need to be the same — exposing partitions in this manner is 
weird — but I think we need to support bulk operations in some capacity. 
Creating a compute job for each record would quickly get unmanageable and 
broadcast isn’t equivalent as it doesn’t take into account the possibility of 
partitions migrating during calculations.

> On 15 May 2023, at 14:30, Andrey Gura  wrote:
> 
>> In Ignite 2, for bulk operations there’s an affinityRun method that takes a 
>> partition as a parameter. I don’t see an equivalent here. Is one in the plan?
> 
> Ideally users don't need to know a partition for execution of
> computation and the right tool for it is colocation. Method with
> partitions was introduced in order to execute scan queries on specific
> partitions and it is some kind of hack. I believe this approach should
> be revised if possible.
> 
>> Are other APIs like apply, async methods, map-reduce being considered in 
>> later phases?
> 
> All methods are async and return CompletableFuture. Yes, Map/reduce,
> etc will be designed in later phases.
> 
> On Mon, May 15, 2023 at 4:08 PM Stephen Darlington
>  wrote:
>> 
>> In Ignite 2, for bulk operations there’s an affinityRun method that takes a 
>> partition as a parameter. I don’t see an equivalent here. Is one in the plan?
>> 
>> Are other APIs like apply, async methods, map-reduce being considered in 
>> later phases?
>> 
>>> On 15 May 2023, at 12:52, Andrey Gura  wrote:
>>> 
>>> Hi, Igniters!
>>> 
>>> Please take a look at the first phase of proposal for Apache ignite 3
>>> Compute API: Simple remote job execution [1].
>>> 
>>> While most issues are already resolved due to simplicity of the proposal
>>> any feedback and ideas will be really useful for the next phases.
>>> 
>>> Thanks!
>>> 
>>> [1]
>>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution
>> 

Re: [DISCUSSION] IEP-100: Compute API: Phase 1 - Simple remote job execution

[Stephen Darlington]

In Ignite 2, for bulk operations there’s an affinityRun method that takes a 
partition as a parameter. I don’t see an equivalent here. Is one in the plan?

Are other APIs like apply, async methods, map-reduce being considered in later 
phases?

> On 15 May 2023, at 12:52, Andrey Gura  wrote:
> 
> Hi, Igniters!
> 
> Please take a look at the first phase of proposal for Apache ignite 3
> Compute API: Simple remote job execution [1].
> 
> While most issues are already resolved due to simplicity of the proposal
> any feedback and ideas will be really useful for the next phases.
> 
> Thanks!
> 
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-100%3A+Compute+API%3A+Phase+1+-+Simple+remote+job+execution

Re: Clustering in k8s minikube...

[Stephen Darlington]

This probably belongs on the user mailing list.

Having said that, what version of Minikube and kubectl? I saw something smilier 
but assumed that I’d messed up and didn’t spend the time to debug. My suspicion 
in that something changed on the k8s side.

> On 15 Dec 2022, at 14:05, Greg Sylvain  wrote:
> 
> Hi,
> 
> I'm trying to get a simple statefulset of two replicas clustering together
> and they are  failing to connect to the MasterUrl. We are running
> minikube over docker on RHEL 7.
> 
> I set the MasterUrl, but I am still unable to lookup the address?
> 
> Thanks for any clues.
> Greg

Re: [DISCUSSION] Removal of ignitevisorcmd

[Stephen Darlington]

One option that’s in Visor but not control.sh is the ability to view cache 
content (cache -scan). I do hear of people using that.

> On 2 Dec 2022, at 09:17, Andrey Mashenkov  wrote:
> 
> Hi Igniters,
> 
> I agree with Slava, we have to provide alternative ways for users before
> dropping a feature that just worked.
> A user may use some feature for a long time without any issues, so, this is
> why there is no mention in chats/mails.
> I mean here, the feature is not only the whole visor command, but a command
> option that just makes UX better.
> 
> I think we can revise visor features, which are not supported by
> control.sh, and create an umbrella ticket for supporting in control.sh.
> Set this ticket either as 'required' for a 'dropping visor ticket', or as a
> blocker for the next release if visor dropping will come first.
> 
> On Fri, Dec 2, 2022 at 11:09 AM Nikolay Izhikov  wrote:
> 
>> Hello, Slava.
>> 
>>> However, do we have a list of useful features that do not exist in our
>> control utility?
>> 
>> I don't have the list on hand.
>> Do you have some feature in mind that must be transferred in control.sh?
>> 
>> I don't find any mentions of visorcmd on user list and in user chats so my
>> guess is - this utility is not used by the users in any way.
>> Do you have some other inputs?
>> 
>> If yes, don't hesitate to create a ticket with the IEP-94 label.
>> 
>> 
>> чт, 1 дек. 2022 г. в 19:11, Вячеслав Коптилин :
>> 
>>> Hi Igniters,
>>> 
>>> I fully support the idea to stop supporting Visor and removing its
>>> implementation.
>>> However, do we have a list of useful features that do not exist in our
>>> control utility?
>>> Perhaps, it makes sense to reimplement such functionality and provide it
>>> based on control.sh script.
>>> 
>>> Thanks,
>>> Slava.
>>> 
>>> чт, 1 дек. 2022 г. в 15:51, Nikolay Izhikov :
>>> 
 PR is ready for review - https://github.com/apache/ignite/pull/10411
 
 чт, 1 дек. 2022 г. в 16:49, Taras Ledkov :
 
> Hi,
> 
> +1 for remove Visor related code.
> Unfortunately we have to migrate `control-utility` to use
>> IgniteClient
> (thin client) before drop GridClient. I guess we will do it in the
 future.
> 
> Also, the old thin JDBC based on the GridClient (classes placed at
>> the
> org.apache.ignite.internal.jdbc.*) must be removed.
> 
 
>>> 
>> 
> 
> 
> -- 
> Best regards,
> Andrey V. Mashenkov

Re: spring-boot-*-autoconfigure extensions release

[Stephen Darlington]

It’s not a blocker for the Spring Initializr but would certainly be nice to 
include if it’s possible.

> On 15 Jul 2022, at 18:56, Maxim Muzafarov  wrote:
> 
> Stephen,
> 
> I've checked the issue you mentioned. I think it's possible to prepare
> a new release.
> 
> On Thu, 14 Jul 2022 at 16:08, Stephen Darlington
>  wrote:
>> 
>> Sorry, I wasn’t clear. The Spring Data change 
>> (https://issues.apache.org/jira/browse/IGNITE-13029) was made after the 
>> 1.0.0 release. Is there a plan to make a 1.1.0 release?
>> 
>>> On 14 Jul 2022, at 14:01, Maxim Muzafarov  wrote:
>>> 
>>> Hello Stephen,
>>> 
>>> Why do you think they haven't been released yet?
>>> 
>>> https://mvnrepository.com/artifact/org.apache.ignite/ignite-spring-boot-thin-client-autoconfigure-ext/1.0.0
>>> https://mvnrepository.com/artifact/org.apache.ignite/ignite-spring-boot-autoconfigure-ext/1.0.0
>>> 
>>> On Thu, 14 Jul 2022 at 15:57, Stephen Darlington
>>>  wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Are there any plans to release the spring-boot-*-autoconfigure extensions? 
>>>> I note there’s a change that auto-configures Spring Data repos in master 
>>>> but not in the current release.
>>>> 
>>>> The reason I ask: I’m trying to get Ignite added to the Spring Initializr 
>>>> (960 
>>>> <https://github.com/spring-io/start.spring.io/issues/960#issuecomment-1181412216>)
>>>>  and they asked about exactly that.
>>>> 
>>>> Regards,
>>>> Stephen
>> 

Re: spring-boot-*-autoconfigure extensions release

[Stephen Darlington]

Sorry, I wasn’t clear. The Spring Data change 
(https://issues.apache.org/jira/browse/IGNITE-13029) was made after the 1.0.0 
release. Is there a plan to make a 1.1.0 release?

> On 14 Jul 2022, at 14:01, Maxim Muzafarov  wrote:
> 
> Hello Stephen,
> 
> Why do you think they haven't been released yet?
> 
> https://mvnrepository.com/artifact/org.apache.ignite/ignite-spring-boot-thin-client-autoconfigure-ext/1.0.0
> https://mvnrepository.com/artifact/org.apache.ignite/ignite-spring-boot-autoconfigure-ext/1.0.0
> 
> On Thu, 14 Jul 2022 at 15:57, Stephen Darlington
>  wrote:
>> 
>> Hi,
>> 
>> Are there any plans to release the spring-boot-*-autoconfigure extensions? I 
>> note there’s a change that auto-configures Spring Data repos in master but 
>> not in the current release.
>> 
>> The reason I ask: I’m trying to get Ignite added to the Spring Initializr 
>> (960 
>> <https://github.com/spring-io/start.spring.io/issues/960#issuecomment-1181412216>)
>>  and they asked about exactly that.
>> 
>> Regards,
>> Stephen

spring-boot-*-autoconfigure extensions release

[Stephen Darlington]

Hi,

Are there any plans to release the spring-boot-*-autoconfigure extensions? I 
note there’s a change that auto-configures Spring Data repos in master but not 
in the current release.

The reason I ask: I’m trying to get Ignite added to the Spring Initializr (960 
)
 and they asked about exactly that.

Regards,
Stephen

Java thin client defaults

[Stephen Darlington]

I’ve been thinking about the “first run” experience and some of the defaults we 
have.

For example, I can start a thick client entirely with the defaults:

var cfg = new IgniteConfiguration();
var ignite = Ignition.start(cfg);

But the same does not work for the thin client:

var cfg = new ClientConfiguration();
var ignite = Ignition.startClient(cfg);

This throws an exception, saying we didn’t set an address.

Why not default to 127.0.0.1:10800? For a first run/developer experience this 
will often be correct. And it would be consistent with some of the other 
thin-clients, such as Python, which default to the loopback address.

Thoughts?

Regards,
Stephen

Re: 2.13 and Calcite

[Stephen Darlington]

Excellent, thanks.

> On 29 Apr 2022, at 10:23, Nikita Amelchev  wrote:
> 
> Hi Stephen,
> 
> Yes, I know about this. The calcite module was not published to the
> maven. The problem will be fixed at the nearest time.
> 
> пт, 29 апр. 2022 г. в 11:50, Stephen Darlington
> :
>> 
>> Hi all,
>> 
>> There’s a question in the user mailing list about the ignite-calcite module. 
>> It does not appear to be in the Maven repo. Is this a deliberate omission — 
>> because it’s an experimental feature — or did something go wrong with the 
>> release?
>> 
>> Regards,
>> Stephen
> 
> 
> 
> -- 
> Best wishes,
> Amelchev Nikita

2.13 and Calcite

[Stephen Darlington]

Hi all,

There’s a question in the user mailing list about the ignite-calcite module. It 
does not appear to be in the Maven repo. Is this a deliberate omission — 
because it’s an experimental feature — or did something go wrong with the 
release?

Regards,
Stephen

Re: CVE-2022-22963

[Stephen Darlington]

Ignite doesn’t ship with Spring Cloud, so no. There is a related vulnerability 
(CVE 2022-22965), which is in Spring Core. However,

> “In order to exploit the vulnerabilities, the following requirements must be 
> met:
> 
>   • JDK 9 or higher
>   • Apache Tomcat as the Servlet container
>   • Packaged as WAR
>   • spring-webmvc or spring-webflux dependency"

(https://sysdig.com/blog/cve-2022-22965-spring-core-spring4shell/ 
)

So, again, Ignite is not vulnerable. Having said that, if you perform an 
automated security scan it may flag it.

> On 31 Mar 2022, at 08:24, Vishwas Bm  wrote:
> 
> Hi All,
> 
> Is ignite impacted by this critical vulnerability?
> 
> https://securityboulevard.com/2022/03/cyrc-vulnerability-analysis-two-distinct-spring-vulnerabilities-discovered-spring4shell-and-cve-2022-22963/
> 
> 
> Regards,
> Vishwas

[IGNITE-16293] Support multi-platform Docker images - ASF JIRA

[Stephen Darlington]

Hi all,

I opened this ticket recently, but think it needs further discussion. 
https://issues.apache.org/jira/browse/IGNITE-16293 


The motivation for the ticket came from a user request for an image that ran 
natively on an M1 MacBook (70695484 
).
 I’m honestly quite surprised that it’s taken this long for anyone to ask! I’ve 
run Ignite on my Raspberry Pi from time-to-time and ARM-based servers are also 
becoming increasingly common.

However, this question led me down a rabbit-hole, resulting in three further 
questions:
Should we support Docker images with architectures other than x86-64?
If yes, which ones?
And how should we support them?
We currently have images for x86-64 and, for a couple of versions, additional 
images for s390x. So one argument in favour of the first point is that we 
already do! Of course, extra code requires more support and some work on the 
build servers. I think the effort is low but it’s non-zero.

I don’t think we need to go crazy here. I think adding an ARM image would open 
up support for Raspberry Pi’s and Apple Silicon Macs.

Docker also has the ability to cross-compile (working-with-buildx 
) so there’s no need for 
any specialised new hardware. (I’m unsure if this option works for s390x?)

Currently, our cross-platform support is achieved just with tags. There is a 
better way where, in addition to the tags, you add a manifest. The manifest 
allows a client to pick the right image automatically. 
(multi-platform-docker-builds 
) As an example, 
when I run this on my Raspberry Pi I get the ARMv7 image:

sudo docker run -it --rm --net=host ghcr.io/sdarlington/ignite-arm:2.12.0

But if I run the same command on an Apple Silicon Mac (I think — I don’t have 
one to test!), it would use the aarch64 image. I built both the ARMv7 and 
aarch64 images on my Intel Mac.

Thoughts?

Regards,
Stephen

Re: Move Azure, AWS, GCE to the ignite-extensions

[Stephen Darlington]

Do we know what happened with this thread? I just saw a question in the user 
list asking about where the ignite-aws-ext module is now that Ignite 2.12 is 
out.

Regards,
Stephen

> On 24 Nov 2021, at 11:11, Maxim Muzafarov  wrote:
> 
> Petr,
> 
> There is only the GCE suite left to be configured. I've created an
> issue [1] to do this. Please, take a look.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-15981
> 
> On Wed, 24 Nov 2021 at 12:00, Petr Ivanov  wrote:
>> 
>> Hi, Maksim.
>> 
>> 
>> Can you file a ticket about recreating test suites for extensions, please?
>> I will attend to it in nearest time.
>> 
>> 
>>> On 23 Nov 2021, at 17:14, Maxim Muzafarov  wrote:
>>> 
>>> Hello Petr,
>>> 
>>> Can you assist me with configuring the GCE [1] suite on the TC
>>> Extensions project? Currently, I have an issue with moving environment
>>> variables from the old GCE suite [2] to the new one.
>>> 
>>> I need to create the following envs:
>>> - env.test.gce.account.id
>>> - env.test.gce.p12.path
>>> - env.test.gce.project.name
>>> 
>>> However the `id` seems to be a password, so it's hidden on the admin
>>> panel. Can you please help me with this?
>>> 
>>> [1] 
>>> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteExtensions_Tests_Gce_IgniteExtensions_Tests=%3Cdefault%3E=buildTypeStatusDiv
>>> [2] 
>>> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_GceOld_IgniteTests24Java8=ignite-2.12=buildTypeStatusDiv
>>> 
>>> On Mon, 25 Oct 2021 at 14:22, Maxim Muzafarov  wrote:
>>>> 
>>>> Folks,
>>>> 
>>>> I've moved the azure, gce, aws modules to the ignite-extensions project.
>>>> https://issues.apache.org/jira/browse/IGNITE-15541
>>>> 
>>>> Building the modules in the ignite-extension project will prepare an
>>>> appropriate release zip file containing all the necessary
>>>> dependencies:
>>>> - ignite-aws-ext.zip
>>>> - ignite-gce-ext.zip
>>>> - ignite-auzre-ext.zip
>>>> 
>>>> 
>>>> On Wed, 13 Oct 2021 at 17:09, Stephen Darlington
>>>>  wrote:
>>>>> 
>>>>> Okay, I phrased that badly. I mean an extra platform-specific ZIP file 
>>>>> that I used to augment the generic Ignite ZIP file.
>>>>> 
>>>>> So, to run on Azure I’d download ignite.zip + azure.zip.
>>>>> 
>>>>> Extending ignite.sh would also be great, kind of like what’s happening 
>>>>> with Ignite 3 as far as I can tell.
>>>>> 
>>>>> What I’m advocating is not needing to use Maven just to run Ignite on 
>>>>> Azure, AWS, etc.
>>>>> 
>>>>>> On 13 Oct 2021, at 14:35, Petr Ivanov  wrote:
>>>>>> 
>>>>>> Our self-contained zip file currently is over 400Mb and continues to 
>>>>>> grow.
>>>>>> Even considering that internet speeds has grown too, it is nonsense to 
>>>>>> force user to download such an archive where 90% are useless for most 
>>>>>> cases.
>>>>>> 
>>>>>> Also we can:
>>>>>> — pack all extensions in single binary with latests releases (and update 
>>>>>> after each extension release) or even one by one
>>>>>> — extend ignite.sh to download remote libs when extension is activated 
>>>>>> via command line
>>>>>> 
>>>>>> 
>>>>>> Antoine de Saint-Exupéry once said that 'perfection is achieved, not 
>>>>>> when there is nothing more to add, but when there is nothing left to 
>>>>>> take away'.
>>>>>> We are not obliged to make Apache Ignite ideal, but we certainly can 
>>>>>> move that way — I am sure the result will exceed expectations.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On 13 Oct 2021, at 16:02, Stephen Darlington 
>>>>>>>  wrote:
>>>>>>> 
>>>>>>> Having extensions in Maven Central makes perfect sense for tools that 
>>>>>>> need to be built and integrated with other code, Spring integrations 
>>>>>>> for example.
>>>>>>> 
>>>>>>> That’s not the case for extensions that are required just to run 
>>&g

Re: Apache Ignite 2.11.1 RELEASE [Time, Scope, Manager]

[Stephen Darlington]

Given that 2.12 is so close, my preference would be to limit the scope of 
2.11.1 to just the log4j update. Would that help bring the vote/release date 
forward?

> On 16 Dec 2021, at 12:44, Maxim Muzafarov  wrote:
> 
> Dear Ignite Community!
> 
> I suggest preparing the Apache Ignite 2.11.1 release and I want to
> propose myself to be the release manager of the minor release.
> 
> 
> * RELEASE TIMELINE *
> 
> Scope Freeze: December 16, 2021
> Code Freeze: December 16, 2021
> Voting Date: December 21, 2021
> Release Date: December 24, 2021
> 
> 
> * RELEASE SCOPE *
> 
> LOG4J dependency update
> https://issues.apache.org/jira/browse/IGNITE-16101
> https://issues.apache.org/jira/browse/IGNITE-16127
> 
> B+Tree Corrupted exception when using a key extracted from a BinaryObject
> https://issues.apache.org/jira/browse/IGNITE-12911
> 
> Regression: Ignite node crash(CorruptedTreeException: B+Tree is corrupted)
> https://issues.apache.org/jira/browse/IGNITE-15943
> 
> .NET: ClientFailoverSocket sets logger too late, resulting in null
> loggers downstream
> https://issues.apache.org/jira/browse/IGNITE-14776
> 
> The iterator of the ClientCacheQueryCursor can be closed during serialization.
> https://issues.apache.org/jira/browse/IGNITE-15346
> 
> Possible owners desync when a node is restarted while rebalancing with
> enabled persistence
> https://issues.apache.org/jira/browse/IGNITE-15315
> 
> Thin client: Tx can fail if there are concurrent tx rollbacks by timeout
> https://issues.apache.org/jira/browse/IGNITE-15732
> 
> AssertionError: Unexpected rebalance on rebalanced cluster
> https://issues.apache.org/jira/browse/IGNITE-15033
> 
> JmxMetricExporterSpi throws assertion error on a filtered metric unregister
> https://issues.apache.org/jira/browse/IGNITE-15252
> 
> ClassNotFoundException on an attempt to invoke service method from
> Java ThinClient after a cluster failover
> https://issues.apache.org/jira/browse/IGNITE-15256
> 
> NullPointerException on an attempt to create a Java ThinClient with
> BinaryConfiguration
> https://issues.apache.org/jira/browse/IGNITE-15138
> 
> Java thin client: Type name is not cached on client-side for
> OptimizerMarshaller types
> https://issues.apache.org/jira/browse/IGNITE-15924
> 
> select count * returns multiple rows
> https://issues.apache.org/jira/browse/IGNITE-14120
> 
> Fix StackOverflowError in case if an exception is suppressed with itself
> https://issues.apache.org/jira/browse/IGNITE-15716
> 
> 
> WDYT?

Re: 0-day CVE in log4j

[Stephen Darlington]

Another workaround appears to be using the -Dlog4j2.formatMsgNoLookups=true 
option. Also, “Java versions greater than 6u211, 7u201, 8u191, and 11.0.1 are 
less affected by this attack vector, at least in theory, because the JNDI can't 
load remote code using LDAP.”

(https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/)

> On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> 
> Hi Igniters,
> 
> Preliminary: change of the log4j version does not affect any tests
> (Alexander Nikolaev, correct me if I'm wrong).
> 
> If you're using embedded Ignite, it's perfectly possible to enforce jog4j2
> dependency to be 2.15.0 in your project final pom.xml or build.gradle or
> any other build system properties.
> 
> https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> a blocker for 2.12. But for now, as a workaround, it's possible to select
> the latest version manually.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> 
>> Hello.
>> 
>> The issue to update dependency was created:
>> https://issues.apache.org/jira/browse/IGNITE-16101
>> 
>> I want to include it to the 2.12 scope.
>> 
>> сб, 11 дек. 2021 г., 09:19 Raymond Wilson :
>> 
>>> All
>>> 
>>> This blew up today: CVE-2021-44228 (
>>> 
>>> 
>> https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
>>> )
>>> 
>>> Will there be a risk assessment with respect to Ignite for this CVE?
>>> 
>>> Thanks,
>>> Raymond.
>>> 
>>> --
>>> 
>>> Raymond Wilson
>>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
>>> 11 Birmingham Drive | Christchurch, New Zealand
>>> raymond_wil...@trimble.com
>>> 
>>> <
>>> 
>> https://worksos.trimble.com/?utm_source=Trimble_medium=emailsign_campaign=Launch
 
>>> 
>> 

Re: Move Azure, AWS, GCE to the ignite-extensions

[Stephen Darlington]

Okay, I phrased that badly. I mean an extra platform-specific ZIP file that I 
used to augment the generic Ignite ZIP file.

So, to run on Azure I’d download ignite.zip + azure.zip.

Extending ignite.sh would also be great, kind of like what’s happening with 
Ignite 3 as far as I can tell.

What I’m advocating is not needing to use Maven just to run Ignite on Azure, 
AWS, etc.

> On 13 Oct 2021, at 14:35, Petr Ivanov  wrote:
> 
> Our self-contained zip file currently is over 400Mb and continues to grow.
> Even considering that internet speeds has grown too, it is nonsense to force 
> user to download such an archive where 90% are useless for most cases.
> 
> Also we can:
> — pack all extensions in single binary with latests releases (and update 
> after each extension release) or even one by one
> — extend ignite.sh to download remote libs when extension is activated via 
> command line
> 
> 
> Antoine de Saint-Exupéry once said that 'perfection is achieved, not when 
> there is nothing more to add, but when there is nothing left to take away'.
> We are not obliged to make Apache Ignite ideal, but we certainly can move 
> that way — I am sure the result will exceed expectations.
> 
> 
> 
>> On 13 Oct 2021, at 16:02, Stephen Darlington 
>>  wrote:
>> 
>> Having extensions in Maven Central makes perfect sense for tools that need 
>> to be built and integrated with other code, Spring integrations for example.
>> 
>> That’s not the case for extensions that are required just to run Ignite. A 
>> self-contained zip file for each platform would work.
>> 
>>> On 13 Oct 2021, at 13:41, Maxim Muzafarov  wrote:
>>> 
>>> Nikolay,
>>> 
>>> All extensions will be available at the maven central for download.
>>> 
>>> Previously extensions have a dependent version on the ignite core, so
>>> each time the Ignite was released it made sense to include all the
>>> extensions into the uber-zip file. Each extension has its own release
>>> version now, so an extension can be upgraded and used independently,
>>> what is the reason include it in the single uber-zip file? Probably it
>>> would be better to provide a self-contained zip file for each cloud
>>> platform.
>>> 
>>> If I've missed your issue, so can you clarify the problem in more detail?
>>> 
>>> On Wed, 13 Oct 2021 at 14:37, Nikolay Izhikov  wrote:
>>>> 
>>>> Maxim.
>>>> 
>>>>> Currently, they are copied from the optional
>>>>> directory of the ignite binary package but would be copied from an
>>>>> appropriate ignite extension binary package.
>>>> 
>>>> But how, the user will download this binary package?
>>>> Right now, all the user need is Ignite distributive.
>>>> 
>>>> 
>>>>> 13 окт. 2021 г., в 14:32, Maxim Muzafarov  написал(а):
>>>>> 
>>>>> Stephen,
>>>>> 
>>>>> I guess the required classes of IP-finders should be in the classpath
>>>>> (libs directory). Currently, they are copied from the optional
>>>>> directory of the ignite binary package but would be copied from an
>>>>> appropriate ignite extension binary package. Probably I'm missing
>>>>> something but almost nothing changes in that process from my point of
>>>>> view. The documentation pages will be updated prior to the release.
>>>>> 
>>>>> On Wed, 13 Oct 2021 at 13:44, Stephen Darlington
>>>>>  wrote:
>>>>>> 
>>>>>> I understand the motivation from a development point of view, but how 
>>>>>> will this work for end users? Currently, the documentation talks about 
>>>>>> extensions only in terms of importing maven dependencies (download.cgi 
>>>>>> <https://ignite.apache.org/download.cgi#extensions>). If I’m trying to 
>>>>>> start a cluster on Azure, how does that work? Do I need to build my own 
>>>>>> server?
>>>>>> 
>>>>>> Regards,
>>>>>> Stephen
>>>>>> 
>>>>>>> On 13 Oct 2021, at 11:35, Nikita Amelchev  wrote:
>>>>>>> 
>>>>>>> +1 to migrate and include to the Ignite 2.12 scope
>>>>>>> 
>>>>>>> пн, 20 сент. 2021 г. в 17:09, Denis Magda :
>>>>>>>> 
>>>>>>>> Perfect, thanks, Maxim!
>>>>>>>> 
>>>>>>>> -
>>>>>>>> Denis
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Sep 20, 2021 at 8:29 AM Maxim Muzafarov  
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Folks,
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I've created an issue [1] to move all cloud-based IP-finders to the
>>>>>>>>> ignite-extensions. The motivation is the same as with migration of
>>>>>>>>> Spring Data integration - to remove integration dependency of the
>>>>>>>>> release cycle on Ignite releases.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15541
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Best wishes,
>>>>>>> Amelchev Nikita
>>>>>> 
>>>>>> 
>>>> 
>> 
>> 
> 

Re: Move Azure, AWS, GCE to the ignite-extensions

[Stephen Darlington]

Having extensions in Maven Central makes perfect sense for tools that need to 
be built and integrated with other code, Spring integrations for example.

That’s not the case for extensions that are required just to run Ignite. A 
self-contained zip file for each platform would work.

> On 13 Oct 2021, at 13:41, Maxim Muzafarov  wrote:
> 
> Nikolay,
> 
> All extensions will be available at the maven central for download.
> 
> Previously extensions have a dependent version on the ignite core, so
> each time the Ignite was released it made sense to include all the
> extensions into the uber-zip file. Each extension has its own release
> version now, so an extension can be upgraded and used independently,
> what is the reason include it in the single uber-zip file? Probably it
> would be better to provide a self-contained zip file for each cloud
> platform.
> 
> If I've missed your issue, so can you clarify the problem in more detail?
> 
> On Wed, 13 Oct 2021 at 14:37, Nikolay Izhikov  wrote:
>> 
>> Maxim.
>> 
>>> Currently, they are copied from the optional
>>> directory of the ignite binary package but would be copied from an
>>> appropriate ignite extension binary package.
>> 
>> But how, the user will download this binary package?
>> Right now, all the user need is Ignite distributive.
>> 
>> 
>>> 13 окт. 2021 г., в 14:32, Maxim Muzafarov  написал(а):
>>> 
>>> Stephen,
>>> 
>>> I guess the required classes of IP-finders should be in the classpath
>>> (libs directory). Currently, they are copied from the optional
>>> directory of the ignite binary package but would be copied from an
>>> appropriate ignite extension binary package. Probably I'm missing
>>> something but almost nothing changes in that process from my point of
>>> view. The documentation pages will be updated prior to the release.
>>> 
>>> On Wed, 13 Oct 2021 at 13:44, Stephen Darlington
>>>  wrote:
>>>> 
>>>> I understand the motivation from a development point of view, but how will 
>>>> this work for end users? Currently, the documentation talks about 
>>>> extensions only in terms of importing maven dependencies (download.cgi 
>>>> <https://ignite.apache.org/download.cgi#extensions>). If I’m trying to 
>>>> start a cluster on Azure, how does that work? Do I need to build my own 
>>>> server?
>>>> 
>>>> Regards,
>>>> Stephen
>>>> 
>>>>> On 13 Oct 2021, at 11:35, Nikita Amelchev  wrote:
>>>>> 
>>>>> +1 to migrate and include to the Ignite 2.12 scope
>>>>> 
>>>>> пн, 20 сент. 2021 г. в 17:09, Denis Magda :
>>>>>> 
>>>>>> Perfect, thanks, Maxim!
>>>>>> 
>>>>>> -
>>>>>> Denis
>>>>>> 
>>>>>> 
>>>>>> On Mon, Sep 20, 2021 at 8:29 AM Maxim Muzafarov  
>>>>>> wrote:
>>>>>> 
>>>>>>> Folks,
>>>>>>> 
>>>>>>> 
>>>>>>> I've created an issue [1] to move all cloud-based IP-finders to the
>>>>>>> ignite-extensions. The motivation is the same as with migration of
>>>>>>> Spring Data integration - to remove integration dependency of the
>>>>>>> release cycle on Ignite releases.
>>>>>>> 
>>>>>>> 
>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-15541
>>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Best wishes,
>>>>> Amelchev Nikita
>>>> 
>>>> 
>> 

Re: Move Azure, AWS, GCE to the ignite-extensions

[Stephen Darlington]

I understand the motivation from a development point of view, but how will this 
work for end users? Currently, the documentation talks about extensions only in 
terms of importing maven dependencies (download.cgi 
). If I’m trying to start a 
cluster on Azure, how does that work? Do I need to build my own server?

Regards,
Stephen

> On 13 Oct 2021, at 11:35, Nikita Amelchev  wrote:
> 
> +1 to migrate and include to the Ignite 2.12 scope
> 
> пн, 20 сент. 2021 г. в 17:09, Denis Magda :
>> 
>> Perfect, thanks, Maxim!
>> 
>> -
>> Denis
>> 
>> 
>> On Mon, Sep 20, 2021 at 8:29 AM Maxim Muzafarov  wrote:
>> 
>>> Folks,
>>> 
>>> 
>>> I've created an issue [1] to move all cloud-based IP-finders to the
>>> ignite-extensions. The motivation is the same as with migration of
>>> Spring Data integration - to remove integration dependency of the
>>> release cycle on Ignite releases.
>>> 
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-15541
>>> 
> 
> 
> 
> -- 
> Best wishes,
> Amelchev Nikita

Re: [VOTE] Release pyignite 0.5.2-rc0

[Stephen Darlington]

I know you’re joking, but this iPad app claims to support third party Python 
libraries: https://apps.apple.com/gb/app/pyto-python-3/id1436650069 
 I didn’t try it but 
kind of curious now...

> On 14 Sep 2021, at 10:40, Pavel Tupitsyn  wrote:
> 
>> +1
>> Отправлено с iPhone
> 
> Nikolay, did you test pyignite from your iPhone? Does it work? :)
> 
> On Tue, Sep 14, 2021 at 12:22 PM Николай Ижиков  wrote:
> 
>> +1
>> 
>> Отправлено с iPhone
>> 
>>> 14 сент. 2021 г., в 11:39, Pavel Tupitsyn 
>> написал(а):
>>> 
>>> +1
>>> 
>>> Checked on Ubuntu 20.04, ran a few examples.
>>> 
 On Tue, Sep 14, 2021 at 11:12 AM Ivan Daschinsky 
 wrote:
 
 +1 from me
 Checked on windows 10 x86_64 (visual studio 2017) and ubuntu 20.04
>> x86_64
 and on pythons 3.6, 3.7, 3.8, 3.9 (both win and linux):
 1. Installing from source -- passe
 2. Building wheels from source -- passed
 3. Installing wheels -- passed
 
 Checked on each steps C module and examples. -- passed
 
 Checked hashsums and gpg signatures -- passed
 
 пт, 10 сент. 2021 г. в 19:08, Ivan Daschinsky :
 
> Dear Igniters!
> 
> Release candidate binaries for subj are uploaded and ready for vote
> You can find them here:
> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.2-rc0
> 
> If you follow the link above, you will find source packages (*.tar.gz
>> and
> *.zip)
> and binary packages (wheels) for windows (amd64) and linux (x86_64)
> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> Code signing keys can be found here --
> https://downloads.apache.org/ignite/KEYS
> Here you can find instructions how to verify packages
> https://www.apache.org/info/verification.html
> 
> You can install binary package for specific version of python using pip
> For example do this on linux for python 3.8
>>> pip install pyignite-0.5.2-cp38-cp38-manylinux1_x86_64.whl
> 
> You can build and install package from source using this command:
>>> pip install pyignite-0.5.2.tar.gz
> You can build wheel on your platform using this command:
>>> pip wheel --no-deps pyignite-0.5.2.tar.gz
> 
> For building C module, you should have python headers and C compiler
> installed.
> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> In Mac OS X xcode-tools and python from homebrew are the best option.
> 
> In order to check whether C module works, use following:
>>> from pyignite import _cutils
>>> print(_cutils.hashcode('test'))
>>> 3556498
> 
> You can find documentation here:
> 
 
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/
> 
> You can find examples here (to check them, you should start ignite
> locally):
> 
> 
 
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.2.rc0/examples.html
> Also, examples can be found in source archive in examples subfolder.
> docker-compose.yml is supplied in order to start ignite quickly. (Use
> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> down` to shut down it)
> 
> Release notes:
> 
> 
 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=a67624a9c141ad5457cdda1a50bd57af5ac62615;hb=1222f29abca4c44a8a2f23e413eafb6acd332e76
> 
> Git release tag was created:
> 
> 
 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.2.rc0
> 
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
> 
> +1 - to accept pyignite-0.5.2-rc0
> 0 - don't care either way
> -1 - DO NOT accept pyignite-0.5.2-rc0
> 
> The vote finishes at 09/15/2021 15:00 UTC
> 
 
 
 --
 Sincerely yours, Ivan Daschinskiy
 
>> 

Re: Support. Problem connecting Ignite with Kafka

[Stephen Darlington]

This is probably more a question for the user list than the developer list. I’m 
not sure there’s more detailed documentation but if you have specific questions 
someone might be able to help you.

> On 19 Aug 2021, at 12:48, Arnedo Nieto, Daniel  wrote:
> 
> Hello,
>  
> We are working on a project where we need to deploy Ignite in our cluster and 
> now we are exploring and discovering this technology for the first time by 2 
> ways: with a docker container of a single node of ignite and with a 2 nodes 
> cluster of ignite in k8s. We also have a Kafka cluster in our own cluster 
> integrated with a Cloudera-Hortonworks distribution, so we are trying to 
> connect Ignite with our Kafka in the Hadoop ecosystem.
>  
> We have based on the documentation but we are not able to make it work, so we 
> would like to ask for a more detailed documentation or for a more specific 
> support. Could it be possible?
>  
> Sincerely,
> Daniel
>  
>  
>  
> Daniel Arnedo Nieto
> Digital Labs
>  
> indracompany.com 
> 
>  

Re: [VOTE] Release pyignite 0.5.0-rc1

[Stephen Darlington]

You’re right, it’s documented and works as described. My bad.

> On 18 Jun 2021, at 11:24, Igor Sapego  wrote:
> 
> Well,
> 
> This behaviour maybe is not obvious, but it seems to be safe and
> should not cause user issues. On the other hand, if we change this
> behavior now, it may lead to implicit disable of SSL for users that
> updated their client which seems more dangerous to me.
> 
> So I'd keep the current behaviour.
> 
> Best Regards,
> Igor
> 
> 
> On Fri, Jun 18, 2021 at 1:17 PM Ivan Daschinsky  wrote:
> 
>> I suppose that we should not cancel this release, despite the fact that
>> this is not obvious behaviour. This is not a regression, this behaviour is
>> documented and this behaviour lasts for few years. Lets remove it, if the
>> majority are against it, but in the next release.
>> 
>> пт, 18 июн. 2021 г. в 13:08, Ivan Daschinsky :
>> 
>>>>> we explicitly set use_ssl=True.
>>> Sorry, typo -- implicitly
>>> 
>>> пт, 18 июн. 2021 г. в 12:59, Ivan Daschinsky :
>>> 
>>>> AHA! I see, this is not a bug -- this is a feature. If you pass username
>>>> and password, we explicitly set use_ssl=True. So if your cluster is
>>>> configured without ssl but with authentication,
>>>> you should explicitly pass use_ssl=False.
>>>> 
>>>> This behaviour is from old version and I suppose it is correct. Who
>> wants
>>>> authentication that sent without encryption?
>>>> 
>>>> пт, 18 июн. 2021 г. в 12:54, Ivan Daschinsky :
>>>> 
>>>>> Just rechecked test on release branch, add extra check with cluster
>>>>> activation and putting some data -- everything works ok. Authentication
>>>>> enabled, persistence enabled,
>>>>> with and without ssl. Could you please provide you ignite config and
>>>>> your code.
>>>>> 
>>>>> пт, 18 июн. 2021 г. в 12:46, Ivan Daschinsky :
>>>>> 
>>>>>> There is a test for it.
>>>>>> 
>>>>>> пт, 18 июн. 2021 г. в 12:30, Stephen Darlington <
>>>>>> stephen.darling...@gridgain.com>:
>>>>>> 
>>>>>>> Oh… can someone else check this: it appears that authenticated
>>>>>>> connections fail.
>>>>>>> 
>>>>>>> With Ignite 2.10 the connection times-out:
>>>>>>> 
>>>>>>> 
>> [10:28:58,015][WARNING][grid-timeout-worker-#22][ClientListenerNioListener]
>>>>>>> Unable to perform handshake within timeout [timeout=1,
>> remoteAddr=/
>>>>>>> 127.0.0.1:54044]
>>>>>>> 
>>>>>>> Didn’t try this with 0.4.0 so not sure if it’s a regression, but it’s
>>>>>>> not great.
>>>>>>> 
>>>>>>>> On 18 Jun 2021, at 09:36, Stephen Darlington <
>>>>>>> stephen.darling...@gridgain.com> wrote:
>>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> Checked on macOS, played with the new expiry APIs and a bunch of
>>>>>>> thefundamentals.
>>>>>>>> 
>>>>>>>>> On 17 Jun 2021, at 12:46, Pavel Tupitsyn 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> +1
>>>>>>>>> 
>>>>>>>>> Checked pip install from tar.gz on Python 3.8 on Ubuntu 20.04, ran
>>>>>>> some of
>>>>>>>>> the examples.
>>>>>>>>> 
>>>>>>>>> On Thu, Jun 17, 2021 at 2:32 PM Igor Sapego 
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> +1 from me
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Igor
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky <
>>>>>>> ivanda...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> +1 From me
>>>>>>>>>>> Checked on Ubuntu 20.04 and windows 10
>>>>>>>>>>> 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
>>>&g

Re: [VOTE] Release pyignite 0.5.0-rc1

[Stephen Darlington]

Oh… can someone else check this: it appears that authenticated connections fail.

With Ignite 2.10 the connection times-out:

[10:28:58,015][WARNING][grid-timeout-worker-#22][ClientListenerNioListener] 
Unable to perform handshake within timeout [timeout=1, 
remoteAddr=/127.0.0.1:54044]

Didn’t try this with 0.4.0 so not sure if it’s a regression, but it’s not great.

> On 18 Jun 2021, at 09:36, Stephen Darlington 
>  wrote:
> 
> +1
> 
> Checked on macOS, played with the new expiry APIs and a bunch of 
> thefundamentals.
> 
>> On 17 Jun 2021, at 12:46, Pavel Tupitsyn  wrote:
>> 
>> +1
>> 
>> Checked pip install from tar.gz on Python 3.8 on Ubuntu 20.04, ran some of
>> the examples.
>> 
>> On Thu, Jun 17, 2021 at 2:32 PM Igor Sapego  wrote:
>> 
>>> +1 from me
>>> 
>>> Best Regards,
>>> Igor
>>> 
>>> 
>>> On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky 
>>> wrote:
>>> 
>>>> +1 From me
>>>> Checked on Ubuntu 20.04 and windows 10
>>>> 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
>>>> 2. Native module work
>>>> 3. Examples
>>>> 
>>>> Checked on Ubuntu 20.04 building from source package and correct work of
>>>> result package.
>>>> 
>>>> Checked all sha256 checksums and gpg signatures.
>>>> 
>>>> Let's extend voting period till June 18, 15:00 UTC
>>>> 
>>>> 
>>>> 
>>>> ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky :
>>>> 
>>>>> The vote will end at June, 17 15:00 UTC.
>>>>> 
>>>>> ср, 16 июн. 2021 г. в 17:33, Ivan Daschinsky :
>>>>>> 
>>>>>> Dear Igniters!
>>>>>> 
>>>>>> Release candidate binaries for subj are uploaded and ready for vote
>>>>>> You can find them here:
>>>>>> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.0-rc1
>>>>>> 
>>>>>> If you follow the link above, you will find source package (*.tar.gz
>>>> and
>>>>> *.zip)
>>>>>> and binary packages (wheels) for windows (amd64) and linux (x86_64)
>>>>>> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg
>>> signatures.
>>>>>> Code signing keys can be found here --
>>>>> https://downloads.apache.org/ignite/KEYS
>>>>>> Here you can find instructions how to verify packages
>>>>>> https://www.apache.org/info/verification.html
>>>>>> 
>>>>>> You can install binary package for specific version of python using
>>> pip
>>>>>> For example do this on linux for python 3.8
>>>>>>>> pip install pyignite-0.5.0-cp38-cp38-manylinux1_x86_64.whl
>>>>>> 
>>>>>> You can build and install package from source using this command:
>>>>>>>> pip install pyignite-0.5.0.tar.gz
>>>>>> You can build wheel on your platform using this command:
>>>>>>>> pip wheel --no-deps pyignite-0.5.0.tar.gz
>>>>>> 
>>>>>> For building C module, you should have python headers and C compiler
>>>>> installed.
>>>>>> (i.e. for ubuntu sudo apt install build-essential python3-dev)
>>>>>> In Mac OS X xcode-tools and python from homebrew are the best option.
>>>>>> 
>>>>>> In order to check whether C module works, use following:
>>>>>>>> from pyignite import _cutils
>>>>>>>> print(_cutils.hashcode('test'))
>>>>>>>> 3556498
>>>>>> 
>>>>>> You can find documentation here:
>>>>>> 
>>>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1
>>>>>> 
>>>>>> You can find examples here (to check them, you should start ignite
>>>>> locally):
>>>>>> 
>>>>> 
>>>> 
>>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1/examples.html
>>>>>> Also, examples can be found in source archive in examples subfolder.
>>>>>> docker-compose.yml is supplied in order to start ignite quickly. (Use
>>>>>> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
>>>>>> down` to shut down it)
>>>>>> 
>>>>>> Release notes:
>>>>>> 
>>>>> 
>>>> 
>>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9d2ae81af2de22ce9e8c9d3b7ece14dd9e75ca0e;hb=61c83cb0ab6752f019518b4a2cb0724bd027755f
>>>>>> 
>>>>>> Git release tag was created:
>>>>>> 
>>>>> 
>>>> 
>>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.0.rc1
>>>>>> 
>>>>>> The vote is formal, see voting guidelines
>>>>>> https://www.apache.org/foundation/voting.html
>>>>>> 
>>>>>> +1 - to accept pyignite-0.5.0-rc1
>>>>>> 0 - don't care either way
>>>>>> -1 - DO NOT accept pyignite-0.5.0-rc1
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Sincerely yours, Ivan Daschinskiy
>>>> 
>>> 
> 
> 

Re: [VOTE] Release pyignite 0.5.0-rc1

[Stephen Darlington]

+1

Checked on macOS, played with the new expiry APIs and a bunch of 
thefundamentals.

> On 17 Jun 2021, at 12:46, Pavel Tupitsyn  wrote:
> 
> +1
> 
> Checked pip install from tar.gz on Python 3.8 on Ubuntu 20.04, ran some of
> the examples.
> 
> On Thu, Jun 17, 2021 at 2:32 PM Igor Sapego  wrote:
> 
>> +1 from me
>> 
>> Best Regards,
>> Igor
>> 
>> 
>> On Thu, Jun 17, 2021 at 12:10 PM Ivan Daschinsky 
>> wrote:
>> 
>>> +1 From me
>>> Checked on Ubuntu 20.04 and windows 10
>>> 1. Installation from wheels for pythons 3.6 3.7 3.8 3.9
>>> 2. Native module work
>>> 3. Examples
>>> 
>>> Checked on Ubuntu 20.04 building from source package and correct work of
>>> result package.
>>> 
>>> Checked all sha256 checksums and gpg signatures.
>>> 
>>> Let's extend voting period till June 18, 15:00 UTC
>>> 
>>> 
>>> 
>>> ср, 16 июн. 2021 г. в 17:34, Ivan Daschinsky :
>>> 
 The vote will end at June, 17 15:00 UTC.
 
 ср, 16 июн. 2021 г. в 17:33, Ivan Daschinsky :
> 
> Dear Igniters!
> 
> Release candidate binaries for subj are uploaded and ready for vote
> You can find them here:
> https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.5.0-rc1
> 
> If you follow the link above, you will find source package (*.tar.gz
>>> and
 *.zip)
> and binary packages (wheels) for windows (amd64) and linux (x86_64)
> for pythons 36, 37, 38, 39. Also, there are sha512 and gpg
>> signatures.
> Code signing keys can be found here --
 https://downloads.apache.org/ignite/KEYS
> Here you can find instructions how to verify packages
> https://www.apache.org/info/verification.html
> 
> You can install binary package for specific version of python using
>> pip
> For example do this on linux for python 3.8
>>> pip install pyignite-0.5.0-cp38-cp38-manylinux1_x86_64.whl
> 
> You can build and install package from source using this command:
>>> pip install pyignite-0.5.0.tar.gz
> You can build wheel on your platform using this command:
>>> pip wheel --no-deps pyignite-0.5.0.tar.gz
> 
> For building C module, you should have python headers and C compiler
 installed.
> (i.e. for ubuntu sudo apt install build-essential python3-dev)
> In Mac OS X xcode-tools and python from homebrew are the best option.
> 
> In order to check whether C module works, use following:
>>> from pyignite import _cutils
>>> print(_cutils.hashcode('test'))
>>> 3556498
> 
> You can find documentation here:
> 
>>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1
> 
> You can find examples here (to check them, you should start ignite
 locally):
> 
 
>>> 
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.5.0.rc1/examples.html
> Also, examples can be found in source archive in examples subfolder.
> docker-compose.yml is supplied in order to start ignite quickly. (Use
> `docker-compose up -d` to start 3 nodes cluster and `docker-compose
> down` to shut down it)
> 
> Release notes:
> 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9d2ae81af2de22ce9e8c9d3b7ece14dd9e75ca0e;hb=61c83cb0ab6752f019518b4a2cb0724bd027755f
> 
> Git release tag was created:
> 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.5.0.rc1
> 
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
> 
> +1 - to accept pyignite-0.5.0-rc1
> 0 - don't care either way
> -1 - DO NOT accept pyignite-0.5.0-rc1
 
>>> 
>>> 
>>> --
>>> Sincerely yours, Ivan Daschinskiy
>>> 
>> 

[IGNITE-7641] Add CacheEntry#ttl method

[Stephen Darlington]

Hi all,

I created a pull request that creates new method on IgniteCache that returns 
the current TTL of a record. I’m not entirely clear who I should tag for a 
review, so I’m posting here.

> https://issues.apache.org/jira/browse/IGNITE-7641 
> 
I 
https://github.com/apache/ignite/pull/8780

I decided to implement as a method on IgniteCache rather than CacheEntry to 
avoid having to copy the whole record over the network just to figure out the 
TTL. Unless I missed something, it didn’t appear to be as simple to implement 
as the comments in the ticket would make you think!

Thoughts?

Regards,
Stephen

Re: [DISCUSS] IEP-71 Public API for secondary index search

[Stephen Darlington]

Is this a replacement for IndexingSpi? Put bluntly, do we deprecate (and 
remove) it?

Or do you see them as complimentary?

> On 12 Apr 2021, at 11:29, Maksim Timonin  wrote:
> 
> Hi Stephen!
> 
> Please have a look at the QueryProcessing paragraph [1]. I've described
> why IndexingSpi doesn't fit us well.
> 
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-71+Public+API+for+secondary+index+search#IEP71PublicAPIforsecondaryindexsearch-2)QueryProcessing
> 
> On Mon, Apr 12, 2021 at 1:24 PM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> How does this fit with the current IndexingSpi? Superficially they appear
>> to do very similar things?
>> 
>> Regards,
>> Stephen
>> 
>>> On 6 Apr 2021, at 14:13, Maksim Timonin  wrote:
>>> 
>>> Hi, Igniters!
>>> 
>>> I'd like to propose a new feature - opportunity to query and create
>> indexes
>>> from public API.
>>> 
>>> It will help in some cases, where:
>>> 1. SQL is not applicable by design of user application;
>>> 2. Where IndexScan is preferable than ScanQuery for performance reasons;
>>> 3. Functional indexes are required.
>>> 
>>> Also it'll be great to have a transactional support for such queries,
>> like
>>> the "select for update" query provides. But I don't dig there much. It
>> will
>>> be a next step if this API will be implemented.
>>> 
>>> I've prepared an IEP-71 for that [1] with more details. Please share your
>>> thoughts.
>>> 
>>> 
>>> [1]
>>> 
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-71+Public+API+for+secondary+index+search
>> 
>> 
>> 

Re: [DISCUSS] IEP-71 Public API for secondary index search

[Stephen Darlington]

How does this fit with the current IndexingSpi? Superficially they appear to do 
very similar things?

Regards,
Stephen

> On 6 Apr 2021, at 14:13, Maksim Timonin  wrote:
> 
> Hi, Igniters!
> 
> I'd like to propose a new feature - opportunity to query and create indexes
> from public API.
> 
> It will help in some cases, where:
> 1. SQL is not applicable by design of user application;
> 2. Where IndexScan is preferable than ScanQuery for performance reasons;
> 3. Functional indexes are required.
> 
> Also it'll be great to have a transactional support for such queries, like
> the "select for update" query provides. But I don't dig there much. It will
> be a next step if this API will be implemented.
> 
> I've prepared an IEP-71 for that [1] with more details. Please share your
> thoughts.
> 
> 
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-71+Public+API+for+secondary+index+search

[jira] [Created] (IGNITE-14326) Add CacheEntry#setTtl method

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-14326:
---

 Summary: Add CacheEntry#setTtl method
 Key: IGNITE-14326
 URL: https://issues.apache.org/jira/browse/IGNITE-14326
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.3
Reporter: Stephen Darlington
Assignee: Stephen Darlington


Ignite provides a way to specify an expiry policy on per entry level, but there 
is no way to know the current TTL for a particular key.

We can add {{CacheEntry#ttl()}} and/or {{IgniteCache#ttl(K key)}} method that 
will provide this information. Looks like it's already available via 
{{GridCacheMapEntry#ttl()}}, so we just need to properly expose it to public 
API.

Here is the user forum discussion about this: 
http://apache-ignite-users.70518.x6.nabble.com/Get-TTL-of-the-specific-K-V-entry-td19817.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14220) Shuffle server addresses for thin clients

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-14220:
---

 Summary: Shuffle server addresses for thin clients
 Key: IGNITE-14220
 URL: https://issues.apache.org/jira/browse/IGNITE-14220
 Project: Ignite
  Issue Type: Improvement
  Components: thin client
Affects Versions: 2.9.1
Reporter: Stephen Darlington


To connect to a grid from a thin-client, we provide a list of servers and 
ports. If partition awareness is disabled, Ignite connects to the first node.

In the case where there are lots of thin clients, they all end up connecting to 
the same server node.

This is improved somewhat when partition awareness is enabled. JCache requests 
go directly to the data node in question. But SQL queries will all be sent to 
the same node (since the algorithm is deterministic).

If the list were "shuffled" before use, the load would end up balanced across 
the cluster. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 3.x: order in repository

[Stephen Darlington]

I think we need to be able answer the question “Why are there so many inactive 
PRs?" before we automate their removal. If perfectly good changes are being 
ignored, we have a problem.

Removing branches of merged PRs and protecting the main branch make sense.

> On 20 Feb 2021, at 18:30, Pavel Tupitsyn  wrote:
> 
> +1
> 
> - Close inactive PRs (1 month or so?)
> - Enable main branch protection (no force pushes, require linear history,
> require status checks)
> 
> On Sat, Feb 20, 2021 at 2:31 PM Petr Ivanov  wrote:
> 
>> Hi, Igniters!
>> 
>> 
>> When we started Ignite 3.x in new repository, not only we have received a
>> chance to cleanup codebase, but to maintain some order in development
>> tools, like GitHub.
>> Currently in 2.x repository we have lots of stalled PRs and branches,
>> which not only clog the repository, but also indirectly influence TC
>> performance (due to necessity to check for updates every ref: branches and
>> PRs).
>> 
>> Could I suggest we devise some recommendations for using PR's and branches
>> in new repo and add some rules about stalled PRs at least, like closing
>> them if inactive for some time.
>> Also we can activate some settings in repo's configuration, like auto
>> delete branch after PR is merged.
>> 
>> 
>> WDYT?

[jira] [Created] (IGNITE-14067) CREATE TABLE with template doesn't use isEncryptionEnabled flag

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-14067:
---

 Summary: CREATE TABLE with template doesn't use 
isEncryptionEnabled flag
 Key: IGNITE-14067
 URL: https://issues.apache.org/jira/browse/IGNITE-14067
 Project: Ignite
  Issue Type: Bug
  Components: cache, sql
Affects Versions: 2.9.1, 2.8.1, 2.9
Reporter: Stephen Darlington
Assignee: Stephen Darlington


If I create a cache template:

{{CacheConfiguration encryptedCacheTemplate = new CacheConfiguration();}}
{{encryptedCacheTemplate.setName("EncryptedCacheTemplate")}}
{{ .setEncryptionEnabled(true)}}
{{ .setBackups(1)}}
{{ .setCacheMode(CacheMode.PARTITIONED);}}
{{ignite.addCacheConfiguration(encryptedCacheTemplate);}}

And then try to use it:

{{create table ignite_enc (id long primary key, name varchar) with 
"template=EncryptedCacheTemplate";}}

The encryption setting is not used.

Workaround: specify the encryption flag "manually":

{{create table ignite_enc2 (id long primary key, name varchar) with 
"template=EncryptedCacheTemplate,encrypted=true"}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13996) JMX beans in a format not understood by Prometheus

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13996:
---

 Summary: JMX beans in a format not understood by Prometheus
 Key: IGNITE-13996
 URL: https://issues.apache.org/jira/browse/IGNITE-13996
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.9.1, 2.8.1, 2.9
Reporter: Stephen Darlington


When I connect Ignite to Prometheus' JMX exporter, I get the following 
exception:

{{ }}
{{ Jan 13, 2021 5:31:49 PM 
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector collect}}
{{ SEVERE: JMX scrape failed: java.lang.IllegalArgumentException: Not an 
Attribute: 
javax.management.openmbean.TabularDataSupport(tabularType=javax.management.openmbean.TabularType(name=org.apache.ignite.spi.systemview.view.ScanQueryView,rowType=javax.management.openmbean.CompositeType(name=org.apache.ignite.spi.systemview.view.ScanQueryView,items=((itemName=cacheGroupId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=cacheGroupName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=cacheId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=cacheName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=canceled,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=duration,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=filter,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=keepBinary,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=local,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=originNodeId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=pageSize,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=partition,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=queryId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=startTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=subjectId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=systemViewRowId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=taskName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=topology,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=transformer,itemType=javax.management.openmbean.SimpleType(name=java.lang.String,indexNames=(systemViewRowId)),contents={})}}
{{ at javax.management.AttributeList.adding(AttributeList.java:328)}}
{{ at javax.management.AttributeList.adding(AttributeList.java:335)}}
{{ at javax.management.AttributeList.asList(AttributeList.java:165)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:156)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:117)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:460)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)}}
{{ at 
io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:59)}}
{{ at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)}}
{{ at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)}}
{{ at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)}}
{{ at 
sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)}}
{{ at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)}}
{{ at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)}}
{{ at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{  }}

And only the basic JVM metrics appear in Prometheus. A workaround would be to 
use OpenCensus, but a lot of people seem to prefer JMX.

It's not clear to me Ignite's JMX output is incorrect ([Prometheus' developers 
don't seem keen to resolve on their 
side|https://github.com/prometheus/jmx_exporter/issues/483]) but ideally, 
Ignite would work correctly with a common monitoring tool.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Define Apache Ignite as a Distributed Database

[Stephen Darlington]

-1

I think the compute APIs are an important part of Ignite’s value. Calling it a 
database diminishes that in my opinion.

> On 24 Nov 2020, at 08:40, Ivan Pavlukhin  wrote:
> 
> +1
> 
> 2020-11-24 11:33 GMT+03:00, Anton Vinogradov :
>> +1
>> 
>> On Tue, Nov 24, 2020 at 10:24 AM Pavel Tupitsyn 
>> wrote:
>> 
>>> +1
>>> 
>>> On Tue, Nov 24, 2020 at 3:25 AM Saikat Maitra 
>>> wrote:
>>> 
 +1
 
 On Mon, Nov 23, 2020 at 4:55 PM Valentin Kulichenko <
 valentin.kuliche...@gmail.com> wrote:
 
> +1
> 
> On Mon, Nov 23, 2020 at 2:44 PM Denis Magda 
> wrote:
> 
>> Igniters,
>> 
>> With this vote, I'd like to formally wrap up our discussion on
>>> defining
>> Ignite as a "distributed database" instead of an "in-memory
>>> computing"
>> platform. See the following discussion for the rationale and
>> context:
>> 
>> 
> 
 
>>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Renaming-Ignite-s-product-category-td49246.html
>> 
>> If the vote passes, the website will define Ignite as "a
>> distributed
>> database for in-memory speed at petabyte scale" to underscore our
> in-memory
>> origins (and the primary reason why the project is selected) but
>> not
>> diminishing our persistence capabilities. All prominent use cases
>>> such
 as
>> caching, high-performance computing, etc. will remain visible on
>> the
>> website. There is nothing wrong if a distributed database is used
>> as
>>> a
>> cache or high-performance compute cluster; it's up to an
>> application
>> developer to decide.
>> 
>> Overall, please cast your vote for defining Ignite as a
>> "distributed
>> database":
>> +1 - support the change
>> -1 - disagree with the change, explain why
>> 0 - neutral
>> 
>> This is a majority vote that is open for the next 7 days and to be
 closed
>> on Monday, Nov 30th:
>> https://www.timeanddate.com/countdown/to?year=2020=11=30
>> 
>> -
>> Denis
>> 
> 
 
>>> 
>> 
> 
> 
> -- 
> 
> Best regards,
> Ivan Pavlukhin

Re: Using deprecated active() and unhelpful message in console

[Stephen Darlington]

Great observations! Did you open a ticket? Are you interested in contributing a 
patch?

> On 8 Oct 2020, at 05:41, Ilya Kazakov  wrote:
> 
> Hello!
> 
> As a new user of Ignite I faced this problem. When cluster started in
> INACTIVE state in logs writes uninformative message:
> 
 Ignite cluster is not active (limited functionality available). Use
> control.(sh|bat) script or IgniteCluster interface to activate.
> 
> This message contains two words "active" and does not contain the word
> "state".
> From this message, users try to use deprecated IgniteCluster.active(boolean
> active), but need to use IgniteCluster.state(ClusterState newState)
> 
> I propose to change this message. There are two options:
 Ignite cluster is in INACTIVE state (limited functionality available).
> Use control.(ch|bat) script or IgniteCluster interface to change the state.
> or
 Ignite cluster is in INACTIVE state (limited functionality available).
> Use control.(ch|bat) script or IgniteCluster.state() to change the state.
> 
> Also in the class IgniteKernal there are 3 usages of deprecated methods
> active() and active(boolean b).
> I propose to change
> - each using .active() to .state() != ClusterState.INACTIVE
> - each using .active(boolean b) to .state(b ? ClusterState.ACTIVE :
> ClusterState.INACTIVE)
> 
> -
> Ilya Kazakov

[jira] [Created] (IGNITE-13503) On-heap cache eviction policy based on size rather than number of records

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13503:
---

 Summary: On-heap cache eviction policy based on size rather than 
number of records
 Key: IGNITE-13503
 URL: https://issues.apache.org/jira/browse/IGNITE-13503
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Stephen Darlington


Currently, you can set the maximum size of an on-heap cache strictly by the 
number of records it holds. In cases where records vary substantially in size 
this could result in the cache being far bigger or far smaller than anticipated.

It would be nice to have the option to specify a maximum size in bytes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13502) Events when off-heap eviction is triggered

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13502:
---

 Summary: Events when off-heap eviction is triggered
 Key: IGNITE-13502
 URL: https://issues.apache.org/jira/browse/IGNITE-13502
 Project: Ignite
  Issue Type: Improvement
  Components: cache, persistence
Reporter: Stephen Darlington


If I configure my data region with an eviction policy, it's not currently 
possible to see when it is triggered or which records were affected.

There is EVT_CACHE_ENTRY_EVICTED, but I understand that this is for _on_-heap 
eviction (certainly it's not triggered in my tests).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13468) Destroying a caches doesn't clean up cache directory

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13468:
---

 Summary: Destroying a caches doesn't clean up cache directory
 Key: IGNITE-13468
 URL: https://issues.apache.org/jira/browse/IGNITE-13468
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.8.1
Reporter: Stephen Darlington


If I create a cache – either with SQL or with Ignite#getOrCreateCache() – in a 
data region with persistence enabled, destroying the cache removes all the data 
files but not the directory in which they're stored.

For example. Before:

{{$ ls cache*}}
{{ cache-DESTROY_DEMO:}}
{{ cache_data.dat index.bin part-2.bin part-3.bin part-4.bin}}

 

{{cache-ignite-sys-cache:}}
{{ cache_data.dat index.bin}}
{{ $}}

After:

{{$ ls cache*}}
{{ cache-DESTROY_DEMO:}}

 

{{cache-ignite-sys-cache:}}
{{ cache_data.dat index.bin}}
{{ $}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: critical security vulnerability for /opt/ignite/apache-ignite/libs/optional/ignite-rest-http/log4j-1.2.17.jar

[Stephen Darlington]

https://issues.apache.org/jira/browse/IGNITE-13464

> On 21 Sep 2020, at 11:02, Ilya Kasnacheev  wrote:
> 
> Hello!
> 
> Good catch! I think you should file a critical level ticket about it.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пн, 21 сент. 2020 г. в 12:56, Stephen Darlington 
> mailto:stephen.darling...@gridgain.com>>:
> Actually, this is an interesting one: it’s not the top level ignite-log4j 
> module, but a dependency of ignite-rest-http. Why does the REST API have 
> log4j (and slf4j) dependencies at all?
> 
>> On 21 Sep 2020, at 10:19, Ilya Kasnacheev > <mailto:ilya.kasnach...@gmail.com>> wrote:
>> 
>> Hello!
>> 
>> Log4J 1.x does not have any non-vulnerable releases, and Log4J2 is not 
>> binary compatible.
>> 
>> You can sidestep this by not including ignite-log4j module and instead 
>> resorting to ignite-log4j2.
>> 
>> Regards,
>> -- 
>> Ilya Kasnacheev
>> 
>> 
>> сб, 19 сент. 2020 г. в 01:47, Andrew Story > <mailto:andrewst...@fico.com>>:
>> Would it be possible in the next release of Ignite to upgrade the 3rd party
>> component
>> /opt/ignite/apache-ignite/libs/optional/ignite-rest-http/log4j-1.2.17.jar to
>> log4j-core-2.13.3.jar?
>> 
>> This component log4j-1.2.17.jar is flagged as having a critical security
>> vulnerability which is described here:
>> https://nvd.nist.gov/vuln/detail/CVE-2019-17571 
>> <https://nvd.nist.gov/vuln/detail/CVE-2019-17571>
>> 
>> The latest version of this component appears to be 2.13.3 which should
>> resolve the vulnerability:
>> https://logging.apache.org/log4j/2.x/download.html 
>> <https://logging.apache.org/log4j/2.x/download.html>.
>> 
>> Thanks,
>> 
>> Andrew Story
>> 
>> 
>> 
>> 
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 
>> <http://apache-ignite-users.70518.x6.nabble.com/>
> 
> 

[jira] [Created] (IGNITE-13464) Ignite-rest-http includes vulnerable dependencies

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13464:
---

 Summary: Ignite-rest-http includes vulnerable dependencies
 Key: IGNITE-13464
 URL: https://issues.apache.org/jira/browse/IGNITE-13464
 Project: Ignite
  Issue Type: Bug
  Components: rest
Affects Versions: 2.8.1, 2.9
Reporter: Stephen Darlington


The ignite-rest-http module includes a [vulnerable 
version|https://nvd.nist.gov/vuln/detail/CVE-2019-17571] of the log4j library. 
It also appears to include slf4j. Why does the REST API include its own logging 
libraries?

This was spotted in 2.8.1 but still appears to be an issue in master and 2.9.

More here:

http://apache-ignite-users.70518.x6.nabble.com/critical-security-vulnerability-for-opt-ignite-apache-ignite-libs-optional-ignite-rest-http-log4j-1-r-td34031.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Renaming Ignite's product category

[Stephen Darlington]

I think this is a great question. Explaining what Ignite does is always a 
challenge, so having a useful “tag line” would be very valuable.

I’m not sure what the answer is but I think calling it a “database” devalues 
all the compute facilities. "Computing platform” may be too vague but it at 
least says that we do more than “just” store data.

> On 17 Sep 2020, at 06:29, Valentin Kulichenko  
> wrote:
> 
> My vote is for the "distributed memory-first database". It clearly states 
> that Ignite is a database (which is true at this point), while still 
> emphasizing the in-memory computing power endorsed by the platform.
> 
> The "in-memory computing platform" is an ambiguous term and doesn't really 
> reflect what Ignite is, especially in its current state.
> 
> -Val
> 
> On Wed, Sep 16, 2020 at 3:53 PM Denis Magda  > wrote:
> Igniters,
> 
> Throughout the history of our project, we could see how the addition of
> certain features required us to reassess the project's name and category.
> 
> Before Ignite joined the ASF, it supported only compute APIs resembling the
> MapReduce engine of Hadoop. Those days, it was fair to define Ignite as "a
> distributed in-memory computing engine". Next, at the time of the project
> donation, it already included key-value/SQL/transactional APIs, was used as
> a distributed cache, and significantly outgrew the "in-memory computing
> engine" use case. That's how the project transitioned to the product
> category of in-memory caches and we started to name it as an "in-memory
> data grid" or "in-memory computing platform" to differentiate from
> classical caching products such as Memcached and Redis.
> 
> Nowadays, the project outgrew its caching use case, and the classification
> of Ignite as an "in-memory data grid" or "in-memory computing platform"
> doesn't sound accurate. We rebuilt our storage engine by replacing a
> typical key-value engine with a B-tree engine that spans across memory and
> disk tiers. And it's not surprising to see more deployments of Ignite as a
> database on its own. So, it feels like we need to reconsider Ignite
> positioning again so that a) application developers can discover it easily
> via search engines and b) the project can stand out from in-memory projects
> with intersecting capabilities.
> 
> To the point, I'm suggesting to reposition Ignite in one of the following
> ways:
> 
>1. Ignite is a "distributed X database". We are indeed a distributed
>partitioned database where X can be "multi-tiered" or "memory-first" to
>emphasize that we are more than an in-memory database.
>2. Keep defining Ignite as "an in-memory computing platform" but name
>our storage engine uniquely as "IgniteDB" to highlight that the platform is
>powered by a "distributed multi-tiered/memory-first database".
> 
> What is your thinking?
> 
> 
> (Also, regardless of a selected name, Ignite still will be used as a cache
> and grid, and we're not going to stop appealing to those use cases. But
> those are just use cases while Ignite has to figure out its new identity
> ... again).
> 
> 
> -
> Denis

[IGNITE-13404] ignite.sh fails to set default JVM parameters - ASF JIRA

[Stephen Darlington]

Hi all,

I found and created a patch for an issue I discovered last week. The ignite.sh 
script tries to set default JVM memory parameters if the user didn’t set 
JVM_OPTS, only those lines of code never get called.

> https://issues.apache.org/jira/browse/IGNITE-13404 
> 


What else do I need to do?

(I think there’s possibly another question about whether they’re the correct 
parameters, but I didn’t change them here.)

Regards,
Stephen

Re: [DISCUSSION] Output IgniteSystemProperties via ignite.sh

[Stephen Darlington]

But to Ilya’s point, what’s the logic? Why are some things set using IGNITE_ 
properties, others on the command line and others in IgniteConfiguration? It’s 
confusing for the user and makes maintenance harder.

I’m not necessarily arguing against this change, though. Perfect being the 
enemy of good and all that.

> On 7 Sep 2020, at 13:02, Nikolay Izhikov  wrote:
> 
> Hello, Ilya.
> 
>> I think this is a bad idea since it legitimizes wide use of IGNITE_ 
>> properties, which shows weakness of our configuration API, etc.
> 
> We already have IGNITE options in the product as a part of public API. See 
> `org.apache.ignite.IgniteSystemProperties`.
> 
>> 7 сент. 2020 г., в 14:40, Ilya Kasnacheev  
>> написал(а):
>> 
>> Hello!
>> 
>> I think this is a bad idea since it legitimizes wide use of IGNITE_
>> properties, which shows weakness of our configuration API, etc.
>> 
>> My take:
>> 
>> All of IGNITE_ properties which are useful (and will go to -X) should
>> instead be turned into configuration/metastore settings.
>> All of IGNITE_ properties which are dangerous and/or useless should be
>> removed.
>> 
>> Regards,
>> -- 
>> Ilya Kasnacheev
>> 
>> 
>> пт, 21 авг. 2020 г. в 16:50, Nikolay Izhikov :
>> 
>>> Hello, Igniters.
>>> 
>>> For now, we have dozens of the `IgniteSystemProperties` [1]  that can
>>> tweak Ignite behaviour in the very wide limits.
>>> But, the issue, for the administrator is the following
>>> 
>>> - Documentation about existing properties can be outdated.
>>> - The only place of the truth is the source code.
>>> - It’s hard to understand what flag is supported in what version.
>>> 
>>> I propose to implement output of all available properties with it’s
>>> descriptions in the `ignite.sh -X` command.
>>> 
>>> Example of the JVM output:
>>> 
>>> ```
>>> [16:25:49]~/src/ignite:[master]$ java -X
>>> 
>>>   -Xbatch   disable background compilation
>>>   -Xbootclasspath/a:
>>> append to end of bootstrap class path
>>>   -Xcheck:jni   perform additional checks for JNI functions
>>>   -Xcompforces compilation of methods on first invocation
>>>   -Xdebug   provided for backward compatibility
>>>   -Xdiagshow additional diagnostic messages
>>>   -Xfuture  enable strictest checks, anticipating future default
>>>   -Xint interpreted mode execution only
>>>   -Xinternalversion
>>> displays more detailed JVM version information than
>>> the
>>> 
>>> [16:28:45]~/src/ignite:[master]$ java -XX:+UnlockDiagnosticVMOptions
>>> -XX:+PrintFlagsFinal -version
>>> 
>>> [Global flags]
>>> ccstrlist AOTLibrary   =
>>>{product} {default}
>>>bool AbortVMOnCompilationFailure  = false
>>> {diagnostic} {default}
>>>   ccstr AbortVMOnException   =
>>> {diagnostic} {default}
>>>   ccstr AbortVMOnExceptionMessage=
>>> {diagnostic} {default}
>>>bool AbortVMOnSafepointTimeout= false
>>> {diagnostic} {default}
>>>bool AbortVMOnVMOperationTimeout  = false
>>> {diagnostic} {default}
>>>intx AbortVMOnVMOperationTimeoutDelay = 1000
>>>{diagnostic} {default}
>>> int ActiveProcessorCount = -1
>>>   {product} {default}
>>> 
>>> ```
>>> 
>>> Example of the Ignite output:
>>> 
>>> 
 ignite.sh -X
>>> IGNITE_CONFIG_URL
>>>-   System property to hold optional configuration URL.
>>> IGNITE_SSH_HOST -
>>>System property to hold SSH host for visor-started nodes.
>>> IGNITE_MIN_BUFFERED_COMMUNICATION_MSG_CNT   -   [DEPRECATED]
>>> System property to disable buffered communication if node sends less
>>> messages count than specified by this property. Default value is {@code
>>> 512}.
>>> 
>>> …
>>> 
>>> ```
>>> 
>>> WDYT?
>>> 
>>> [1]
>>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java#L56
> 

[jira] [Created] (IGNITE-13404) ignite.sh fails to set default JVM parameters

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13404:
---

 Summary: ignite.sh fails to set default JVM parameters
 Key: IGNITE-13404
 URL: https://issues.apache.org/jira/browse/IGNITE-13404
 Project: Ignite
  Issue Type: Bug
 Environment: Anything but Windows.
Reporter: Stephen Darlington
Assignee: Stephen Darlington


ignite.sh [tries to set parameters for the 
JVM|https://github.com/apache/ignite/blob/master/bin/ignite.sh#L102] if they've 
not been set by the user (defaulting to 1Gb heap), unfortunately it's always 
set by an earlier part of the startup scripts.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: A few small pull requests

[Stephen Darlington]

I had the same problem with “your" link… not sure what is going on!

I’ve added suggestions for documentation in the ticket as suggested.

Regards,
Stephen

> On 17 Jul 2020, at 01:00, Denis Magda  wrote:
> 
> Weird, works now. Seems to be some browser-related issue on my end.
> 
> Overall, thanks a lot for the contribution. I'm suggesting us to merge
> IGNITE-11715 into the Ignite 2.9 release branch. Left details in JIRA.
> 
> -
> Denis
> 
> 
> On Thu, Jul 16, 2020 at 9:44 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> This link works for me: https://issues.apache.org/jira/browse/IGNITE-11715
>> <https://issues.apache.org/jira/browse/IGNITE-11715>
>> 
>>> On 16 Jul 2020, at 16:47, Denis Magda  wrote:
>>> 
>>> Hi Stephen,
>>> 
>>> Thanks for reminding us about these pending contributions.
>>> 
>>> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
>>>> https://github.com/apache/ignite/pull/6877>
>>> 
>>> 
>>> Merged the changes. Thanks!
>>> 
>>> IGNITE-12182 ExecutorService defaults to only server nodes <
>>>> https://github.com/apache/ignite/pull/6876>
>>> 
>>> 
>>> I'm afraid we can't merge this change until Ignite 3.0. Otherwise, it
>> will
>>> break backward compatibility.
>>> 
>>> IGNITE-11715 Allow extra options to be passed in to ignite.sh from
>> Docker <
>>>> https://github.com/apache/ignite/pull/6552>
>>> 
>>> 
>>> Could you confirm that the JIRA ticket is correct? Whenever I try to open
>>> IGNITE-11715 in JIRA it redirects me to
>>> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-13264
>>> 
>>> 
>>> -
>>> Denis
>>> 
>>> 
>>> On Wed, Jul 15, 2020 at 6:42 AM Stephen Darlington <
>>> stephen.darling...@gridgain.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> I have a few small quality-of-life pull requests languishing in the
>>>> backlog. Can anyone take a look or suggest what else I need to do to
>>>> progress them?
>>>> 
>>>> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
>>>> https://github.com/apache/ignite/pull/6877>
>>>> IGNITE-12182 ExecutorService defaults to only server nodes <
>>>> https://github.com/apache/ignite/pull/6876>
>>>> IGNITE-11715 Allow extra options to be passed in to ignite.sh from
>> Docker <
>>>> https://github.com/apache/ignite/pull/6552>
>>>> 
>>>> Thanks,
>>>> Stephen
>>>> 
>>>> 
>> 
>> 
>> 

Re: A few small pull requests

[Stephen Darlington]

This link works for me: https://issues.apache.org/jira/browse/IGNITE-11715 
<https://issues.apache.org/jira/browse/IGNITE-11715> 

> On 16 Jul 2020, at 16:47, Denis Magda  wrote:
> 
> Hi Stephen,
> 
> Thanks for reminding us about these pending contributions.
> 
> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
>> https://github.com/apache/ignite/pull/6877>
> 
> 
> Merged the changes. Thanks!
> 
> IGNITE-12182 ExecutorService defaults to only server nodes <
>> https://github.com/apache/ignite/pull/6876>
> 
> 
> I'm afraid we can't merge this change until Ignite 3.0. Otherwise, it will
> break backward compatibility.
> 
> IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker <
>> https://github.com/apache/ignite/pull/6552>
> 
> 
> Could you confirm that the JIRA ticket is correct? Whenever I try to open
> IGNITE-11715 in JIRA it redirects me to
> https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-13264
> 
> 
> -
> Denis
> 
> 
> On Wed, Jul 15, 2020 at 6:42 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> Hi,
>> 
>> I have a few small quality-of-life pull requests languishing in the
>> backlog. Can anyone take a look or suggest what else I need to do to
>> progress them?
>> 
>> IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D <
>> https://github.com/apache/ignite/pull/6877>
>> IGNITE-12182 ExecutorService defaults to only server nodes <
>> https://github.com/apache/ignite/pull/6876>
>> IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker <
>> https://github.com/apache/ignite/pull/6552>
>> 
>> Thanks,
>> Stephen
>> 
>> 

A few small pull requests

[Stephen Darlington]

Hi,

I have a few small quality-of-life pull requests languishing in the backlog. 
Can anyone take a look or suggest what else I need to do to progress them?

IGNITE-12192 Allow ignitevisorcmd to quit when pressing ^D 

IGNITE-12182 ExecutorService defaults to only server nodes 

IGNITE-11715 Allow extra options to be passed in to ignite.sh from Docker 


Thanks,
Stephen

[jira] [Created] (IGNITE-13234) SqlFieldsQuery as ContinuousQuery.InitialQuery

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13234:
---

 Summary: SqlFieldsQuery as ContinuousQuery.InitialQuery
 Key: IGNITE-13234
 URL: https://issues.apache.org/jira/browse/IGNITE-13234
 Project: Ignite
  Issue Type: New Feature
  Components: platforms
Affects Versions: 2.8.1, 2.8
Reporter: Stephen Darlington


SqlQuery has been deprecated in favour of SqlFieldsQuery, but ContinuousQuery 
in Ignite does not allow SqlFieldsQuery as InitialQuery.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Ignite.C++ and CMake

[Stephen Darlington]

Not sure if it’ll help, but I made some changes to get it working on a Mac with 
the current built system. There may be some code worth taking.

https://github.com/apache/ignite/pull/4872 


Regards,
Stephen

> On 26 May 2020, at 16:02, Ivan Daschinsky  wrote:
> 
> I appreciate any help, thank you, Ilya.
> 
> Currently I have a small PR without ticket (link in first post),but I
> decided not to file a jira issue before discussion.
> Now I see, that this feature are of great interest to community. So I file
> a ticket, test myself on my home laptop (ubuntu 20.04)
> and add detailed instructions to DEVNOTES.txt in a few days.
> 
> I would be happy if my someone can follow the instruction and write
> possible issues.
> 
> I will notify about status update in this thread in next few days.
> 
> Thank you all very much for support!
> 
> 
> вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev :
> 
>> Hello!
>> 
>> I will assist with checking on Linux if you would contribute a patch.
>> Please start with a ticket (or even an IEP maybe?)
>> 
>> Regards,
>> --
>> Ilya Kasnacheev
>> 
>> 
>> вт, 26 мая 2020 г. в 16:47, Ivan Daschinsky :
>> 
>>> Guys, I will certainly thoroughly test my fix not only unices, but on
>>> windows too.
>>> And I will describe it very thoroughly.
>>> 
>>> When I was C++ developer (more than 10 years ago), I have not any trouble
>>> at all with CMake and Visual Studio 2005.
>>> Everything works and works good. Moreover, you can build with NMake,
>>> msbuild and generate solutions for development.
>>> 
>>> I suppose, for CI purposes, using NMake is a way better, than use vs
>>> solutions.
>>> 
>>> вт, 26 мая 2020 г. в 16:42, Nikolay Izhikov :
>>> 
 Hello, Igor.
 
> Nikolay, removing support for a certain build system is a breaking
 change.
 
 No, it’s not.
 Why do you think so?
 
 Development environment and build system is a subject to change in any
 project.
 We can drop or add support of any build system any time we want.
 
> 26 мая 2020 г., в 16:35, Ilya Kasnacheev 
 написал(а):
> 
> Hello!
> 
> I don't see why we can't get rid of autotools in a minor release,
 provided
> that cmake actually works. Removing native VS support may be a
>>> different
> thing.
> 
> Build system and precise set of dependencies is not a part of public
>>> API
 in
> my opinion.
> 
> Regards,
> --
> Ilya Kasnacheev
> 
> 
> вт, 26 мая 2020 г. в 16:02, Igor Sapego :
> 
>> Great!
>> 
>> Let's start with creating a TC suite for it.
>> 
>> The only concern I have is that it is one more build system
>> to support. Should we get rid of autotools in 3.0?
>> 
>> Best Regards,
>> Igor
>> 
>> 
>> On Tue, May 26, 2020 at 2:44 PM Alexey Kukushkin <
>> kukushkinale...@gmail.com>
>> wrote:
>> 
>>> +1. I recently completed a cross-IDE (MS Visual Studio & GCC/GDB)
 Ignite
>>> C++ project and CMake in Ignite C++ would save me a day of effort.
>>> 
>>> вт, 26 мая 2020 г. в 12:09, Pavel Tupitsyn :
>>> 
 +1
 
 On Tue, May 26, 2020 at 12:02 PM Zhenya Stanilovsky
  wrote:
 
> 
> Ivan huge +1 with your proposal.
> I had some problems with odbc tests building too, looks like
>> cmake
>> will
> make it more easy.
>> Hello Igniters.
>> 
>> I’d like to discuss porting build process of Ignite.C++. I think
>> that
> there is time to change it.
>> 
>> *Motivation*
>> Currently, it is hard to build Ignite.C++. Different build
>> process
>> for
> windows and linux, lack of building support on Mac OS X (quite
>> popular
>>> OS
> among developers), absolutely not IDE support, except windows and
>> only
> Visual Studio is supported.
>> 
>> *Suggestion*
>> I’d suggest to migrate to CMake build system. It is very popular
>> among
> open source projects, and in Apache Software Foundation too.
>>> Notable
 user:
> Apache Mesos, Apache Zookeeper (C client offers CMake as an
>> alternative
 to
> autoconf and only option on windows), Apache Kafka (librdkafka -
>> C/C++
> client), Apache Thrift. Popular column-oriented database
>> ClickHouse
>>> also
> uses CMake.
>> 
>> CMake is widely supported in many IDE’s on various platforms,
>> notably
> Visual Studio, CLion, Xcode, QtCreator, KDevelop.
>> 
>> *Current status*
>> 
>> Currently, the most of work is done (see [1]) and tested on Mac
>>> OS X
> 10.15 (some C++ porting). All tests are run without any flaws,
>>> including
> odbc (unixodbc), ssl, thin and thick client, installation, IDE
 integration

Re: Using GraalVM instead of standard JVM

[Stephen Darlington]

I just switched out OpenJDK and used GraalVM instead. Everything seemed to work 
but I wasn’t looking terribly hard. We’d need to do some more QA but I think 
chances are good that it’ll work just fine.

Regards,
Stephen

>> On 6 May 2020, at 20:31, Denis Magda  wrote:
> Stephen, that's terrific! To Ivan's first question, did you just swap
> HotSpot with GraalVM and got the thing working? Or did it require some
> extra work?
> 
> -
> Denis
> 
> 
>> On Wed, May 6, 2020 at 10:10 AM Stephen Darlington <
>> stephen.darling...@gridgain.com> wrote:
>> 
>> I’ve been playing around with it. I’ve was really impressed that I could
>> run JavaScript on Ignite with comparatively little code:
>> 
>> https://github.com/sdarlington/ignite-graalvm <
>> https://github.com/sdarlington/ignite-graalvm>
>> 
>> I’ve not been looking at performance, though.
>> 
>> Regards,
>> Stephen
>> 
>>>>> On 6 May 2020, at 17:52, Denis Magda  wrote:
>>> I'll leave this reference here so that we have a better understanding of
>>> why it's worthwhile to support GraalVM:
>>> https://blogs.oracle.com/graalvm/apache-spark
>>> —lightning-fast-on-graalvm-enterprise
>>> Spark benefits from running on GraalVM, so should we. Apart from memory
>>> usage and performance advantages, this JVM can execute Python code. With
>>> that, we can enable compute APIs support for Python.
>>> -
>>> Denis
>>> On Sun, May 13, 2018 at 12:23 PM Sven Beauprez <
>> sven.beaup...@theglue.com>
>>> wrote:
>>>> Thnx all for the feedback.
>>>> Looking forward to the results of such a test run.
>>>> Regards,
>>>> Sven
>>>> SVEN BEAUPREZ
>>>> L e a d   A r c h i t e c t
>>>> De Kleetlaan 5, B-1831 Diegem
>>>> www.theglue.com <http://www.theglue.com/>
>>>> On 10/05/2018, 17:44, "Petr Ivanov"  wrote:
>>>>   File the ticket and specify priority — and I will start researching.
>>>>   For test runs — we can have a copy of current test project and run
>>>> some tests in different VMs (as you rightly remarked — right after JDK9
>>>> task is complete).
>>>>> On 10 May 2018, at 18:34, Dmitry Pavlov 
>>>> wrote:
>>>>> Hi Peter,
>>>>> It seems it is one more argument to implement selectable VM for
>>>> existing run-all chain instead of creating one more.
>>>>> Would it be easy to add one more option once JDK 9 run is ready?
>>>>> Sincerely,
>>>>> Dmitriy Pavlov
>>>>> чт, 10 мая 2018 г. в 15:58, Dmitriy Setrakyan >>> <mailto:dsetrak...@apache.org>>:
>>>>> Would be nice to have a TC run on Graal, just to have an
>>>> understanding
>>>>> whether we support it or not.
>>>>> D.
>>>>> On Wed, May 9, 2018 at 4:28 PM, Denis Magda >>> <mailto:dma...@apache.org>> wrote:
>>>>>> The performance might become better just by replacing HotSpot with
>>>> Graal,
>>>>>> but something suggests me that Ignite has to be adopted for this
>>>> JVM (as
>>>>>> well as for Azul VM) to get more benefits. Probably, someone will
>>>> get
>>>>>> interested and pick this task up.
>>>>>> What stands out is that the Graal folks also see this VM as an
>>>> opportunity
>>>>>> to run custom code on a database side like Oracle or MySQL:
>>>>>> https://oracle.github.io/oracle-db-mle/ <
>>>> https://oracle.github.io/oracle-db-mle/> It's a sort of their response
>> to
>>>>>> compute grid functionality of data grids and Hadoop ecosystem.
>>>>>> --
>>>>>> Denis
>>>>>> On Wed, May 9, 2018 at 5:23 AM, sbeaupre <
>>>> sven.beaup...@theglue.com <mailto:sven.beaup...@theglue.com>>
>>>>>> wrote:
>>>>>>> This is just a thought that came out of a discussion with
>>>> Dimitry this
>>>>>>> morning. Recently Oracle has released GraalVM 1.0 after many
>>>> years of
>>>>>>> research and development, as a replacement for standard JVM.
>>>>>>> It should come with huge improvements on several areas
>>>> (interesting for
>>>>>>> ignite: AOT, native compilation, remove object allocation in
>>>> many cases,
>>>>>>> ...)
>>>>>>> Any interest from GG in this? Do you guys think it would give
>>>> ignite a
>>>>>>> performance boost (haven't tested it myself, just checking if it
>>>> is
>>>>>>> worthwhile in the first place, probably low on our prio list).
>>>>>>> More info:
>>>>>>> - GraalVM for Java:
>>>>>>>   http://www.graalvm.org/docs/why-graal/#for-java-programs
>>>> <http://www.graalvm.org/docs/why-graal/#for-java-programs>
>>>>>>> - Twitter is running GraalVM in production for a while now:
>>>>>>>   https://www.youtube.com/watch?v=pR5NDkIZBOA <
>>>> https://www.youtube.com/watch?v=pR5NDkIZBOA>
>>>>>>> - Getting started:
>>>>>>>   http://www.graalvm.org/docs/getting-started/ <
>>>> http://www.graalvm.org/docs/getting-started/>
>>>>>>> regards,
>>>>>>> Sven
>>>>>>> --
>>>>>>> Sent from:
>>>> http://apache-ignite-developers.2346864.n4.nabble.com/ <
>>>> http://apache-ignite-developers.2346864.n4.nabble.com/>

Re: Using GraalVM instead of standard JVM

[Stephen Darlington]

I’ve been playing around with it. I’ve was really impressed that I could run 
JavaScript on Ignite with comparatively little code:

https://github.com/sdarlington/ignite-graalvm 


I’ve not been looking at performance, though.

Regards,
Stephen

> On 6 May 2020, at 17:52, Denis Magda  wrote:
> 
> I'll leave this reference here so that we have a better understanding of
> why it's worthwhile to support GraalVM:
> https://blogs.oracle.com/graalvm/apache-spark
> —lightning-fast-on-graalvm-enterprise
> 
> Spark benefits from running on GraalVM, so should we. Apart from memory
> usage and performance advantages, this JVM can execute Python code. With
> that, we can enable compute APIs support for Python.
> 
> -
> Denis
> 
> 
> On Sun, May 13, 2018 at 12:23 PM Sven Beauprez 
> wrote:
> 
>> Thnx all for the feedback.
>> 
>> Looking forward to the results of such a test run.
>> 
>> Regards,
>> 
>> Sven
>> 
>> 
>> 
>> SVEN BEAUPREZ
>> 
>> L e a d   A r c h i t e c t
>> 
>> 
>> 
>> De Kleetlaan 5, B-1831 Diegem
>> 
>> www.theglue.com 
>> On 10/05/2018, 17:44, "Petr Ivanov"  wrote:
>> 
>>File the ticket and specify priority — and I will start researching.
>> 
>>For test runs — we can have a copy of current test project and run
>> some tests in different VMs (as you rightly remarked — right after JDK9
>> task is complete).
>> 
>> 
>> 
>> 
>>> On 10 May 2018, at 18:34, Dmitry Pavlov 
>> wrote:
>>> 
>>> Hi Peter,
>>> 
>>> It seems it is one more argument to implement selectable VM for
>> existing run-all chain instead of creating one more.
>>> 
>>> Would it be easy to add one more option once JDK 9 run is ready?
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> чт, 10 мая 2018 г. в 15:58, Dmitriy Setrakyan > >:
>>> Would be nice to have a TC run on Graal, just to have an
>> understanding
>>> whether we support it or not.
>>> 
>>> D.
>>> 
>>> On Wed, May 9, 2018 at 4:28 PM, Denis Magda > > wrote:
>>> 
 The performance might become better just by replacing HotSpot with
>> Graal,
 but something suggests me that Ignite has to be adopted for this
>> JVM (as
 well as for Azul VM) to get more benefits. Probably, someone will
>> get
 interested and pick this task up.
 
 What stands out is that the Graal folks also see this VM as an
>> opportunity
 to run custom code on a database side like Oracle or MySQL:
 https://oracle.github.io/oracle-db-mle/ <
>> https://oracle.github.io/oracle-db-mle/> It's a sort of their response to
 compute grid functionality of data grids and Hadoop ecosystem.
 
 --
 Denis
 
 On Wed, May 9, 2018 at 5:23 AM, sbeaupre <
>> sven.beaup...@theglue.com >
 wrote:
 
> This is just a thought that came out of a discussion with
>> Dimitry this
> morning. Recently Oracle has released GraalVM 1.0 after many
>> years of
> research and development, as a replacement for standard JVM.
> 
> It should come with huge improvements on several areas
>> (interesting for
> ignite: AOT, native compilation, remove object allocation in
>> many cases,
> ...)
> 
> Any interest from GG in this? Do you guys think it would give
>> ignite a
> performance boost (haven't tested it myself, just checking if it
>> is
> worthwhile in the first place, probably low on our prio list).
> 
> More info:
> - GraalVM for Java:
>http://www.graalvm.org/docs/why-graal/#for-java-programs
>> 
> - Twitter is running GraalVM in production for a while now:
>https://www.youtube.com/watch?v=pR5NDkIZBOA <
>> https://www.youtube.com/watch?v=pR5NDkIZBOA>
> - Getting started:
>http://www.graalvm.org/docs/getting-started/ <
>> http://www.graalvm.org/docs/getting-started/>
> 
> regards,
> 
> Sven
> 
> 
> 
> 
> 
> --
> Sent from:
>> http://apache-ignite-developers.2346864.n4.nabble.com/ <
>> http://apache-ignite-developers.2346864.n4.nabble.com/>
> 
 
>> 
>> 
>> 
>> 

Re: Active nodes aliveness WatchDog

[Stephen Darlington]

Yes. Nodes are always chatting to each another even if there are no requests 
coming In.

Here’s the status message: 
https://github.com/apache/ignite/blob/e9b3c4cebaecbeec9fa51bd6ec32a879fb89948a/modules/core/src/main/java/org/apache/ignite/spi/discovery/tcp/messages/TcpDiscoveryStatusCheckMessage.java

Regards,
Stephen

> On 8 Apr 2020, at 10:04, Anton Vinogradov  wrote:
> 
> It seems you're talking about Failure Detection (Timeouts).
> Will it detect node failure on still cluster?
> 
> On Wed, Apr 8, 2020 at 11:52 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> The configuration parameters that I’m aware of are here:
>> 
>> 
>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.html
>> 
>> Other people would be better placed to discuss the internals.
>> 
>> Regards.
>> Stephen
>> 
>>> On 8 Apr 2020, at 09:32, Anton Vinogradov  wrote:
>>> 
>>> Stephen,
>>> 
>>>> Nodes check on their neighbours and notify the remaining nodes if one
>>> disappears.
>>> Could you explain how this works in detail?
>>> How can I set/change check frequency?
>>> 
>>> On Wed, Apr 8, 2020 at 11:13 AM Stephen Darlington <
>>> stephen.darling...@gridgain.com> wrote:
>>> 
>>>> This is one of the functions of the DiscoverySPI. Nodes check on their
>>>> neighbours and notify the remaining nodes if one disappears. When the
>>>> topology changes, it triggers a rebalance, which relocates primary
>>>> partitions to live nodes. This is entirely transparent to clients.
>>>> 
>>>> It gets more complex… like there’s the partition loss policy and
>>>> rebalancing doesn’t always happen (configurable, persistence, etc)… but
>>>> broadly it does as you expect.
>>>> 
>>>> Regards,
>>>> Stephen
>>>> 
>>>>> On 8 Apr 2020, at 08:40, Anton Vinogradov  wrote:
>>>>> 
>>>>> Igniters,
>>>>> Do we have some feature allows to check nodes aliveness on a regular
>>>> basis?
>>>>> 
>>>>> Scenario:
>>>>> Precondition
>>>>> The cluster has no load but some node's JVM crashed.
>>>>> 
>>>>> Expected actual
>>>>> The user performs an operation (eg. cache put) related to this node
>> (via
>>>>> another node) and waits for some timeout to gain it's dead.
>>>>> The cluster starts the switch to relocate primary partitions to alive
>>>>> nodes.
>>>>> Now user able to retry the operation.
>>>>> 
>>>>> Desired
>>>>> Some WatchDog checks nodes aliveness on a regular basis.
>>>>> Once a failure detected, the cluster starts the switch.
>>>>> Later, the user performs an operation on an already fixed cluster and
>>>>> waits for nothing.
>>>>> 
>>>>> It would be good news if the "Desired" case is already Actual.
>>>>> Can somebody point to the feature that performs this check?
>>>> 
>>>> 
>>>> 
>> 
>> 
>> 

Re: Active nodes aliveness WatchDog

[Stephen Darlington]

The configuration parameters that I’m aware of are here:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.html

Other people would be better placed to discuss the internals.

Regards.
Stephen

> On 8 Apr 2020, at 09:32, Anton Vinogradov  wrote:
> 
> Stephen,
> 
>> Nodes check on their neighbours and notify the remaining nodes if one
> disappears.
> Could you explain how this works in detail?
> How can I set/change check frequency?
> 
> On Wed, Apr 8, 2020 at 11:13 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> This is one of the functions of the DiscoverySPI. Nodes check on their
>> neighbours and notify the remaining nodes if one disappears. When the
>> topology changes, it triggers a rebalance, which relocates primary
>> partitions to live nodes. This is entirely transparent to clients.
>> 
>> It gets more complex… like there’s the partition loss policy and
>> rebalancing doesn’t always happen (configurable, persistence, etc)… but
>> broadly it does as you expect.
>> 
>> Regards,
>> Stephen
>> 
>>> On 8 Apr 2020, at 08:40, Anton Vinogradov  wrote:
>>> 
>>> Igniters,
>>> Do we have some feature allows to check nodes aliveness on a regular
>> basis?
>>> 
>>> Scenario:
>>> Precondition
>>> The cluster has no load but some node's JVM crashed.
>>> 
>>> Expected actual
>>> The user performs an operation (eg. cache put) related to this node (via
>>> another node) and waits for some timeout to gain it's dead.
>>> The cluster starts the switch to relocate primary partitions to alive
>>> nodes.
>>> Now user able to retry the operation.
>>> 
>>> Desired
>>> Some WatchDog checks nodes aliveness on a regular basis.
>>> Once a failure detected, the cluster starts the switch.
>>> Later, the user performs an operation on an already fixed cluster and
>>> waits for nothing.
>>> 
>>> It would be good news if the "Desired" case is already Actual.
>>> Can somebody point to the feature that performs this check?
>> 
>> 
>> 

Re: Active nodes aliveness WatchDog

[Stephen Darlington]

This is one of the functions of the DiscoverySPI. Nodes check on their 
neighbours and notify the remaining nodes if one disappears. When the topology 
changes, it triggers a rebalance, which relocates primary partitions to live 
nodes. This is entirely transparent to clients.

It gets more complex… like there’s the partition loss policy and rebalancing 
doesn’t always happen (configurable, persistence, etc)… but broadly it does as 
you expect.

Regards,
Stephen

> On 8 Apr 2020, at 08:40, Anton Vinogradov  wrote:
> 
> Igniters,
> Do we have some feature allows to check nodes aliveness on a regular basis?
> 
> Scenario:
> Precondition
>  The cluster has no load but some node's JVM crashed.
> 
> Expected actual
>  The user performs an operation (eg. cache put) related to this node (via
> another node) and waits for some timeout to gain it's dead.
>  The cluster starts the switch to relocate primary partitions to alive
> nodes.
>  Now user able to retry the operation.
> 
> Desired
>  Some WatchDog checks nodes aliveness on a regular basis.
>  Once a failure detected, the cluster starts the switch.
>  Later, the user performs an operation on an already fixed cluster and
> waits for nothing.
> 
> It would be good news if the "Desired" case is already Actual.
> Can somebody point to the feature that performs this check?

[jira] [Created] (IGNITE-12867) Python thin client: support for DB-API

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-12867:
---

 Summary: Python thin client: support for DB-API
 Key: IGNITE-12867
 URL: https://issues.apache.org/jira/browse/IGNITE-12867
 Project: Ignite
  Issue Type: Improvement
  Components: python
Affects Versions: 2.8
Reporter: Stephen Darlington


Python has a database connection standard called [DB-API|http://example.com/]. 
It would make the Ignite Python thin client a lot more useful if it could use 
this in addition to its current Ignite-specific APIs. Implementing this would 
enable other Apache tools such as SuperSet and Airflow to connect to and use 
Ignite.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Slim binary release and docker image for Apache Ignite

[Stephen Darlington]

Why ignite-jta and ignite-urideploy? Anecdotally at least, I know very few 
people who use either. 

> On 6 Mar 2020, at 11:09, Ilya Kasnacheev  wrote:
> 
> Hello!
> 
> Re-posting from *[DISCUSSION] Release Apache Ignite 2.8.0 RC1*
> 
> I have prepared assemblies for Apache Ignite slim packaging:
> https://github.com/apache/ignite/tree/ignite-slim
> 
> It is based on ignite-2.8
> 
> You can build it with mvn initialize -Prelease,lgpl -Dignite.edition=apache-
> ignite-slim after a normal release build.
> 
> Please consider the contents of resulting
> target/bin/apache-ignite-slim-2.8.0-bin.zip
> It will be a 65M download as opposed to main 455M apache-ignite-2.8.0
> distribution.
> 
> My suggestion is that we can publish it as a post-release step since it
> does not affect the release in any way. If we do, we should probably
> indicate size for every kind of artifact in our download section, so our
> users can choose based on that information.
> 
> The following modules are included:
> 
> libs:
> core/shmem/jcache
> ignite-indexing
> ignite-spring
> 
> libs/optional:
> ignite-compress  ignite-kubernetes  ignite-log4j2  ignite-rest-http
> ignite-spring-data_2.2
> ignite-jta   ignite-log4j   ignite-opencensus  ignite-slf4j
> ignite-urideploy
> 
> I have kept examples, but removed benchmarks. sqlline still present, of
> course.
> 
> ignite-zookeeper has a lot of dependencies (8M) which we do not update
> often enough (such as guava, curator, jackson), and which may form an
> attack surface.
> 
> Not a pressing problem for 'integrated' ignite-zookeeper users, since they
> can re-import these dependencies with more recent versions using maven or
> gradle.
> But for our users who rely on binary package for all JARs, outdated
> dependencies may pose a problem.
> 
> Therefore my opinion is to exclude this dependency and not put our faith on
> zookeeper dependency version.
> 
> The same can be put for ignite-compress, and indeed, I'm not sure if we
> should keep it.
> 
> We can have an ad-hoc vote here.
> 
> I would like to hear arguments for both inclusion and exclusion of
> ignite-zookeeper and ignite-compress into slim package (in any combination).
> 
> I would also like to know if you want a formal vote on the issue.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пн, 27 янв. 2020 г. в 21:13, Denis Magda :
> 
>> Alex, could you please list all the modules that will be excluded? It will
>> help to confirm we haven't dumped anything essential.
>> 
>> -
>> Denis
>> 
>> 
>> On Mon, Jan 27, 2020 at 12:33 AM Alexey Goncharuk <
>> alexey.goncha...@gmail.com> wrote:
>> 
>>> Got it, sounds good!
>>> Should we consider the list of modules included in the slim package
>>> finalized?
>>> 
>>> чт, 16 янв. 2020 г. в 13:13, Igor Sapego :
>>> 
 Alexey, if I understand correctly, Ilya does not suggest to pre-built
 binaries, just to ship it with configure script pre-generated, which
 is a common practice for autotools packages. Building will be still
 required for the user, but there will be less requirements and
 possible errors during build.
 
 I like the idea. Let's do this.
 
 Best Regards,
 Igor
 
 
 On Thu, Jan 16, 2020 at 11:57 AM Alexey Goncharuk <
 alexey.goncha...@gmail.com> wrote:
 
> To me it doesn't really matter if it will be 'slim' or 'lite' :) I
>>> would
> not name it 'core' because indeed it would be confusing with the core
> module name.
> 
> Agree that platforms support is useful, so I would keep them as Ilya
> suggested. As for the C++ packages pre-build - let's hear out Igor's
> opinion on this. Pre-built binaries certainly add usability, but I am
>>> not
> sure how those binaries should be tested afterwards.
> 
> ср, 15 янв. 2020 г. в 18:33, Alexey Kuznetsov >> :
> 
>> I'm +1 for "SLIM" it is a common name in Docker world.
>> 
>> On Wed, Jan 15, 2020 at 9:48 PM Petr Ivanov 
 wrote:
>> 
>>> +1 for slim binary
>>> Plus docker-slim
>>> Plus RPM / DEB packages modularisation like PHP distribution —
>> with
> core
>>> and lots of integrations / modules.
>>> 
 On 15 Jan 2020, at 17:40, Ilya Kasnacheev <
 ilya.kasnach...@gmail.com
>> 
>>> wrote:
 
 Hello!
 
 I think we should name it "core" since we already have
>>> ignite-core
> and
>> it
 will be confusing. Maybe we should go full 00s and call it
>>> "lite"?
 
 I also think we should keep both .Net and C++. .Net is runnable
>>> out
> of
>>> box
 which is awesome, and C++ needs building but it is rather small
>>> in
>> source
 form.
 
 I also suggest a different change to build process. Let's ship
>>> C++
> with
 automake, etc, already run, for all binary packaging options?
 WDYT? I
>> can
 assist in build process tuning.
 
 Regards,

Re: Apache Ignite downloads are redirecting from https to http

[Stephen Darlington]

Thanks, Ilya!

> On 19 Feb 2020, at 15:26, Ilya Kasnacheev  wrote:
> 
> Hello!
> 
> I have fixed these links, now they point to HTTPS. Please check that it
> would work now.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> ср, 19 февр. 2020 г. в 16:49, Denis Magda :
> 
>> Peter,
>> 
>> Would you mind checking this issue and suggest a proper solution?
>> 
>> Dmitry,
>> 
>> Might this be somehow related to the RPM changes ASF INFRA suggested to
>> make?
>> 
>> -
>> Denis
>> 
>> 
>>> On Wed, Feb 19, 2020 at 4:33 AM Stephen Darlington <
>>> stephen.darling...@gridgain.com> wrote:
>>> 
>>> As seen in the user mailing list. Looks like there’s a redirect from a
>>> https on apache.org to a non-https link. Is there anyone who can fi
>> this?
>>> 
>>> Regards,
>>> Stephen
>>> 
>>>> Begin forwarded message:
>>>> 
>>>> From: Devin Anderson 
>>>> Subject: Apache Ignite downloads are redirecting from https to http
>>>> Date: 18 February 2020 at 01:42:14 GMT
>>>> To: 
>>>> Reply-To: u...@ignite.apache.org
>>>> 
>>>> Hi all,
>>>> 
>>>> I'm not sure if this is the correct mailing list to bring up this
>>> issue.  If I'm writing the wrong mailing list, please let me know and
>> I'll
>>> gladly send my message to a more appropriate list/person.
>>>> 
>>>> When I attempt to run `apt-get update` on Ubuntu 18.04 with the Apache
>>> Ignite repository in /etc/apt/sources.list, the update fails due to an
>>> https to http redirect.  Here's the output from stdout:
>>>> 
>>>> --
>>>> 
>>>> Hit:1 http://us.archive.ubuntu.com/ubuntu <
>>> http://us.archive.ubuntu.com/ubuntu> bionic InRelease
>>>> Hit:2 http://security.ubuntu.com/ubuntu <
>>> http://security.ubuntu.com/ubuntu> bionic-security InRelease
>>>> Hit:3 http://us.archive.ubuntu.com/ubuntu <
>>> http://us.archive.ubuntu.com/ubuntu> bionic-updates InRelease
>>>> Ign:4 https://downloads.apache.org/ignite/deb <
>>> https://downloads.apache.org/ignite/deb> apache-ignite InRelease
>>>> Err:5 https://downloads.apache.org/ignite/deb <
>>> https://downloads.apache.org/ignite/deb> apache-ignite Release
>>>>  Redirection from https to '
>>> http://dl.bintray.com/apache/ignite-deb/dists/apache-ignite/Release <
>>> http://dl.bintray.com/apache/ignite-deb/dists/apache-ignite/Release>' is
>>> forbidden [IP: 88.99.95.219 443]
>>>> Reading package lists...
>>>> 
>>>> --
>>>> 
>>>> ... and stderr:
>>>> 
>>>> --
>>>> 
>>>> E: The repository 'http://apache.org/dist/ignite/deb <
>>> http://apache.org/dist/ignite/deb> apache-ignite Release' does not have
>> a
>>> Release file.
>>>> 
>>>> --
>>>> 
>>>> You can see the redirect for yourself by visiting:
>>>> 
>>>>https://downloads.apache.org/ignite/deb <
>>> https://downloads.apache.org/ignite/deb>
>>>> 
>>>> ... and noting that your browser is redirected to:
>>>> 
>>>>http://dl.bintray.com/apache/ignite-deb/ <
>>> http://dl.bintray.com/apache/ignite-deb/>
>>>> 
>>>> I found one or two other places on downloads.apache.org that
>> redirected
>>> to dl.bintray.com, and those instances appear to redirect to an https
>>> URI, so this looks like a problem specific to the way redirects are
>>> configured for Apache Ignite assets.
>>>> 
>>>> Any ideas?
>>>> 
>>>> Thanks,
>>>> --
>>>> Devin
>>>> 
>>> 
>>> 
>>> 
>> 

Fwd: Apache Ignite downloads are redirecting from https to http

[Stephen Darlington]

As seen in the user mailing list. Looks like there’s a redirect from a https on 
apache.org to a non-https link. Is there anyone who can fi this?

Regards,
Stephen

> Begin forwarded message:
> 
> From: Devin Anderson 
> Subject: Apache Ignite downloads are redirecting from https to http
> Date: 18 February 2020 at 01:42:14 GMT
> To: 
> Reply-To: u...@ignite.apache.org
> 
> Hi all,
> 
> I'm not sure if this is the correct mailing list to bring up this issue.  If 
> I'm writing the wrong mailing list, please let me know and I'll gladly send 
> my message to a more appropriate list/person.
> 
> When I attempt to run `apt-get update` on Ubuntu 18.04 with the Apache Ignite 
> repository in /etc/apt/sources.list, the update fails due to an https to http 
> redirect.  Here's the output from stdout:
> 
> --
> 
> Hit:1 http://us.archive.ubuntu.com/ubuntu 
>  bionic InRelease
> Hit:2 http://security.ubuntu.com/ubuntu  
> bionic-security InRelease
> Hit:3 http://us.archive.ubuntu.com/ubuntu 
>  bionic-updates InRelease
> Ign:4 https://downloads.apache.org/ignite/deb 
>  apache-ignite InRelease
> Err:5 https://downloads.apache.org/ignite/deb 
>  apache-ignite Release
>   Redirection from https to 
> 'http://dl.bintray.com/apache/ignite-deb/dists/apache-ignite/Release 
> ' is 
> forbidden [IP: 88.99.95.219 443]
> Reading package lists...
> 
> --
> 
> ... and stderr:
> 
> --
> 
> E: The repository 'http://apache.org/dist/ignite/deb 
>  apache-ignite Release' does not have a 
> Release file.
> 
> --
> 
> You can see the redirect for yourself by visiting:
> 
> https://downloads.apache.org/ignite/deb 
> 
> 
> ... and noting that your browser is redirected to:
> 
> http://dl.bintray.com/apache/ignite-deb/ 
> 
> 
> I found one or two other places on downloads.apache.org that redirected to 
> dl.bintray.com, and those instances appear to redirect to an https URI, so 
> this looks like a problem specific to the way redirects are configured for 
> Apache Ignite assets.
> 
> Any ideas?
> 
> Thanks,
> -- 
> Devin
> 

Re: Mark MVCC with @IgniteExperimental

[Stephen Darlington]

Yes! I’ve already seen people try to use this without awareness that it’s not 
production ready.

What happens with the annotation, incidentally? Is it just in the documentation 
or do you get a compile-time warning?

> On 6 Feb 2020, at 11:32, Nikolay Izhikov  wrote:
> 
> Hello, Igniters.
> 
> Should we mark MVCC feature with the new @IgniteExperimental?
> 
> We explicitly note users that MVCC has beta status, for now [1]
> 
>> Beta version of Transactional SQL and MVCC
>> In Ignite v2.7, Transactional SQL and MVCC are released as beta versions to 
>> allow users to experiment and share feedback. 
>> This version of Transactional SQL and MVCC should not be considered for 
>> production.
> 
> 
> [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control
> 
> 

Re: Calcite based SQL query engine. Local queries

[Stephen Darlington]

I made a (bad) assumption that this would also affect queries against 
partitions. If “setLocal()” goes away but “setPartitions()” remains I’m happy.

What I would say is that the “broadcast / local” method is one I see fairly 
often. Do we need to do a better job educating people of the “correct” way?

Regards,
Stephen

> On 7 Nov 2019, at 08:30, Alexey Goncharuk  wrote:
> 
> Denis, Stephen,
> 
> Running a local query in a broadcast closure won't work on changing
> topology. We specifically added an affinityCall method to the compute API
> in order to pin a partition to prevent its moving and eviction throughout
> the task execution. Therefore, the query inside an affinityCall is always
> executed against some partitions (otherwise the query may give incorrect
> results when topology is changed).
> 
> I support Igor's question and think that the 'local' flag for the query
> should be deprecated and eventually removed. A 'local' query can always be
> expressed as a query agains a set of partitions. If those partitions are
> located on the same node - good, we get fast and correct results. If not -
> we may either raise an exception and ask user to remap the query, or
> fallback to a distributed query execution.
> 
> Given that the Calcite prototype is in its early stages, it's likely its
> first version will be available in 3.x, and it's a good chance to get rid
> of wrong API pieces.
> 
> --AG
> 
> пн, 4 нояб. 2019 г. в 14:02, Stephen Darlington <
> stephen.darling...@gridgain.com>:
> 
>> A common use case is where you want to work on many rows of data across
>> the grid. You’d broadcast a closure, running the same code on every node
>> with just the local data. SQL doesn’t work in isolation — it’s often used
>> as a filter for future computations.
>> 
>> Regards,
>> Stephen
>> 
>>> On 1 Nov 2019, at 17:53, Ivan Pavlukhin  wrote:
>>> 
>>> Denis,
>>> 
>>> I am mostly concerned about gathering use cases. It would be great to
>>> critically assess such cases to identify why it cannot be solved by
>>> using distributed SQL. Also it sounds similar to some kind of "hints",
>>> but very limited and with all hints drawbacks (impossibility to use
>>> full strength of CBO). We can provide better "hints" support with new
>>> engine as well.
>>> 
>>> пт, 1 нояб. 2019 г. в 20:14, Denis Magda :
>>>> 
>>>> Ivan,
>>>> 
>>>> I was involved in a couple of such use cases personally, so, that's not
>> my
>>>> imagination ;) Even more, as far as I remember, the primary reason why
>> we
>>>> improved our affinityRuns ensuring no partition is purged from a node
>> until
>>>> a task is completed is because many users were running local SQL from
>>>> compute tasks and needed a guarantee that SQL will always return a
>> correct
>>>> result set.
>>>> 
>>>> -
>>>> Denis
>>>> 
>>>> 
>>>> On Fri, Nov 1, 2019 at 10:01 AM Ivan Pavlukhin 
>> wrote:
>>>> 
>>>>> Denis,
>>>>> 
>>>>> Would be nice to see real use-cases of affinity call + local SQL
>>>>> combination. Generally, new engine will be able to infer collocation
>>>>> resulting in the same collocated execution automatically.
>>>>> 
>>>>> пт, 1 нояб. 2019 г. в 19:11, Denis Magda :
>>>>>> 
>>>>>> Hi Igor,
>>>>>> 
>>>>>> Local queries feature is broadly used together with affinity-based
>>>>> compute
>>>>>> tasks:
>>>>>> 
>>>>> 
>> https://apacheignite.readme.io/docs/collocate-compute-and-data#section-affinity-call-and-run-methods
>>>>>> 
>>>>>> The use case is as follows. The user knows that all required data
>> needed
>>>>>> for computation is collocated, and SQL is used as an advanced API for
>>>>> data
>>>>>> retrieval from the computation code. The affinity task ensures that
>>>>>> partitions won't be discarded from the node(s) if the topology changes
>>>>>> during the task execution and, thus, it's safe to run SQL locally
>>>>> skipping
>>>>>> distributed phases.
>>>>>> 
>>>>>> The combination of affinity compute tasks with local SQL is a real and
>>>>>> valuable use case, and this is what we need to support with Calcite.
>> Do
>>>&

Re: Lets add measure units for metrics

[Stephen Darlington]

Another thing that I would like to see in the metrics is the scope: is this 
metric cluster-wide or only for this node?

Like the units of measure, this could be an extra field. Or it could be naming 
convention? (Currently many node-specific metrics have “local” in the name, but 
this is not entirely consistent.) Either way, I agree that hiding this 
information in a plain-text field is not a good idea; it should be possible to 
determine programatically.

Regards,
Stephen

> On 6 Nov 2019, at 12:48, Alexey Kuznetsov  wrote:
> 
> Andrey,
> 
> The main reason of separate field - is a simplicity of usage in tools.
> Yes, we can have measurement units in description, but external tool will
> be forced to parse textual representation of units.
> Description is a text that has no special format and how we can ask
> developers to not forget to specify units in description?
> But with enum we can make it mandatory field. Or write a test that will
> check all metrics for "not null" for this field.
> 
> What do you think?
> 
> 
> On Wed, Nov 6, 2019 at 6:28 PM Andrey Gura  wrote:
> 
>> Units just can be added to a metric description. What is the purpose
>> of dedicated field for it?
>> 
>> We doesn't provide any API for work with measurement units and should
>> not do it. Metrics interpretation is responsibility of external
>> systems for metrics gathering. Enum with metrics measurement doesn't
>> bring any value for external systems and only useful thing is self
>> documentation.
>> 
>> On Wed, Nov 6, 2019 at 12:42 PM Nikolay Izhikov 
>> wrote:
>>> 
>>> Hello, Alexey.
>>> 
>>> I'm +1.
>>> Let's do it.
>>> 
>>> В Ср, 06/11/2019 в 16:37 +0700, Alexey Kuznetsov пишет:
 Hi, All!
 
 I'm working on a tool for metrics monitoring.
 And found that Metric interface [1] has a name and description, but
>> has not
 measurement units.
 
 When you are working on a tool that collects all metrics and shows it
>> in
 human readable format it is a very important piece of information.
 
 Because we have a lot of different metrics, some of them in bytes,
>> some of
 them in percents, some of them are simple counters and so on and so
>> forth.
 
 How about to add Enum with all possible measurement units to  Metric
 interface [1] ?
 
 Thoughts?
 
 [1]
 
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/spi/metric/Metric.java
 
>> 
> 
> 
> -- 
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com

Re: Calcite based SQL query engine. Local queries

[Stephen Darlington]

A common use case is where you want to work on many rows of data across the 
grid. You’d broadcast a closure, running the same code on every node with just 
the local data. SQL doesn’t work in isolation — it’s often used as a filter for 
future computations.

Regards,
Stephen

> On 1 Nov 2019, at 17:53, Ivan Pavlukhin  wrote:
> 
> Denis,
> 
> I am mostly concerned about gathering use cases. It would be great to
> critically assess such cases to identify why it cannot be solved by
> using distributed SQL. Also it sounds similar to some kind of "hints",
> but very limited and with all hints drawbacks (impossibility to use
> full strength of CBO). We can provide better "hints" support with new
> engine as well.
> 
> пт, 1 нояб. 2019 г. в 20:14, Denis Magda :
>> 
>> Ivan,
>> 
>> I was involved in a couple of such use cases personally, so, that's not my
>> imagination ;) Even more, as far as I remember, the primary reason why we
>> improved our affinityRuns ensuring no partition is purged from a node until
>> a task is completed is because many users were running local SQL from
>> compute tasks and needed a guarantee that SQL will always return a correct
>> result set.
>> 
>> -
>> Denis
>> 
>> 
>> On Fri, Nov 1, 2019 at 10:01 AM Ivan Pavlukhin  wrote:
>> 
>>> Denis,
>>> 
>>> Would be nice to see real use-cases of affinity call + local SQL
>>> combination. Generally, new engine will be able to infer collocation
>>> resulting in the same collocated execution automatically.
>>> 
>>> пт, 1 нояб. 2019 г. в 19:11, Denis Magda :
 
 Hi Igor,
 
 Local queries feature is broadly used together with affinity-based
>>> compute
 tasks:
 
>>> https://apacheignite.readme.io/docs/collocate-compute-and-data#section-affinity-call-and-run-methods
 
 The use case is as follows. The user knows that all required data needed
 for computation is collocated, and SQL is used as an advanced API for
>>> data
 retrieval from the computation code. The affinity task ensures that
 partitions won't be discarded from the node(s) if the topology changes
 during the task execution and, thus, it's safe to run SQL locally
>>> skipping
 distributed phases.
 
 The combination of affinity compute tasks with local SQL is a real and
 valuable use case, and this is what we need to support with Calcite. Do
>>> you
 see any challenges?
 
 -
 Denis
 
 
 On Fri, Nov 1, 2019 at 8:46 AM Roman Kondakov >>> 
 wrote:
 
> Hi Igor!
> 
> IMO we need to maintain the backward compatibility between old and new
> query engines as much as possible. And therefore we shouldn't change
>>> the
> behavior of local queries.
> 
> So, for local queries Calcite's planner shouldn't consider the
> distribution trait at all.
> 
> 
> --
> Kind Regards
> Roman Kondakov
> 
> On 01.11.2019 17:07, Seliverstov Igor wrote:
>> Hi Igniters,
>> 
>> Working on new generation of Ignite SQL I faced a question: «Do we
>>> need
> local queries at all and, if so, what semantic they should have?».
>> 
>> Current planing flow consists of next steps:
>> 
>> 1) Parsing SQL to AST
>> 2) Validating AST (against Schema)
>> 3) Optimizing (Building execution graph)
>> 4) Splitting (into query fragments which executes on target nodes)
>> 5) Mapping (query fragments to nodes/partitions)
>> 
>> At last step we check that all Fragment sources (a table or result)
>>> have
> the same distribution (in other words all sources have to be
>>> co-located)
>> 
>> Planner and Splitter guarantee that all caches in a Fragment are
> co-located, an Exchange is produced otherwise. But if we force local
> execution we cannot produce Exchanges, that means we may face two
> non-co-located caches inside a single query fragment (result of local
>>> query
> planning is a single query fragment). So, we cannot pass the check.
>> 
>> Should we throw an exception or omit the check for local query
>>> planning
> or prohibit local queries at all?
>> 
>> Your thoughts?
>> 
>> Regards,
>> Igor
> 
>>> 
>>> 
>>> 
>>> --
>>> Best regards,
>>> Ivan Pavlukhin
>>> 
> 
> 
> 
> -- 
> Best regards,
> Ivan Pavlukhin

Re: [IGNITE-9836] Invalid check of ea java versions

[Stephen Darlington]

Done: https://github.com/apache/ignite/pull/6920

While we’re talking about the startup scripts… 
https://issues.apache.org/jira/browse/IGNITE-11856

Regards,
Stephen

> On 26 Sep 2019, at 17:02, Ilya Kasnacheev  wrote:
> 
> Hello!
> 
> Please do!
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> вт, 17 сент. 2019 г. в 11:13, Stephen Darlington <
> stephen.darling...@gridgain.com>:
> 
>> I can’t take any credit for the patch but if the original author has lost
>> interest I’m happy to help push it through.
>> 
>> Regards,
>> Stephen
>> 
>>> On 16 Sep 2019, at 19:31, Denis Magda  wrote:
>>> 
>>> Stephen,
>>> 
>>> Thanks for sending the patch! Seems that Igniters are already actively
>>> reviewing it in JIRA.
>>> 
>>> -
>>> Denis
>>> 
>>> 
>>> On Mon, Sep 16, 2019 at 7:03 AM Stephen Darlington <
>>> stephen.darling...@gridgain.com> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> Would someone mind taking a quick look at this ticket? Basically, a
>> clean
>>>> download of Ignite won’t start if the version of Java you’re using has a
>>>> number like “java version "1.8.0_202-ea””. (This is the default if you
>> get
>>>> your JDK using Homebrew on a Mac.)
>>>> 
>>>>> https://issues.apache.org/jira/browse/IGNITE-9836 <
>>>> https://issues.apache.org/jira/browse/IGNITE-9836>
>>>> 
>>>> This has been bugging me for ages and now that I look at it I find that
>>>> there’s already a tiny, working patch available.
>>>> 
>>>> Regards,
>>>> Stephen
>> 
>> 
>> 

[jira] [Created] (IGNITE-12192) visorcmd hangs if you ^D to quit

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-12192:
---

 Summary: visorcmd hangs if you ^D to quit
 Key: IGNITE-12192
 URL: https://issues.apache.org/jira/browse/IGNITE-12192
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.7.5
Reporter: Stephen Darlington
Assignee: Stephen Darlington


You can exit most Unix utilities by pressing ^D. However, if you do that in 
ignitevisorcmd, the process hangs. You have to press ^C to quit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12182) ExecutorService is inconsistent with Compute and Services, runs on clients

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-12182:
---

 Summary: ExecutorService is inconsistent with Compute and 
Services, runs on clients
 Key: IGNITE-12182
 URL: https://issues.apache.org/jira/browse/IGNITE-12182
 Project: Ignite
  Issue Type: Bug
  Components: compute
Affects Versions: 2.7.5
Reporter: Stephen Darlington
Assignee: Stephen Darlington


In IGNITE-860 the default behaviour was changed so that compute and service 
tasks would be executed only on server nodes. This is a sensible default and 
it's confusing that it's not also true for jobs run using the ExecutorService.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [IGNITE-9836] Invalid check of ea java versions

[Stephen Darlington]

I can’t take any credit for the patch but if the original author has lost 
interest I’m happy to help push it through.

Regards,
Stephen

> On 16 Sep 2019, at 19:31, Denis Magda  wrote:
> 
> Stephen,
> 
> Thanks for sending the patch! Seems that Igniters are already actively
> reviewing it in JIRA.
> 
> -
> Denis
> 
> 
> On Mon, Sep 16, 2019 at 7:03 AM Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> 
>> Hi,
>> 
>> Would someone mind taking a quick look at this ticket? Basically, a clean
>> download of Ignite won’t start if the version of Java you’re using has a
>> number like “java version "1.8.0_202-ea””. (This is the default if you get
>> your JDK using Homebrew on a Mac.)
>> 
>>> https://issues.apache.org/jira/browse/IGNITE-9836 <
>> https://issues.apache.org/jira/browse/IGNITE-9836>
>> 
>> This has been bugging me for ages and now that I look at it I find that
>> there’s already a tiny, working patch available.
>> 
>> Regards,
>> Stephen

[IGNITE-9836] Invalid check of ea java versions

[Stephen Darlington]

Hi,

Would someone mind taking a quick look at this ticket? Basically, a clean 
download of Ignite won’t start if the version of Java you’re using has a number 
like “java version "1.8.0_202-ea””. (This is the default if you get your JDK 
using Homebrew on a Mac.)

> https://issues.apache.org/jira/browse/IGNITE-9836 
> 

This has been bugging me for ages and now that I look at it I find that there’s 
already a tiny, working patch available.

Regards,
Stephen

Re: Replacing default work dir from tmp to current dir

[Stephen Darlington]

Yes, when data is a stake, fail early is the absolutely the right thing to do.

Regards,
Stephen

> On 12 Aug 2019, at 16:37, Ivan Rakov  wrote:
> 
> Hi Anton,
> 
> Actually, the issue is even more unpleasant.
> 
> Official Ignite documentation says that it's possible to configure path where 
> your persistence files will be stored: 
> https://apacheignite.readme.io/docs/distributed-persistent-store
> However, even if you have set all path options (storage, WAL, WAL archive), 
> Ignite will still store crucial metadata in resolved work directory 
> (java.io.tmpdir by default). Example is binary metadata files, absence of 
> which can make your data unavailable.
> 
> I propose to fail Ignite node in case neither IGNITE_HOME nor 
> IgniteConfiguration#igniteWorkDir is set. It's better to let user know about 
> missing configuration options during startup than let OS corrupt storage by 
> cleaning temp dirs.
> 
> Thoughts?
> 
> Best Regards,
> Ivan Rakov
> 
> On 12.08.2019 18:10, Anton Kalashnikov wrote:
>> Hello, Igniters.
>> 
>> Currently, in the case, when work directory wasn't set by user ignite can 
>> resolve it to tmp directory which leads to some problem - tmp directory can 
>> be cleared at some unexpected moment by operation system and different types 
>> of critical data would be lost(ex. binary_meta, persistance data).
>> 
>> Looks like it is not expected behaviour and maybe it is better instead of 
>> tmp directory use the current working directory("user.dir")? Or any other 
>> idea?
>> 
>> A little more details you can find in the ticket - 
>> https://issues.apache.org/jira/browse/IGNITE-12057
>> -- 
>> Best regards,
>> Anton Kalashnikov
>> 

Re: StackOverflow question about encryption

[Stephen Darlington]

Assume he means this one: 
https://stackoverflow.com/questions/57124826/apache-ignite-transparent-data-encryption-master-key-digest-differs-node-join

> On 23 Jul 2019, at 11:02, Dmitriy Pavlov  wrote:
> 
> Hi Vladimir, could you please share link to the question?
> 
> вт, 23 июл. 2019 г. в 12:47, Vladimir Pligin :
> 
>> Hi igniters,
>> 
>> I can see a question on SO
>> http://apache-ignite-developers.2346864.n4.nabble.com. The question is
>> related to encryption.
>> Nikolay Izhikov, as far as know you're the author of the feature and
>> unfortunately I don't know anyone else who is familiar with it. Could you
>> please answer? Thanks a lot in advance.
>> 
>> 
>> 
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>> 

[jira] [Created] (IGNITE-11856) Running ignite.sh with nojmx flag stops with error

[Stephen Darlington (JIRA)]

Stephen Darlington created IGNITE-11856:
---

 Summary: Running ignite.sh with nojmx flag stops with error
 Key: IGNITE-11856
 URL: https://issues.apache.org/jira/browse/IGNITE-11856
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.7
Reporter: Stephen Darlington
Assignee: Stephen Darlington


Running ignite.sh with the {{-nojmx}} flag results in an error:
{code:java}
./ignite.sh: line 228: JMX_MON: unbound variable{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11553) Ignite metrics to Prometheus.io

[Stephen Darlington (JIRA)]

Stephen Darlington created IGNITE-11553:
---

 Summary: Ignite metrics to Prometheus.io
 Key: IGNITE-11553
 URL: https://issues.apache.org/jira/browse/IGNITE-11553
 Project: Ignite
  Issue Type: New Feature
Reporter: Stephen Darlington
Assignee: Stephen Darlington


It would be nice if Ignite could send its metrics (or at least some of them) to 
[Prometheus|https://prometheus.io] metrics and alerting system.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Created] (IGNITE-10577) ignite-kubernetes is missing jackson-annotations dependency

[Stephen Darlington]

Yeah, you can fix by manually adding the dependency either in your own code or 
by adding ignite-aws or ignite-rest-http to your list of OPTION_LIBS (since 
those also contain the correct dependency).

Regards,
Stephen

> On 7 Dec 2018, at 14:45, Dmitriy Pavlov  wrote:
> 
> I've checked this patch and it looks good to me, it is really simple.
> 
> But probably we should consider auto-test it somehow. Nikolay Kulagin,
> could you chime in?
> 
> Stephen, could it be a workaround for end-user code? I guess user can just
> add this dependency in the pom and everything will work well.
> 
> Because release 2.7 is already done, the nearest time the issue could be
> fixed is 2.8.
> 
> чт, 6 дек. 2018 г. в 20:43, Stephen Darlington <
> stephen.darling...@gridgain.com>:
> 
>> Not sure what the etiquette on this is, but this is a regression in 2.7. I
>> added a simple patch to the ticket.
>> 
>> Regards,
>> Stephen
>> 
>>> On 6 Dec 2018, at 14:42, Stephen Darlington (JIRA) 
>> wrote:
>>> 
>>> Stephen Darlington created IGNITE-10577:
>>> ---
>>> 
>>>Summary: ignite-kubernetes is missing jackson-annotations
>> dependency
>>>Key: IGNITE-10577
>>>URL: https://issues.apache.org/jira/browse/IGNITE-10577
>>>    Project: Ignite
>>> Issue Type: Bug
>>> Components: build
>>>   Affects Versions: 2.7
>>>   Reporter: Stephen Darlington
>>>   Assignee: Stephen Darlington
>>> 
>>> 
>>> When starting 2.7 with the ignite-kubernetes option I get the following
>> exception on startup:
>>> 
>>> {{[13:44:34,605][SEVERE][main][IgniteKernal] Got exception while
>> starting (will rollback startup
>> routine).}}{{java.lang.NoClassDefFoundError:
>> com/fasterxml/jackson/annotation/JsonView}}{{ at
>> com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.(JacksonAnnotationIntrospector.java:37)}}{{
>> at
>> com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:291)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:151)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)}}
>>> 
>>> It's clearly missing a dependency.
>>> 
>>> 
>>> 
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v7.6.3#76005)
>> 
>> 
>> 

[jira] [Created] (IGNITE-10647) Spark example code potentially confusing

[Stephen Darlington (JIRA)]

Stephen Darlington created IGNITE-10647:
---

 Summary: Spark example code potentially confusing
 Key: IGNITE-10647
 URL: https://issues.apache.org/jira/browse/IGNITE-10647
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.7
Reporter: Stephen Darlington
Assignee: Stephen Darlington


This started with a [question on Stack 
Overflow|https://stackoverflow.com/questions/53704478/apache-ignite-spark-dataframes-client-vs-server-doubts/53704993#53704993].
 In short, the user copy-and-pasted the whole sample code over, without 
realising that in addition to firing up a Spark client it also runs an Ignite 
server.

The advantage of this is obviously that the code is entirely self-contained.

However, there's little to suggest what's going on in the code, which bits are 
just setting stuff up and which bits are actually demonstrating the Spark 
integration.

Maybe there's a way to restructure it to make it clearer? Or a few extra 
comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10646) Typo in partition map exchange diagnostics

[Stephen Darlington (JIRA)]

Stephen Darlington created IGNITE-10646:
---

 Summary: Typo in partition map exchange diagnostics
 Key: IGNITE-10646
 URL: https://issues.apache.org/jira/browse/IGNITE-10646
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.7
Reporter: Stephen Darlington
Assignee: Stephen Darlington


If the partition map failed to exchange in time, you get the following error 
message and suggestion:
{code:java}
"Failed to wait for partition map exchange [...] Consider changing 
TransactionConfiguration.txTimeoutOnPartitionMapSynchronization"{code}
Unfortunately there is no {{txTimeoutOnPartitionMapSynchronization}} property. 
Maybe there used to be? In any case, the code looks at 
{{txTimeoutOnPartitionMapExchange}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Created] (IGNITE-10577) ignite-kubernetes is missing jackson-annotations dependency

[Stephen Darlington]

Not sure what the etiquette on this is, but this is a regression in 2.7. I 
added a simple patch to the ticket. 

Regards,
Stephen

> On 6 Dec 2018, at 14:42, Stephen Darlington (JIRA)  wrote:
> 
> Stephen Darlington created IGNITE-10577:
> ---
> 
> Summary: ignite-kubernetes is missing jackson-annotations 
> dependency
> Key: IGNITE-10577
> URL: https://issues.apache.org/jira/browse/IGNITE-10577
> Project: Ignite
>  Issue Type: Bug
>  Components: build
>Affects Versions: 2.7
>Reporter: Stephen Darlington
>Assignee: Stephen Darlington
> 
> 
> When starting 2.7 with the ignite-kubernetes option I get the following 
> exception on startup:
>  
> {{[13:44:34,605][SEVERE][main][IgniteKernal] Got exception while starting 
> (will rollback startup routine).}}{{java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/annotation/JsonView}}{{ at 
> com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.(JacksonAnnotationIntrospector.java:37)}}{{
>  at 
> com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:291)}}{{
>  at 
> org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:151)}}{{
>  at 
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)}}{{
>  at 
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)}}{{
>  at 
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)}}{{
>  at 
> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)}}
>  
> It's clearly missing a dependency.
> 
> 
> 
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)

[jira] [Created] (IGNITE-10577) ignite-kubernetes is missing jackson-annotations dependency

[Stephen Darlington (JIRA)]

Stephen Darlington created IGNITE-10577:
---

 Summary: ignite-kubernetes is missing jackson-annotations 
dependency
 Key: IGNITE-10577
 URL: https://issues.apache.org/jira/browse/IGNITE-10577
 Project: Ignite
  Issue Type: Bug
  Components: build
Affects Versions: 2.7
Reporter: Stephen Darlington
Assignee: Stephen Darlington


When starting 2.7 with the ignite-kubernetes option I get the following 
exception on startup:
 
{{[13:44:34,605][SEVERE][main][IgniteKernal] Got exception while starting (will 
rollback startup routine).}}{{java.lang.NoClassDefFoundError: 
com/fasterxml/jackson/annotation/JsonView}}{{ at 
com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.(JacksonAnnotationIntrospector.java:37)}}{{
 at 
com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:291)}}{{ 
at 
org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:151)}}{{
 at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)}}{{
 at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)}}{{
 at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)}}{{
 at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)}}
 
It's clearly missing a dependency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [IMPORTANT] Future of Binary Objects

[Stephen Darlington]

Possibly heading into wishlist rather than practical territory here, but you 
did ask...

> What we need is to introduce an interface which will convert a pair of
> key-value objects into a row. This row will be used to store data and to
> get fields from it. 

Rather than mapping objects to a row, how about mapping to a more general 
“internal storage” interface? Assuming that all the data for a row is stored 
together makes it difficult to implement any optimisations that spans multiple 
rows. Think of a string state field where there are only five known values… we 
currently repeat the text over and over. Or a full column store backend.

Regards,
Stephen

Becoming a contributor

[Stephen Darlington]

Hi,

I’ve started to work on a couple of tickets:

https://issues.apache.org/jira/browse/IGNITE-10155
https://issues.apache.org/jira/browse/IGNITE-1436

It would be great if I could be added to the list of contributors so I can 
assign the tickets to myself.

Thanks,
Stephen

[jira] [Created] (IGNITE-10155) Yardstick should allow modifier when starting servers

[Stephen Darlington (JIRA)]

Stephen Darlington created IGNITE-10155:
---

 Summary: Yardstick should allow modifier when starting servers
 Key: IGNITE-10155
 URL: https://issues.apache.org/jira/browse/IGNITE-10155
 Project: Ignite
  Issue Type: Improvement
  Components: yardstick
Reporter: Stephen Darlington


Currently, Yardstick will start as many copies of Ignite as specified in the 
configuration file. However there is currently no way to customise the 
behaviour of each of those nodes. The example I'm currently working with is 
forcing CPU affinity.

Related: it's also error prone to specify multiple nodes (when the number is 
large). It would be nice to have some kind of shorthand to say you want, say, 
four nodes on this particular host.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSSION] Spark Data Frame through Thin Client

[Stephen Darlington]

Are you suggesting making the Thin Client deployment an option or as a 
replacement for the thick-client? If the latter, do we risk making future 
desirable changes more difficult (or impossible)? I’m thinking specifically 
about better support for Spark Streaming, where the lack  of continuous query 
support in thin clients removes a significant optimisation option. I’m sure 
there are other use cases.

Regards,
Stephen

> On 21 Oct 2018, at 09:08, Nikolay Izhikov  wrote:
> 
> Valentin.
> 
> Seems, You made several suggestions, which is not always true, from my point 
> of view:
> 
> 1. "We have access to Spark cluster installation to perform deployment steps" 
> - this is not true in cloud or enterprise environment.
> 
> 2. "Spark cluster is used only for Ignite integration".
> From what I know computational resources for big Spark cluster is divided by 
> many business divisions.
> And it is not convenient to perform some deployment steps on this cluster.
> 
> 3. "When Ignite + Spark are used in real production it's OK to have 
> reasonable deployment overhead"
> What about developer who want to play with this integration?
> And want to do it quickly to see how it works in real life examples.
> Can we do his life much easier?
> 
>> First of all, they will exist with thin client either.
> 
> Spark have an ability to deploy jars on worker and add it to application 
> tasks classpath.
> For 2.6 we must deploy 11 additional jars to start using Ignite.
> Please, see my example on the bottom of documentation page [1]
> 
> Does cache-api-1.0.0.jar and h2-1.4.195.jar seems like obvious dependencies 
> for Ignite integration for you?
> And for our users? :)
> 
> Actually, list of dependencies will be changed in 2.7 - new version of 
> jcache, new version of h2
> So user should change it in code or perform additional deployment steps.
> 
> It overkill for me.
> 
> On the other hand - thin client requires only 1 jar.
> Moreover, thin client protocol have the backward compatibility.
> So thin client will perform correctly when Ignite cluster will be updated 
> from 2.6 to 2.7.
> So, with Spark integration via thin client we will be able to update Ignite 
> cluster and Spark integration separately.
> For now, we should do it in one big step.
> 
> What do you think?
> 
> [1] https://apacheignite-fs.readme.io/docs/installation-deployment
> 
> В Сб, 20/10/2018 в 18:33 -0700, Valentin Kulichenko пишет:
>> Guys,
>> 
>> From my experience, Ignite and Spark clusters typically run in the same
>> environment, which makes client node a more preferable option. Mainly,
>> because of performance. BTW, I doubt partition-awareness on thin client
>> will help either, because in dataframes we only run SQL queries and I
>> believe thin client will execute them through a proxy anyway. But correct
>> me if I’m wrong.
>> 
>> Either way, it sounds like we just have usability issues with Ignite/Spark
>> integration. Why don’t we concentrate on fixing them then? For example, #3
>> can be fixed by loading XML content on master and then distributing it to
>> workers, instead of loading on every worker independently. Then there are
>> certain procedures like deploying JARs, etc. First of all, they will exist
>> with thin client either. Second of all, I’m sure there are ways to simplify
>> this procedures and make integration easier. My opinion is that working on
>> such improvements is going to add more value than another implementation
>> based on thin client.
>> 
>> -Val
>> 
>> On Sat, Oct 20, 2018 at 4:03 PM Denis Magda  wrote:
>> 
>>> Hello Nikolay,
>>> 
>>> Your proposal sounds reasonable. However, I would suggest us to wait while
>>> partition-awareness is supported for Java thin client first. With that
>>> feature, the client can connect to any node directly while presently all
>>> the communication goes through a proxy (a node the client is connected to).
>>> All of that is bad for performance.
>>> 
>>> 
>>> Vladimir, how hard would it be to support the partition-awareness for Java
>>> client? Probably, Nikolay can take over.
>>> 
>>> --
>>> Denis
>>> 
>>> 
>>> On Sat, Oct 20, 2018 at 2:09 PM Nikolay Izhikov 
>>> wrote:
>>> 
 Hello, Igniters.
 
 Currently, Spark Data Frame integration implemented via client node
 connection.
 Whenever we need to retrieve some data into Spark worker(or master) from
 Ignite we start a client node.
 
 It has several major disadvantages:
 
1. We should copy whole Ignite distribution on to each Spark
 worker [1]
2. We should copy whole Ignite distribution on to Spark master to
 get catalogue works.
3. We should have the same absolute path to Ignite configuration
 file on every worker and provide it during data frame construction [2]
4. We should additionally configure Spark workerks classpath to
 include Ignite libraries.
 
 For now, almost all operation we need to do in Spark Data Frame