Re: Ignite community is building Calcite-based prototype

[Denis Magda]

Hi Julian,

Nice to e-meet you and thanks for being ready to help! Hopefully, the
Ignite community will be able to contribute valuable changes back to
Calcite as part of this activity - "pay good for good" :)

You are right that distributed computing, massive-parallel processing, and
calculations/querying at scale is what Ignite is targeted for. However,
while Drill is designed for analytics and IoTDB is for time-series, Ignite
is primarily used for OLTP with an increasing number of real-time analytics
use cases (no adhoc).

Let's stay in touch!

-
Denis


On Tue, Oct 1, 2019 at 6:42 AM Julian Feinauer 
wrote:

> Hi Igor,
>
> I agree that it should be rather similar to what Drill did as distributed
> computing also is a big concern for Ignite, I guess, right?
>
> Julian
>
> Am 01.10.19, 15:06 schrieb "Seliverstov Igor" :
>
> Guys,
>
> The better link:
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-37:+New+query+execution+engine
> >
>
> Almost everything you may see by the link is the same as Drill guys
> already did, the difference is in details but the idea is the same.
>
> Of course we’ll face many issues while development and I'll appreciate
> if some of you assist us.
>
> Regards,
> Igor
>
> > 1 окт. 2019 г., в 12:32, Julian Feinauer <
> j.feina...@pragmaticminds.de> написал(а):
> >
> > Hi Denis,
> >
> > Nice to hear from you and the ignite team... that sounds like an
> excellent idea. I liked the idea of Ignite since I heard about it (I think
> when it became TLP back then). So I would be happy to help you if you have
> specific questions... I‘m currently working on a related topic, namely
> integrate calcite as SQL Layer into Apache IoTDB .
> >
> > Best
> > Julian
> >
> > Holen Sie sich Outlook für iOS
> > 
> > Von: Denis Magda 
> > Gesendet: Tuesday, October 1, 2019 2:37:20 AM
> > An: d...@calcite.apache.org ; dev <
> dev@ignite.apache.org>
> > Betreff: Ignite community is building Calcite-based prototype
> >
> > Hey ASF-mates,
> >
> > Just wanted to send a note for Ignite dev community who has started
> > prototyping
> > <
> http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-td43724.html
> >
> > with a new Ignite SQL engine and Calcite was selected as the most
> favorable
> > option.
> >
> > We will truly appreciate if you help us with questions that might
> hit your
> > dev list. Ignite folks have already studied Calcite well enough and
> carried
> > on with the integration, but there might be tricky parts that would
> require
> > your expertise.
> >
> > Btw, if anybody is interested in Ignite (memory-centric database and
> > compute platform) or would like to learn more details about the
> prototype
> > or join its development, please check these links or send us a note:
> >
> >   - https://ignite.apache.org
> >   -
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
> >
> >
> > -
> > Denis,
> > Ignite PMC Chair
>
>
>
>

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Denis Magda]

Alexey Z.,

Could you please answer some of the questions


> - IGNITE-11942 IGFS and Hadoop Accelerator Discontinuation [2].
> Probably should be moved to the next release due to dependency on
> Tensorflow. Need to check. (Andrey Gura)


Can we decouple Tensorflow from the IGFS?

Mark all the issues related to ML, Spark 2.4, Monitoring major
> features and track their comletion to be sure on there is no
> unfinished major changes will be present in 2.8 release.
> - ML (Alexey Zinoviev)


Are there any other ML contributors who will be helping you with this
release?

-
Denis


On Tue, Oct 1, 2019 at 6:56 AM Maxim Muzafarov  wrote:

> Igniters,
>
>
> Here is the list of activities we've agreed on to prepare the Apache
> Ignite 2.8 release.
>
> 1.
> Preliminary release dates with the ability to shift them if some of
> the planned activities will not be finalized. But anyway we should
> build our engagement based on these dates.
>
> Scope Freeze: November 5, 2019
> Code Freeze: December 2, 2019
> Voting Date: January 10, 2020
> Release Date: January 17, 2020
>
> 2.
> Mark all the issues related to ML, Spark 2.4, Monitoring major
> features and track their comletion to be sure on there is no
> unfinished major changes will be present in 2.8 release.
> - ML (Alexey Zinoviev)
> - Spark [1] [4] (Alexey Zinoviev)
> - Monitoring (Nikolay Izhikov)
>
> 3.
> Review and mark all the partially completed major issues currently
> present in the master branch.
> - major IEPs e.g. IEP-18, non-blocking PME (Maxim Muzafarov)
> - review major commits in the master branch (Maxim Muzafarov)
>
> 4.
> Review blocker issues currently pinned to 2.8 release.
> - IGNITE-11942 IGFS and Hadoop Accelerator Discontinuation [2].
> Probably should be moved to the next release due to dependency on
> Tensorflow. Need to check. (Andrey Gura)
> - IGNITE-9489 CorruptedTreeException on index create [3]. Check all
> the issues releated to this. Some of them already fixed by GridGain.
> Need to check. (Andrey Gura)
> - IGNITE-12181 Rebalance hangs on BLT change. The cause has been
> found. Will be fixed. (Maxim Muzafarov)
> - Need to check all the other blocker issues (Maxim Muzafarov)
>
> 5.
> QA regression (2.7 -> 2.8). I'll provide additional details when I'll get
> them.
> Review and check test-cases, optioannly schedule meeting (Maxim Muzafarov)
>
> 6.
> (optional) Need to create an INFRA ticket to add `Epic` JIRA issue
> type to the Apache Ignite JIRA. The issue [6] has been created but not
> sure that I have the right permission to do so.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/The-Spark-2-4-support-td43777.html
> [2] https://issues.apache.org/jira/browse/IGNITE-11942
> [3] https://issues.apache.org/jira/browse/IGNITE-9489
> [4] https://issues.apache.org/jira/browse/IGNITE-12054
> [5] https://issues.apache.org/jira/browse/IGNITE-12181
> [6] https://issues.apache.org/jira/browse/INFRA-19164
>
> On Mon, 30 Sep 2019 at 12:14, Ivan Pavlukhin  wrote:
> >
> > Maxim, Folks,
> >
> > Could you please share a results of the Slack discussion from Sep 25?
> >
> > ср, 25 сент. 2019 г. в 15:50, Dmitriy Pavlov :
> > >
> > > Hi Maxim,
> > >
> > > Thank you for preparing the release page!
> > >
> > > Could you please add Require release notes filter? You can find an
> example
> > > in
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7.6
> > >
> > > Sincerely
> > > Dmitriy Pavlov
> > >
> > > ср, 25 сент. 2019 г. в 11:58, Maxim Muzafarov :
> > >
> > > > Igniters,
> > > >
> > > >
> > > > It's true that we are still discussing the release dates. But
> > > > nevertheless, all the release blockers are important since some of
> > > > them may require more than one month to be fixed. Let's discuss today
> > > > how we will handle these issues and track Monitoring and ML major
> > > > features to get them into the next release.
> > > >
> > > > The meeting already scheduled. We will use the ASF Slack on September
> > > > 25-th, 17-00 (MSK).
> > > > I've created the channel [2] #ignite-release-2_8 please, join.
> > > > (Discussion will be on Russin language).
> > > >
> > > >
> > > >
> > > > Please, also note that I've created the 2.8 release confluence page
> > > > [1] with additional information. I will review all the issues we have
> > > > and will move some of them to 2.9.
> > > > But currently, we've had pinned to 2.8:
> > > >
> > > > - 604 open issues
> > > > - 57 in progress issues
> > > > - 34 patch available issues
> > > >
> > > > - 17 issues marked as the release blockers
> > > > - 4 of release blocker issues are unassigned
> > > >
> > > >
> > > > [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8
> > > > [2] https://app.slack.com/client/T4S1WH2J3/CNQ51M4FQ
> > > >
> > > > On Wed, 25 Sep 2019 at 10:21, Dmitriy Pavlov 
> wrote:
> > > > >
> > > > > Hi Igniters,
> > > > >
> > > > > I suppose discussion is still at phase 0-Initializing
> > > > > 

Re: ApacheCon North America 2020, project participation

[Denis Magda]

Hello Rich,

Please count on the Apache Ignite community. We'll collect topics and find
those who will be able to present at the NA event.

-
Denis


On Tue, Oct 1, 2019 at 9:37 AM Rich Bowen  wrote:

> Hi, folks,
>
> (Note: You're receiving this email because you're on the dev@ list for
> one or more Apache Software Foundation projects.)
>
> For ApacheCon North America 2019, we asked projects to participate in
> the creation of project/topic specific tracks. This was very successful,
> with about 15 projects stepping up to curate the content for their
> track/summit/event.
>
> We need to know if you're going to do the same for 2020. This informs
> how large a venue we book for the event, how long the event runs, and
> many other considerations.
>
> If you intend to participate again in 2020, we need to hear from you on
> the plann...@apachecon.com mailing list. This is not a firm commitment,
> but we need to know if you're, say, 75% confident that you'll be
> participating.
>
> And, no, we do not have any details at all, but assume that it will be
> in roughly the same calendar space as this year's event, ie, somewhere
> in the August-October timeframe.
>
> Thanks.
>
> --
> Rich Bowen
> VP Conferences
> The Apache Software Foundation
> @apachecon
>

Re: [IGNITE-9836] Invalid check of ea java versions

[Ilya Kasnacheev]

Hello!

Thank you, I have merged both.

Regards,
-- 
Ilya Kasnacheev


пт, 27 сент. 2019 г. в 14:36, Stephen Darlington <
stephen.darling...@gridgain.com>:

> Done: https://github.com/apache/ignite/pull/6920
>
> While we’re talking about the startup scripts…
> https://issues.apache.org/jira/browse/IGNITE-11856
>
> Regards,
> Stephen
>
> > On 26 Sep 2019, at 17:02, Ilya Kasnacheev 
> wrote:
> >
> > Hello!
> >
> > Please do!
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вт, 17 сент. 2019 г. в 11:13, Stephen Darlington <
> > stephen.darling...@gridgain.com>:
> >
> >> I can’t take any credit for the patch but if the original author has
> lost
> >> interest I’m happy to help push it through.
> >>
> >> Regards,
> >> Stephen
> >>
> >>> On 16 Sep 2019, at 19:31, Denis Magda  wrote:
> >>>
> >>> Stephen,
> >>>
> >>> Thanks for sending the patch! Seems that Igniters are already actively
> >>> reviewing it in JIRA.
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Mon, Sep 16, 2019 at 7:03 AM Stephen Darlington <
> >>> stephen.darling...@gridgain.com> wrote:
> >>>
>  Hi,
> 
>  Would someone mind taking a quick look at this ticket? Basically, a
> >> clean
>  download of Ignite won’t start if the version of Java you’re using
> has a
>  number like “java version "1.8.0_202-ea””. (This is the default if you
> >> get
>  your JDK using Homebrew on a Mac.)
> 
> > https://issues.apache.org/jira/browse/IGNITE-9836 <
>  https://issues.apache.org/jira/browse/IGNITE-9836>
> 
>  This has been bugging me for ages and now that I look at it I find
> that
>  there’s already a tiny, working patch available.
> 
>  Regards,
>  Stephen
> >>
> >>
> >>
>
>
>

ApacheCon North America 2020, project participation

[Rich Bowen]

Hi, folks,

(Note: You're receiving this email because you're on the dev@ list for
one or more Apache Software Foundation projects.)

For ApacheCon North America 2019, we asked projects to participate in
the creation of project/topic specific tracks. This was very successful,
with about 15 projects stepping up to curate the content for their
track/summit/event.

We need to know if you're going to do the same for 2020. This informs
how large a venue we book for the event, how long the event runs, and
many other considerations.

If you intend to participate again in 2020, we need to hear from you on
the plann...@apachecon.com mailing list. This is not a firm commitment,
but we need to know if you're, say, 75% confident that you'll be
participating.

And, no, we do not have any details at all, but assume that it will be
in roughly the same calendar space as this year's event, ie, somewhere
in the August-October timeframe.

Thanks.

-- 
Rich Bowen
VP Conferences
The Apache Software Foundation
@apachecon

[jira] [Created] (IGNITE-12254) IO errors during write header of WAL files in FSYNC mode should be handled by failure handler

[Aleksey Plekhanov (Jira)]

Aleksey Plekhanov created IGNITE-12254:
--

 Summary: IO errors during write header of WAL files in FSYNC mode 
should be handled by failure handler
 Key: IGNITE-12254
 URL: https://issues.apache.org/jira/browse/IGNITE-12254
 Project: Ignite
  Issue Type: Bug
Reporter: Aleksey Plekhanov


Currently, such errors can hang the cluster.

Reproducer:

{code:java}
@Test
public void testWalFsyncIOError() throws Exception {
cleanPersistenceDir();

IgniteConfiguration cfg = new IgniteConfiguration();

cfg.setCacheConfiguration(new 
CacheConfiguration(DEFAULT_CACHE_NAME).setAtomicityMode(ATOMIC));

cfg.setDataStorageConfiguration(
new DataStorageConfiguration()
.setDefaultDataRegionConfiguration(
new DataRegionConfiguration()
.setMaxSize(100L * 1024 * 1024)
.setPersistenceEnabled(true))
.setWalMode(WALMode.FSYNC)
.setWalSegmentSize(512 * 1024)
.setWalBufferSize(512 * 1024));

IgniteEx ignite0 = startGrid(new 
IgniteConfiguration(cfg).setIgniteInstanceName("ignite0"));
IgniteEx ignite1 = startGrid(new 
IgniteConfiguration(cfg).setIgniteInstanceName("ignite1"));

ignite0.cluster().active(true);

IgniteCache cache = ignite0.cache(DEFAULT_CACHE_NAME);

for (int i = 0; i < 1_000; i++)
cache.put(i, "Test value " + i);


((FileWriteAheadLogManager)ignite1.context().cache().context().wal()).setFileIOFactory(new
 FileIOFactory() {
FileIOFactory delegateFactory = new RandomAccessFileIOFactory();
@Override public FileIO create(File file, OpenOption... modes) 
throws IOException {
final FileIO delegate = delegateFactory.create(file, modes);

return new FileIODecorator(delegate) {
@Override public int write(ByteBuffer srcBuf) throws 
IOException {
throw new IOException("No space left on device");
}
};
}
});

for (int i = 0; i < 2_000; i++)
try {
cache.put(i, "Test value " + i);
}
catch (Exception ignore) {
}
}
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Maxim Muzafarov]

Alexey,


Thank you for setting up the right issue statuses. I've updated the
list of components we are waiting for [1].
Please, check that I've not missed anything.

One more thing: I propose to exclude `minor' priority issues from that
list or mark them `major`.
WDYT?


[1] 
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Awatingfeaturescompletion

On Tue, 1 Oct 2019 at 17:39, Petr Ivanov  wrote:
>
> Igniters!
>
>
> Do we want to have full JDK 11 support (compile and run) this release?
> Or should we postpone it until at least 3.0?
>
>
> > On 1 Oct 2019, at 17:19, Alexey Zinoviev  wrote:
> >
> > Hi, you could find here
> >
> > ML tickets that should be resolved in the Ignite 2.8 (Umbrella tickets,
> > bugs, minor improvements, integration tests via examples)
> > https://issues.apache.org/jira/browse/IGNITE-10792?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%202.8%20AND%20component%20%3D%20ml%20ORDER%20BY%20priority%20DESC
> >
> >
> > Spark Tickets (Support of Spark 2.4 + bug fixing)
> > https://issues.apache.org/jira/browse/IGNITE-12054?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%202.8%20AND%20component%20%3D%20spark%20ORDER%20BY%20priority%20DESC
> >
> >
> > All ticket priorities are correct.
> >
> > The final scope of Spark-Ignite integration will be discussed next week on
> > ASF slack and result will be published on dev-list and shared with the
> > community.
> >
> >
> >
> > вт, 1 окт. 2019 г. в 16:56, Maxim Muzafarov :
> >
> >> Igniters,
> >>
> >>
> >> Here is the list of activities we've agreed on to prepare the Apache
> >> Ignite 2.8 release.
> >>
> >> 1.
> >> Preliminary release dates with the ability to shift them if some of
> >> the planned activities will not be finalized. But anyway we should
> >> build our engagement based on these dates.
> >>
> >> Scope Freeze: November 5, 2019
> >> Code Freeze: December 2, 2019
> >> Voting Date: January 10, 2020
> >> Release Date: January 17, 2020
> >>
> >> 2.
> >> Mark all the issues related to ML, Spark 2.4, Monitoring major
> >> features and track their comletion to be sure on there is no
> >> unfinished major changes will be present in 2.8 release.
> >> - ML (Alexey Zinoviev)
> >> - Spark [1] [4] (Alexey Zinoviev)
> >> - Monitoring (Nikolay Izhikov)
> >>
> >> 3.
> >> Review and mark all the partially completed major issues currently
> >> present in the master branch.
> >> - major IEPs e.g. IEP-18, non-blocking PME (Maxim Muzafarov)
> >> - review major commits in the master branch (Maxim Muzafarov)
> >>
> >> 4.
> >> Review blocker issues currently pinned to 2.8 release.
> >> - IGNITE-11942 IGFS and Hadoop Accelerator Discontinuation [2].
> >> Probably should be moved to the next release due to dependency on
> >> Tensorflow. Need to check. (Andrey Gura)
> >> - IGNITE-9489 CorruptedTreeException on index create [3]. Check all
> >> the issues releated to this. Some of them already fixed by GridGain.
> >> Need to check. (Andrey Gura)
> >> - IGNITE-12181 Rebalance hangs on BLT change. The cause has been
> >> found. Will be fixed. (Maxim Muzafarov)
> >> - Need to check all the other blocker issues (Maxim Muzafarov)
> >>
> >> 5.
> >> QA regression (2.7 -> 2.8). I'll provide additional details when I'll get
> >> them.
> >> Review and check test-cases, optioannly schedule meeting (Maxim Muzafarov)
> >>
> >> 6.
> >> (optional) Need to create an INFRA ticket to add `Epic` JIRA issue
> >> type to the Apache Ignite JIRA. The issue [6] has been created but not
> >> sure that I have the right permission to do so.
> >>
> >> [1]
> >> http://apache-ignite-developers.2346864.n4.nabble.com/The-Spark-2-4-support-td43777.html
> >> [2] https://issues.apache.org/jira/browse/IGNITE-11942
> >> [3] https://issues.apache.org/jira/browse/IGNITE-9489
> >> [4] https://issues.apache.org/jira/browse/IGNITE-12054
> >> [5] https://issues.apache.org/jira/browse/IGNITE-12181
> >> [6] https://issues.apache.org/jira/browse/INFRA-19164
> >>
> >> On Mon, 30 Sep 2019 at 12:14, Ivan Pavlukhin  wrote:
> >>>
> >>> Maxim, Folks,
> >>>
> >>> Could you please share a results of the Slack discussion from Sep 25?
> >>>
> >>> ср, 25 сент. 2019 г. в 15:50, Dmitriy Pavlov :
> 
>  Hi Maxim,
> 
>  Thank you for preparing the release page!
> 
>  Could you please add Require release notes filter? You can find an
> >> example
>  in
> >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7.6
> 
>  Sincerely
>  Dmitriy Pavlov
> 
>  ср, 25 сент. 2019 г. в 11:58, Maxim Muzafarov :
> 
> > Igniters,
> >
> >
> > It's true that we are still discussing the release dates. But
> > nevertheless, all the release blockers are important since some of
> > them may require more than one month to be fixed. Let's 

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Petr Ivanov]

Igniters!


Do we want to have full JDK 11 support (compile and run) this release?
Or should we postpone it until at least 3.0?


> On 1 Oct 2019, at 17:19, Alexey Zinoviev  wrote:
> 
> Hi, you could find here
> 
> ML tickets that should be resolved in the Ignite 2.8 (Umbrella tickets,
> bugs, minor improvements, integration tests via examples)
> https://issues.apache.org/jira/browse/IGNITE-10792?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%202.8%20AND%20component%20%3D%20ml%20ORDER%20BY%20priority%20DESC
> 
> 
> Spark Tickets (Support of Spark 2.4 + bug fixing)
> https://issues.apache.org/jira/browse/IGNITE-12054?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%202.8%20AND%20component%20%3D%20spark%20ORDER%20BY%20priority%20DESC
> 
> 
> All ticket priorities are correct.
> 
> The final scope of Spark-Ignite integration will be discussed next week on
> ASF slack and result will be published on dev-list and shared with the
> community.
> 
> 
> 
> вт, 1 окт. 2019 г. в 16:56, Maxim Muzafarov :
> 
>> Igniters,
>> 
>> 
>> Here is the list of activities we've agreed on to prepare the Apache
>> Ignite 2.8 release.
>> 
>> 1.
>> Preliminary release dates with the ability to shift them if some of
>> the planned activities will not be finalized. But anyway we should
>> build our engagement based on these dates.
>> 
>> Scope Freeze: November 5, 2019
>> Code Freeze: December 2, 2019
>> Voting Date: January 10, 2020
>> Release Date: January 17, 2020
>> 
>> 2.
>> Mark all the issues related to ML, Spark 2.4, Monitoring major
>> features and track their comletion to be sure on there is no
>> unfinished major changes will be present in 2.8 release.
>> - ML (Alexey Zinoviev)
>> - Spark [1] [4] (Alexey Zinoviev)
>> - Monitoring (Nikolay Izhikov)
>> 
>> 3.
>> Review and mark all the partially completed major issues currently
>> present in the master branch.
>> - major IEPs e.g. IEP-18, non-blocking PME (Maxim Muzafarov)
>> - review major commits in the master branch (Maxim Muzafarov)
>> 
>> 4.
>> Review blocker issues currently pinned to 2.8 release.
>> - IGNITE-11942 IGFS and Hadoop Accelerator Discontinuation [2].
>> Probably should be moved to the next release due to dependency on
>> Tensorflow. Need to check. (Andrey Gura)
>> - IGNITE-9489 CorruptedTreeException on index create [3]. Check all
>> the issues releated to this. Some of them already fixed by GridGain.
>> Need to check. (Andrey Gura)
>> - IGNITE-12181 Rebalance hangs on BLT change. The cause has been
>> found. Will be fixed. (Maxim Muzafarov)
>> - Need to check all the other blocker issues (Maxim Muzafarov)
>> 
>> 5.
>> QA regression (2.7 -> 2.8). I'll provide additional details when I'll get
>> them.
>> Review and check test-cases, optioannly schedule meeting (Maxim Muzafarov)
>> 
>> 6.
>> (optional) Need to create an INFRA ticket to add `Epic` JIRA issue
>> type to the Apache Ignite JIRA. The issue [6] has been created but not
>> sure that I have the right permission to do so.
>> 
>> [1]
>> http://apache-ignite-developers.2346864.n4.nabble.com/The-Spark-2-4-support-td43777.html
>> [2] https://issues.apache.org/jira/browse/IGNITE-11942
>> [3] https://issues.apache.org/jira/browse/IGNITE-9489
>> [4] https://issues.apache.org/jira/browse/IGNITE-12054
>> [5] https://issues.apache.org/jira/browse/IGNITE-12181
>> [6] https://issues.apache.org/jira/browse/INFRA-19164
>> 
>> On Mon, 30 Sep 2019 at 12:14, Ivan Pavlukhin  wrote:
>>> 
>>> Maxim, Folks,
>>> 
>>> Could you please share a results of the Slack discussion from Sep 25?
>>> 
>>> ср, 25 сент. 2019 г. в 15:50, Dmitriy Pavlov :
 
 Hi Maxim,
 
 Thank you for preparing the release page!
 
 Could you please add Require release notes filter? You can find an
>> example
 in
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7.6
 
 Sincerely
 Dmitriy Pavlov
 
 ср, 25 сент. 2019 г. в 11:58, Maxim Muzafarov :
 
> Igniters,
> 
> 
> It's true that we are still discussing the release dates. But
> nevertheless, all the release blockers are important since some of
> them may require more than one month to be fixed. Let's discuss today
> how we will handle these issues and track Monitoring and ML major
> features to get them into the next release.
> 
> The meeting already scheduled. We will use the ASF Slack on September
> 25-th, 17-00 (MSK).
> I've created the channel [2] #ignite-release-2_8 please, join.
> (Discussion will be on Russin language).
> 
> 
> 
> Please, also note that I've created the 2.8 release confluence page
> [1] with additional information. I will review all the issues we have
> and will move some of them to 2.9.
> But currently, we've had pinned to 2.8:
> 
> - 604 

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Alexey Zinoviev]

Hi, you could find here

ML tickets that should be resolved in the Ignite 2.8 (Umbrella tickets,
bugs, minor improvements, integration tests via examples)
https://issues.apache.org/jira/browse/IGNITE-10792?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%202.8%20AND%20component%20%3D%20ml%20ORDER%20BY%20priority%20DESC


Spark Tickets (Support of Spark 2.4 + bug fixing)
https://issues.apache.org/jira/browse/IGNITE-12054?jql=project%20%3D%20IGNITE%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Patch%20Available%22)%20AND%20affectedVersion%20%3D%202.8%20AND%20component%20%3D%20spark%20ORDER%20BY%20priority%20DESC


All ticket priorities are correct.

The final scope of Spark-Ignite integration will be discussed next week on
ASF slack and result will be published on dev-list and shared with the
community.



вт, 1 окт. 2019 г. в 16:56, Maxim Muzafarov :

> Igniters,
>
>
> Here is the list of activities we've agreed on to prepare the Apache
> Ignite 2.8 release.
>
> 1.
> Preliminary release dates with the ability to shift them if some of
> the planned activities will not be finalized. But anyway we should
> build our engagement based on these dates.
>
> Scope Freeze: November 5, 2019
> Code Freeze: December 2, 2019
> Voting Date: January 10, 2020
> Release Date: January 17, 2020
>
> 2.
> Mark all the issues related to ML, Spark 2.4, Monitoring major
> features and track their comletion to be sure on there is no
> unfinished major changes will be present in 2.8 release.
> - ML (Alexey Zinoviev)
> - Spark [1] [4] (Alexey Zinoviev)
> - Monitoring (Nikolay Izhikov)
>
> 3.
> Review and mark all the partially completed major issues currently
> present in the master branch.
> - major IEPs e.g. IEP-18, non-blocking PME (Maxim Muzafarov)
> - review major commits in the master branch (Maxim Muzafarov)
>
> 4.
> Review blocker issues currently pinned to 2.8 release.
> - IGNITE-11942 IGFS and Hadoop Accelerator Discontinuation [2].
> Probably should be moved to the next release due to dependency on
> Tensorflow. Need to check. (Andrey Gura)
> - IGNITE-9489 CorruptedTreeException on index create [3]. Check all
> the issues releated to this. Some of them already fixed by GridGain.
> Need to check. (Andrey Gura)
> - IGNITE-12181 Rebalance hangs on BLT change. The cause has been
> found. Will be fixed. (Maxim Muzafarov)
> - Need to check all the other blocker issues (Maxim Muzafarov)
>
> 5.
> QA regression (2.7 -> 2.8). I'll provide additional details when I'll get
> them.
> Review and check test-cases, optioannly schedule meeting (Maxim Muzafarov)
>
> 6.
> (optional) Need to create an INFRA ticket to add `Epic` JIRA issue
> type to the Apache Ignite JIRA. The issue [6] has been created but not
> sure that I have the right permission to do so.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/The-Spark-2-4-support-td43777.html
> [2] https://issues.apache.org/jira/browse/IGNITE-11942
> [3] https://issues.apache.org/jira/browse/IGNITE-9489
> [4] https://issues.apache.org/jira/browse/IGNITE-12054
> [5] https://issues.apache.org/jira/browse/IGNITE-12181
> [6] https://issues.apache.org/jira/browse/INFRA-19164
>
> On Mon, 30 Sep 2019 at 12:14, Ivan Pavlukhin  wrote:
> >
> > Maxim, Folks,
> >
> > Could you please share a results of the Slack discussion from Sep 25?
> >
> > ср, 25 сент. 2019 г. в 15:50, Dmitriy Pavlov :
> > >
> > > Hi Maxim,
> > >
> > > Thank you for preparing the release page!
> > >
> > > Could you please add Require release notes filter? You can find an
> example
> > > in
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7.6
> > >
> > > Sincerely
> > > Dmitriy Pavlov
> > >
> > > ср, 25 сент. 2019 г. в 11:58, Maxim Muzafarov :
> > >
> > > > Igniters,
> > > >
> > > >
> > > > It's true that we are still discussing the release dates. But
> > > > nevertheless, all the release blockers are important since some of
> > > > them may require more than one month to be fixed. Let's discuss today
> > > > how we will handle these issues and track Monitoring and ML major
> > > > features to get them into the next release.
> > > >
> > > > The meeting already scheduled. We will use the ASF Slack on September
> > > > 25-th, 17-00 (MSK).
> > > > I've created the channel [2] #ignite-release-2_8 please, join.
> > > > (Discussion will be on Russin language).
> > > >
> > > >
> > > >
> > > > Please, also note that I've created the 2.8 release confluence page
> > > > [1] with additional information. I will review all the issues we have
> > > > and will move some of them to 2.9.
> > > > But currently, we've had pinned to 2.8:
> > > >
> > > > - 604 open issues
> > > > - 57 in progress issues
> > > > - 34 patch available issues
> > > >
> > > > - 17 issues marked as the release blockers
> > > > - 4 of release blocker issues are unassigned
> > > >
> > > >
> > > > [1]
> 

[jira] [Created] (IGNITE-12253) [ML] Refactor current hierarchy of ML exeptions

[Alexey Zinoviev (Jira)]

Alexey Zinoviev created IGNITE-12253:


 Summary: [ML] Refactor current hierarchy of ML exeptions
 Key: IGNITE-12253
 URL: https://issues.apache.org/jira/browse/IGNITE-12253
 Project: Ignite
  Issue Type: Improvement
  Components: ml
Affects Versions: 2.8
Reporter: Alexey Zinoviev
Assignee: Alexey Zinoviev


The current hierarchy of ML exceptions is too separated and have no common root 
and rules of writing and logging



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12252) Runtime exceptions during rebalancing should be handled

[Aleksey Plekhanov (Jira)]

Aleksey Plekhanov created IGNITE-12252:
--

 Summary: Runtime exceptions during rebalancing should be handled
 Key: IGNITE-12252
 URL: https://issues.apache.org/jira/browse/IGNITE-12252
 Project: Ignite
  Issue Type: Bug
Reporter: Aleksey Plekhanov


Rebalancing should handle runtime exceptions by failure handler. In current 
implementation runtime exceptions just ignored. They were handled by IO worker 
before IGNITE-3195.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12251) Calcite exchange rules

[Igor Seliverstov (Jira)]

Igor Seliverstov created IGNITE-12251:
-

 Summary: Calcite exchange rules
 Key: IGNITE-12251
 URL: https://issues.apache.org/jira/browse/IGNITE-12251
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Igor Seliverstov
Assignee: Igor Seliverstov


Need to implement IgniteDistributionTraitDef and utility methods (to calculate 
distribution trait for each node type based on its position, type and children 
nodes)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Maxim Muzafarov]

Igniters,


Here is the list of activities we've agreed on to prepare the Apache
Ignite 2.8 release.

1.
Preliminary release dates with the ability to shift them if some of
the planned activities will not be finalized. But anyway we should
build our engagement based on these dates.

Scope Freeze: November 5, 2019
Code Freeze: December 2, 2019
Voting Date: January 10, 2020
Release Date: January 17, 2020

2.
Mark all the issues related to ML, Spark 2.4, Monitoring major
features and track their comletion to be sure on there is no
unfinished major changes will be present in 2.8 release.
- ML (Alexey Zinoviev)
- Spark [1] [4] (Alexey Zinoviev)
- Monitoring (Nikolay Izhikov)

3.
Review and mark all the partially completed major issues currently
present in the master branch.
- major IEPs e.g. IEP-18, non-blocking PME (Maxim Muzafarov)
- review major commits in the master branch (Maxim Muzafarov)

4.
Review blocker issues currently pinned to 2.8 release.
- IGNITE-11942 IGFS and Hadoop Accelerator Discontinuation [2].
Probably should be moved to the next release due to dependency on
Tensorflow. Need to check. (Andrey Gura)
- IGNITE-9489 CorruptedTreeException on index create [3]. Check all
the issues releated to this. Some of them already fixed by GridGain.
Need to check. (Andrey Gura)
- IGNITE-12181 Rebalance hangs on BLT change. The cause has been
found. Will be fixed. (Maxim Muzafarov)
- Need to check all the other blocker issues (Maxim Muzafarov)

5.
QA regression (2.7 -> 2.8). I'll provide additional details when I'll get them.
Review and check test-cases, optioannly schedule meeting (Maxim Muzafarov)

6.
(optional) Need to create an INFRA ticket to add `Epic` JIRA issue
type to the Apache Ignite JIRA. The issue [6] has been created but not
sure that I have the right permission to do so.

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/The-Spark-2-4-support-td43777.html
[2] https://issues.apache.org/jira/browse/IGNITE-11942
[3] https://issues.apache.org/jira/browse/IGNITE-9489
[4] https://issues.apache.org/jira/browse/IGNITE-12054
[5] https://issues.apache.org/jira/browse/IGNITE-12181
[6] https://issues.apache.org/jira/browse/INFRA-19164

On Mon, 30 Sep 2019 at 12:14, Ivan Pavlukhin  wrote:
>
> Maxim, Folks,
>
> Could you please share a results of the Slack discussion from Sep 25?
>
> ср, 25 сент. 2019 г. в 15:50, Dmitriy Pavlov :
> >
> > Hi Maxim,
> >
> > Thank you for preparing the release page!
> >
> > Could you please add Require release notes filter? You can find an example
> > in  https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7.6
> >
> > Sincerely
> > Dmitriy Pavlov
> >
> > ср, 25 сент. 2019 г. в 11:58, Maxim Muzafarov :
> >
> > > Igniters,
> > >
> > >
> > > It's true that we are still discussing the release dates. But
> > > nevertheless, all the release blockers are important since some of
> > > them may require more than one month to be fixed. Let's discuss today
> > > how we will handle these issues and track Monitoring and ML major
> > > features to get them into the next release.
> > >
> > > The meeting already scheduled. We will use the ASF Slack on September
> > > 25-th, 17-00 (MSK).
> > > I've created the channel [2] #ignite-release-2_8 please, join.
> > > (Discussion will be on Russin language).
> > >
> > >
> > >
> > > Please, also note that I've created the 2.8 release confluence page
> > > [1] with additional information. I will review all the issues we have
> > > and will move some of them to 2.9.
> > > But currently, we've had pinned to 2.8:
> > >
> > > - 604 open issues
> > > - 57 in progress issues
> > > - 34 patch available issues
> > >
> > > - 17 issues marked as the release blockers
> > > - 4 of release blocker issues are unassigned
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8
> > > [2] https://app.slack.com/client/T4S1WH2J3/CNQ51M4FQ
> > >
> > > On Wed, 25 Sep 2019 at 10:21, Dmitriy Pavlov  wrote:
> > > >
> > > > Hi Igniters,
> > > >
> > > > I suppose discussion is still at phase 0-Initializing
> > > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process
> > > >
> > > > So it is probably no reason to discuss particular blockers. It would 
> > > > make
> > > > sense when the process of removal irrelevant tickets starts (phase 1.2)
> > > and
> > > > till phase 4-Release candidate building.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > вт, 24 сент. 2019 г. в 18:53, Anton Kalashnikov :
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > I want to notice one more blocker for release [1]. This bug can lead 
> > > > > to
> > > > > some incorrect baseline default enabled flag calculation(more details
> > > in
> > > > > the ticket).
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12227
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Anton Kalashnikov
> > > > >
> > > > >
> > > > > 24.09.2019, 17:01, "Andrey Gura" :
> > > > > > Sergey,
> > > > > >
> > 

Re: Ignite community is building Calcite-based prototype

[Julian Feinauer]

Hi Igor,

I agree that it should be rather similar to what Drill did as distributed 
computing also is a big concern for Ignite, I guess, right?

Julian

Am 01.10.19, 15:06 schrieb "Seliverstov Igor" :

Guys,

The better link: 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
 


Almost everything you may see by the link is the same as Drill guys already 
did, the difference is in details but the idea is the same.

Of course we’ll face many issues while development and I'll appreciate if 
some of you assist us.

Regards,
Igor

> 1 окт. 2019 г., в 12:32, Julian Feinauer  
написал(а):
> 
> Hi Denis,
> 
> Nice to hear from you and the ignite team... that sounds like an 
excellent idea. I liked the idea of Ignite since I heard about it (I think when 
it became TLP back then). So I would be happy to help you if you have specific 
questions... I‘m currently working on a related topic, namely integrate calcite 
as SQL Layer into Apache IoTDB .
> 
> Best
> Julian
> 
> Holen Sie sich Outlook für iOS
> 
> Von: Denis Magda 
> Gesendet: Tuesday, October 1, 2019 2:37:20 AM
> An: d...@calcite.apache.org ; dev 

> Betreff: Ignite community is building Calcite-based prototype
> 
> Hey ASF-mates,
> 
> Just wanted to send a note for Ignite dev community who has started
> prototyping
> 

> with a new Ignite SQL engine and Calcite was selected as the most 
favorable
> option.
> 
> We will truly appreciate if you help us with questions that might hit your
> dev list. Ignite folks have already studied Calcite well enough and 
carried
> on with the integration, but there might be tricky parts that would 
require
> your expertise.
> 
> Btw, if anybody is interested in Ignite (memory-centric database and
> compute platform) or would like to learn more details about the prototype
> or join its development, please check these links or send us a note:
> 
>   - https://ignite.apache.org
>   -
>   
https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
> 
> 
> -
> Denis,
> Ignite PMC Chair

Re: Ignite community is building Calcite-based prototype

[Seliverstov Igor]

Guys,

The better link: 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine
 


Almost everything you may see by the link is the same as Drill guys already 
did, the difference is in details but the idea is the same.

Of course we’ll face many issues while development and I'll appreciate if some 
of you assist us.

Regards,
Igor

> 1 окт. 2019 г., в 12:32, Julian Feinauer  
> написал(а):
> 
> Hi Denis,
> 
> Nice to hear from you and the ignite team... that sounds like an excellent 
> idea. I liked the idea of Ignite since I heard about it (I think when it 
> became TLP back then). So I would be happy to help you if you have specific 
> questions... I‘m currently working on a related topic, namely integrate 
> calcite as SQL Layer into Apache IoTDB .
> 
> Best
> Julian
> 
> Holen Sie sich Outlook für iOS
> 
> Von: Denis Magda 
> Gesendet: Tuesday, October 1, 2019 2:37:20 AM
> An: d...@calcite.apache.org ; dev 
> 
> Betreff: Ignite community is building Calcite-based prototype
> 
> Hey ASF-mates,
> 
> Just wanted to send a note for Ignite dev community who has started
> prototyping
> 
> with a new Ignite SQL engine and Calcite was selected as the most favorable
> option.
> 
> We will truly appreciate if you help us with questions that might hit your
> dev list. Ignite folks have already studied Calcite well enough and carried
> on with the integration, but there might be tricky parts that would require
> your expertise.
> 
> Btw, if anybody is interested in Ignite (memory-centric database and
> compute platform) or would like to learn more details about the prototype
> or join its development, please check these links or send us a note:
> 
>   - https://ignite.apache.org
>   -
>   
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
> 
> 
> -
> Denis,
> Ignite PMC Chair

Re: Improvements for new security approach.

[Denis Garus]

There is the #ignite-security Slack channel; we can use it for discussion.

вт, 1 окт. 2019 г. в 09:14, Anton Vinogradov :

> Folks,
>
> Could you please create a slack channel to discuss this in an effective
> way?
>
> On Mon, Sep 30, 2019 at 5:36 PM Denis Garus  wrote:
>
> > >>As a result, you can't load security on demand.
> >
> > Why?
> > What is the difference between sending SecurityContext with every job's
> > request and sending SecurityContext once when remote node demand it?
> >
> > пн, 30 сент. 2019 г. в 17:25, Maksim Stepachev <
> maksim.stepac...@gmail.com
> > >:
> >
> > > I suppose that code works only with requests are made from
> > > GridRestProcessor (It isn't a client, I call it like a fake client).
> As a
> > > result, you can't load security on demand. If you want to do it, you
> > > should transmit HTTP session and backward address of a node which
> > > received REST request.
> > >
> > > пн, 30 сент. 2019 г. в 16:16, Denis Garus :
> > >
> > >>  Subject's size is unlimited, that can lead to a dramatic increase
> > in
> > >> traffic between nodes.
> > >> >>I added network optimization for this case. I add a subject in the
> > case
> > >> when ctx.discovery().node(secSubjId) == null.
> > >>
> > >> Yes, this optimization is good, but we have to send SecurityContext
> > >> whenever a client starts a job.
> > >> Why?
> > >>
> > >> A better solution would be to send SecurityContext on demand.
> > >>
> > >> Imagine the following scenario.
> > >> A client connects to node A and starts a job that runs on remote node
> B.
> > >> IgniteSecurityProcessor on node B tries to find SecurityContext by
> > >> subjectId.
> > >> If IgniteSecurityProcessor fails, then it sends SecurityContextRequest
> > to
> > >> node A and gets required SecurityContext
> > >> that afterward puts to the IgniteSecurityProcessor's cache on node B.
> > >> The next request to run a job on node B with this subjectId doesn't
> > >> require SecurityContext transmission.
> > >>
> > >> SecurityContextResponse can contain some additional information, for
> > >> example,
> > >> time-to-live of SecurityContext before eviction SecurityContext from
> the
> > >> IgniteSecurityProcessor's cache.
> > >>
> > >> пт, 27 сент. 2019 г. в 15:20, Maksim Stepachev <
> > >> maksim.stepac...@gmail.com>:
> > >>
> > >>> I finished with fixes:
> > >>> https://issues.apache.org/jira/browse/IGNITE-11992
> > >>>
> > >>> >> Subject's size is unlimited, that can lead to a dramatic increase
> in
> > >>> traffic between nodes.
> > >>> I added network optimization for this case. I add a subject in the
> case
> > >>> when ctx.discovery().node(secSubjId) == null.
> > >>>
> > >>> >> Also, we need to get rid of GridTaskThreadContextKey#TC_SUBJ_ID as
> > >>> duplication of IgnitSecurity responsibility.
> > >>> [2]Yes, we should rid of this. But in the next task, because I can't
> > >>> merge it since 18 Jul 19:)
> > >>>
> > >>> [1] I aggry with you.
> > >>>
> > >>>
> > >>> пт, 27 сент. 2019 г. в 11:42, Denis Garus :
> > >>>
> >  Hello, Maksim!
> > 
> >  Thank you for your effort and interest in the security of Ignite.
> > 
> >  I would like you to pay attention to the discussion [1] and issue
> [2].
> >  It looks like not only task should execute in the current security
> >  context but all operations too, that is essential to determine a
> > security
> >  id for events.
> >  Also, we need to get rid of GridTaskThreadContextKey#TC_SUBJ_ID as
> >  duplication of IgnitSecurity responsibility.
> >  I think your task is the right place to do that.
> >  What is your opinion?
> > 
> >  >>It's the reason why subject id isn't enough and we should transmit
> >  subject inside message for this case.
> >  There is a problem with this approach.
> >  Subject's size is unlimited, that can lead to a dramatic increase in
> >  traffic between nodes.
> > 
> >  1.
> > 
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/JavaDoc-for-Event-s-subjectId-methods-td43663.html
> >  2. https://issues.apache.org/jira/browse/IGNITE-9914
> > 
> >  пт, 27 сент. 2019 г. в 08:38, Anton Vinogradov :
> > 
> > > Maksim
> > >
> > > >> I want to fix 2-3-4 points under one ticket.
> > > Please let me know once it's become ready to be reviewed.
> > >
> > > On Thu, Sep 26, 2019 at 5:18 PM Maksim Stepachev <
> > > maksim.stepac...@gmail.com>
> > > wrote:
> > >
> > > > Hi.
> > > >
> > > > Anton Vinogradov,
> > > >
> > > > I want to fix 2-3-4 points under one ticket.
> > > >
> > > > The first was fixed in the ticket:
> > > > https://issues.apache.org/jira/browse/IGNITE-11094
> > > > Also, I aggry with you that 5-6 isn't required to ignite.
> > > >
> > > > Denis Garus,
> > > > I made reproducer for point 3. Looks at the test from my
> > > pull-request:
> > > > JettyRestPropagationSecurityContextTest
> > > 

Re: Unassigned issue IGNITE-12174

[Ivan Pavlukhin]

Hi Jan,

You are right. I remember that we discussed to do similar for TEXT queries [1].

[1] https://issues.apache.org/jira/browse/IGNITE-12190

вт, 1 окт. 2019 г. в 12:07, :
>
> Thank you for the reply.
>
> Quoting Alexei Scherbakov :
> > Lucene and spatial indexes do not support persistence.
> > This is known limitation which would probably be addressed in the future
> > releases.
>
> I think this limitation should at least be mentioned in the
> documentation then:
>
> https://apacheignite-sql.readme.io/docs/geospatial-support
>
> It would be even better to handle this unsupported use case in the
> code as well instead of throwing a NullPointerException. I can add
> this check as improvement to the Ignite code if this is wanted and
> agreed. But I need a hint on what would be the right place to add this
> check in the code base.
>
> Thanks,
>
> Jan
>


-- 
Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-12250) Adds REST requests processing logging.

[PetrovMikhail (Jira)]

PetrovMikhail created IGNITE-12250:
--

 Summary: Adds REST requests processing logging.
 Key: IGNITE-12250
 URL: https://issues.apache.org/jira/browse/IGNITE-12250
 Project: Ignite
  Issue Type: Improvement
Reporter: PetrovMikhail
Assignee: PetrovMikhail


It's needed to add logging of each REST request processing result.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Unassigned issue IGNITE-12174

[j]

Thank you for the reply.

Quoting Alexei Scherbakov :

Lucene and spatial indexes do not support persistence.
This is known limitation which would probably be addressed in the future
releases.


I think this limitation should at least be mentioned in the  
documentation then:


https://apacheignite-sql.readme.io/docs/geospatial-support

It would be even better to handle this unsupported use case in the  
code as well instead of throwing a NullPointerException. I can add  
this check as improvement to the Ignite code if this is wanted and  
agreed. But I need a hint on what would be the right place to add this  
check in the code base.


Thanks,

Jan

Re: New SQL execution engine

[Seliverstov Igor]

Nikolay,

The document you edited is wrong (and outdated).

Since the author meant another idea, I decided not to change IEP-35 and
create a new one - IEP-37 (https://cwiki.apache.org/confluence/x/NBLABw).
It's already have a number of key requirements.

Regards,
Igor

вт, 1 окт. 2019 г., 6:14 Nikolay Izhikov :

> Hello, Igniters.
>
> I extends IEP [1] with the tickets caused by H2 limitations.
>
> Please, let's write down requirements for engine in the IEP.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-33%3A+New+SQL+executor+engine+infrastructure
>
> В Пн, 30/09/2019 в 17:20 -0700, Denis Magda пишет:
> > Ivan, we need more of these discussions, totally agree with you ;)
> >
> > I've updated the Motivation paragraph outlining some high-level users we
> > see by working with our users. Hope it helps. Let's carry on and let me
> > send a note to Apache Calcite community.
> >
> > -
> > Denis
> >
> >
> > On Mon, Sep 30, 2019 at 1:56 AM Ivan Pavlukhin 
> wrote:
> >
> > > Folks,
> > >
> > > Thanks everyone for a hot discussion! Not every open source community
> > > has such open and boiling discussions. It means that people here
> > > really do care. And I am proud of it!
> > >
> > > As I understood, nobody is strictly against the proposed initiative.
> > > And I am glad that we can move forward (with some steps back along the
> > > way).
> > >
> > > пт, 27 сент. 2019 г. в 19:29, Nikolay Izhikov :
> > > >
> > > > Hello, Denis.
> > > >
> > > > Thanks for the clarifications.
> > > >
> > > > Sounds good for me.
> > > > All I try to say in this thread:
> > > > Guys, please, let's take a step back and write down
> requirements(what we
> > >
> > > want to get with SQL engine).
> > > > Which features and use-cases are primary for us.
> > > >
> > > > I'm sure you have done it, already during your research.
> > > >
> > > > Please, share it with the community.
> > > >
> > > > I'm pretty sure we would back to this document again and again during
> > >
> > > migration.
> > > > So good written design is worth it.
> > > >
> > > > В Пт, 27/09/2019 в 09:10 -0700, Denis Magda пишет:
> > > > > Ignite mates, let me try to move the discussion in a constructive
> way.
> > >
> > > It
> > > > > looks like we set a wrong context from the very beginning.
> > > > >
> > > > > Before proposing this idea to the community, some of us were
> > > > > discussing/researching the topic in different groups (the one need
> to
> > >
> > > think
> > > > > it through first before even suggesting to consider changes of this
> > > > > magnitude). The day has come to share this idea with the whole
> > >
> > > community
> > > > > and outline the next actions. But (!) nobody is 100% sure that
> that's
> > >
> > > the
> > > > > right decision. Thus, this will be an *experiment*, some of our
> > >
> > > community
> > > > > members will be developing a *prototype* and only based on the
> > >
> > > prototype
> > > > > outcomes we shall make a final decision. Igor, Roman, Ivan, Andrey,
> > >
> > > hope
> > > > > that nothing has changed and we're on the same page here.
> > > > >
> > > > > Many technical and architectural reasons that justify this project
> have
> > > > > been shared but let me throw in my perspective. There is nothing
> wrong
> > >
> > > with
> > > > > H2, that was the right choice for that time.  Thanks to H2 and
> Ignite
> > >
> > > SQL
> > > > > APIs, our project is used across hundreds of deployments who are
> > > > > accelerating relational databases or use Ignite as a system of
> records.
> > > > > However, these days many more companies are migrating to
> *distributed*
> > > > > databases that speak SQL. For instance, if a couple of years ago 1
> out
> > >
> > > of
> > > > > 10 use cases needed support for multi-joins queries or queries with
> > > > > subselects or efficient memory usage then today there are 5 out of
> 10
> > >
> > > use
> > > > > cases of this kind; in the foreseeable future, it will be a 10 out
> of
> > >
> > > 10.
> > > > > So, the evolution is in progress -- the relational world goes
> > >
> > > distributed,
> > > > > it became exhaustive for both Ignite SQL maintainers and experts
> who
> > >
> > > help
> > > > > to tune it for production usage to keep pace with the evolution
> mostly
> > >
> > > due
> > > > > to the H2-dependency. Thus, Ignite SQL has to evolve and has to be
> > >
> > > ready to
> > > > > face the future reality.
> > > > >
> > > > > Luckily, we don't need to rush and don't have the right to rush
> because
> > > > > hundreds existing users have already trusted their production
> > >
> > > environments
> > > > > to Ignite SQL and we need to roll out changes with such a big
> impact
> > > > > carefully. So, I'm excited that Roman, Igor, Ivan, Andrey stepped
> in
> > >
> > > and
> > > > > agreed to be the first contributors who will be *experimenting*
> with
> > >
> > > the
> > > > > new SQL engine. Let's support them; let's connect them with Apache
> > >
> > > Calcite
> > > > > 

Re: Improvements for new security approach.

[Anton Vinogradov]

Folks,

Could you please create a slack channel to discuss this in an effective way?

On Mon, Sep 30, 2019 at 5:36 PM Denis Garus  wrote:

> >>As a result, you can't load security on demand.
>
> Why?
> What is the difference between sending SecurityContext with every job's
> request and sending SecurityContext once when remote node demand it?
>
> пн, 30 сент. 2019 г. в 17:25, Maksim Stepachev  >:
>
> > I suppose that code works only with requests are made from
> > GridRestProcessor (It isn't a client, I call it like a fake client). As a
> > result, you can't load security on demand. If you want to do it, you
> > should transmit HTTP session and backward address of a node which
> > received REST request.
> >
> > пн, 30 сент. 2019 г. в 16:16, Denis Garus :
> >
> >>  Subject's size is unlimited, that can lead to a dramatic increase
> in
> >> traffic between nodes.
> >> >>I added network optimization for this case. I add a subject in the
> case
> >> when ctx.discovery().node(secSubjId) == null.
> >>
> >> Yes, this optimization is good, but we have to send SecurityContext
> >> whenever a client starts a job.
> >> Why?
> >>
> >> A better solution would be to send SecurityContext on demand.
> >>
> >> Imagine the following scenario.
> >> A client connects to node A and starts a job that runs on remote node B.
> >> IgniteSecurityProcessor on node B tries to find SecurityContext by
> >> subjectId.
> >> If IgniteSecurityProcessor fails, then it sends SecurityContextRequest
> to
> >> node A and gets required SecurityContext
> >> that afterward puts to the IgniteSecurityProcessor's cache on node B.
> >> The next request to run a job on node B with this subjectId doesn't
> >> require SecurityContext transmission.
> >>
> >> SecurityContextResponse can contain some additional information, for
> >> example,
> >> time-to-live of SecurityContext before eviction SecurityContext from the
> >> IgniteSecurityProcessor's cache.
> >>
> >> пт, 27 сент. 2019 г. в 15:20, Maksim Stepachev <
> >> maksim.stepac...@gmail.com>:
> >>
> >>> I finished with fixes:
> >>> https://issues.apache.org/jira/browse/IGNITE-11992
> >>>
> >>> >> Subject's size is unlimited, that can lead to a dramatic increase in
> >>> traffic between nodes.
> >>> I added network optimization for this case. I add a subject in the case
> >>> when ctx.discovery().node(secSubjId) == null.
> >>>
> >>> >> Also, we need to get rid of GridTaskThreadContextKey#TC_SUBJ_ID as
> >>> duplication of IgnitSecurity responsibility.
> >>> [2]Yes, we should rid of this. But in the next task, because I can't
> >>> merge it since 18 Jul 19:)
> >>>
> >>> [1] I aggry with you.
> >>>
> >>>
> >>> пт, 27 сент. 2019 г. в 11:42, Denis Garus :
> >>>
>  Hello, Maksim!
> 
>  Thank you for your effort and interest in the security of Ignite.
> 
>  I would like you to pay attention to the discussion [1] and issue [2].
>  It looks like not only task should execute in the current security
>  context but all operations too, that is essential to determine a
> security
>  id for events.
>  Also, we need to get rid of GridTaskThreadContextKey#TC_SUBJ_ID as
>  duplication of IgnitSecurity responsibility.
>  I think your task is the right place to do that.
>  What is your opinion?
> 
>  >>It's the reason why subject id isn't enough and we should transmit
>  subject inside message for this case.
>  There is a problem with this approach.
>  Subject's size is unlimited, that can lead to a dramatic increase in
>  traffic between nodes.
> 
>  1.
> 
> http://apache-ignite-developers.2346864.n4.nabble.com/JavaDoc-for-Event-s-subjectId-methods-td43663.html
>  2. https://issues.apache.org/jira/browse/IGNITE-9914
> 
>  пт, 27 сент. 2019 г. в 08:38, Anton Vinogradov :
> 
> > Maksim
> >
> > >> I want to fix 2-3-4 points under one ticket.
> > Please let me know once it's become ready to be reviewed.
> >
> > On Thu, Sep 26, 2019 at 5:18 PM Maksim Stepachev <
> > maksim.stepac...@gmail.com>
> > wrote:
> >
> > > Hi.
> > >
> > > Anton Vinogradov,
> > >
> > > I want to fix 2-3-4 points under one ticket.
> > >
> > > The first was fixed in the ticket:
> > > https://issues.apache.org/jira/browse/IGNITE-11094
> > > Also, I aggry with you that 5-6 isn't required to ignite.
> > >
> > > Denis Garus,
> > > I made reproducer for point 3. Looks at the test from my
> > pull-request:
> > > JettyRestPropagationSecurityContextTest
> > >
> > > https://github.com/apache/ignite/pull/6918
> > >
> > > For point 2 you should apply GridRestProcessor from pr and set
> debug
> > into
> > > VisorQueryUtils#scheduleQueryStart between
> > > ignite.context().closure().runLocalSafe  and call:
> > > ignite.context().security().securityContext()
> > >
> > >
> > > For point 3, do action above and call:
> > >
> >