Re: Maven. Issues with flatten plugin

[Petr Ivanov]

The problem is solved by updating flatten-maven-plugin version to 1.0.1.

Nikolay, please, double check it.
If it really solves the problem, please, fill the ticket (or point to existing 
one), so I can update it and check impact on release procedure.



> On 1 Mar 2018, at 17:04, Nikolay Izhikov  wrote:
> 
> Petr.
> 
> Thank you for trying!
> 
> Did you remove 'test' dependencies before running commands?
> 
> Because I commit in master only correct pom.xml for current build process,
> of course.
> 
> But, to make things works I have to copy paste transitive dependencies from
> spark-core.
> 
> 1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" 
> написал:
> 
>> 
>>> 
>>> I don't get what is the point.
>>> Did you try to reproduce issue?
>>> Or should I provide full traces to you?
>>> 
>> 
>> My point in inability to fully understand and describe the problem in
>> terms of mechanism which causes it.
>> For now I can see only indirect guessing.
>> 
>> 
>> And yes, I’ve run your commands and both times (in spark module directory
>> and in root) it produces the same result:
>> 
>> 1.
>> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn
>> install -U -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
>> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
>> [WARNING] The requested profile "ml" could not be activated because it
>> does not exist.
>> 
>> 2.
>> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v
>> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
>> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
>> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
>> [WARNING] The requested profile "lgpl" could not be activated because it
>> does not exist.
>> [WARNING] The requested profile "examples" could not be activated because
>> it does not exist.
>> [WARNING] The requested profile "scala" could not be activated because it
>> does not exist.
>> [WARNING] The requested profile "ml" could not be activated because it
>> does not exist.

[jira] [Created] (IGNITE-7861) Web console: invalid column on Activity details modal

[Vasiliy Sisko (JIRA)]

Vasiliy Sisko created IGNITE-7861:
-

 Summary: Web console: invalid column on Activity details modal
 Key: IGNITE-7861
 URL: https://issues.apache.org/jira/browse/IGNITE-7861
 Project: Ignite
  Issue Type: Bug
Reporter: Vasiliy Sisko
Assignee: Vasiliy Sisko
 Attachments: Selection_094.png

Many items do not have right description



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Denis Magda]

Dmitriy R., Nilokay,

Thanks for the analysis and handout of the architectural design. No doubts,
it would be a valuable addition to Ignite.

I would encourage you creating an IEP on the wiki and break the work into
pieces discussing specific part with the community.

--
Denis


On Thu, Mar 1, 2018 at 9:29 PM, Nikolay Izhikov  wrote:

> Hello, Dmitriy.
>
> Thank you for feedback!
>
> > Will it be supported?
>
> Yes.
>
> TDE shouldn't broke any of existing Ignite features.
> It adds some encrypt/decrypt level when we writing and reading pages
> in/from PDS.
>
> В Пт, 02/03/2018 в 07:29 +0300, Dmitriy Setrakyan пишет:
> > I have looked at the design, but could not find anything about running
> SQL
> > queries against the encrypted data. Will it be supported?
> >
> > D.
> >
> > On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov 
> wrote:
> >
> > > Hell, Dima!
> > >
> > > Thank you for document!
> > >
> > > I'm ready to implement this feature with you.
> > >
> > > Igniters, please, share you thoughts about proposed design
> > >
> > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> > >
> > > В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> > > > Hello, Igniters!
> > > >
> > > > I investigated the issue and wrote some details in a draft document
> > > > [1]. I think we should made IEP for TDE because it is a big change
> and
> > > > should be described in a single place, but not in a message
> > > > conversation.
> > > > Please, look it and write your thoughts. What is not understandable,
> > > > what should be detailed or described?
> > > >
> > > > > Where are we going to store keys (MEK) physically? Would it be
> PKCS#11
> > > > > storage? Where we will store passwords to unlock storage or it
> will be
> > > > > responibilty of user?
> > > >
> > > > I think we should provide interface for MEK storage to let users use
> > > > storages they want. I suppose at the first step we should provide
> very
> > > > simple implementation, which will store MEK on every node and MEK
> will
> > > > be extracted by administrator during cluster activation process. Once
> > > > MEK is extracted from key store, we decrypt CEKs and destroy open
> MEK,
> > > > leaving open only cache keys.
> > > >
> > > > I think external storage is user's worry and we shouldn't give users
> > > > built-in external storage like Oracle Wallet or Microsoft Azure Key
> > > > Vault because it will increase Ignite's complexity too much.
> > > >
> > > > And yes, we should to comply with the standards like PKCS#11.
> > > >
> > > > > One more thing is how "node gets MEK from coordinator", if we send
> > > > > cleartext MEK, such security becomes useless also.
> > > >
> > > > Yeah, that's why we should use secured connection. As I know, we have
> > > > SSL implementation over JDK implementation, am I right? But we must
> > > > ensure to use latest SSL/TLS version.
> > > >
> > > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
>

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Nikolay Izhikov]

> Got it, thanks! In this case this sounds very useful. Do we understand the 
> performance impact?

I don't think we fully understand it.
It's a question of additional research and benchmarking.

So preliminary conclusions, based on common sense and my experiense of usage 
Ignite in production,  are following:

1. CPU is not a bottleneck in most of Ignite use-cases. 
Especially when we use it as a in-memory database.

2. TDE mostly will increase CPU usage: SSL + pages encryption/decryption.

В Пт, 02/03/2018 в 08:33 +0300, Dmitriy Setrakyan пишет:
> On Fri, Mar 2, 2018 at 8:29 AM, Nikolay Izhikov  wrote:
> 
> > Hello, Dmitriy.
> > 
> > Thank you for feedback!
> > 
> > > Will it be supported?
> > 
> > Yes.
> > 
> > TDE shouldn't broke any of existing Ignite features.
> > It adds some encrypt/decrypt level when we writing and reading pages
> > in/from PDS.
> > 
> 
> Got it, thanks! In this case this sounds very useful. Do we understand the
> performance impact?

signature.asc
Description: This is a digitally signed message part

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Dmitriy Setrakyan]

On Fri, Mar 2, 2018 at 8:29 AM, Nikolay Izhikov  wrote:

> Hello, Dmitriy.
>
> Thank you for feedback!
>
> > Will it be supported?
>
> Yes.
>
> TDE shouldn't broke any of existing Ignite features.
> It adds some encrypt/decrypt level when we writing and reading pages
> in/from PDS.
>

Got it, thanks! In this case this sounds very useful. Do we understand the
performance impact?

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Nikolay Izhikov]

Hello, Dmitriy.

Thank you for feedback!

> Will it be supported?

Yes. 

TDE shouldn't broke any of existing Ignite features.
It adds some encrypt/decrypt level when we writing and reading pages in/from 
PDS.

В Пт, 02/03/2018 в 07:29 +0300, Dmitriy Setrakyan пишет:
> I have looked at the design, but could not find anything about running SQL
> queries against the encrypted data. Will it be supported?
> 
> D.
> 
> On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov  wrote:
> 
> > Hell, Dima!
> > 
> > Thank you for document!
> > 
> > I'm ready to implement this feature with you.
> > 
> > Igniters, please, share you thoughts about proposed design
> > 
> > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
> > 
> > В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> > > Hello, Igniters!
> > > 
> > > I investigated the issue and wrote some details in a draft document
> > > [1]. I think we should made IEP for TDE because it is a big change and
> > > should be described in a single place, but not in a message
> > > conversation.
> > > Please, look it and write your thoughts. What is not understandable,
> > > what should be detailed or described?
> > > 
> > > > Where are we going to store keys (MEK) physically? Would it be PKCS#11
> > > > storage? Where we will store passwords to unlock storage or it will be
> > > > responibilty of user?
> > > 
> > > I think we should provide interface for MEK storage to let users use
> > > storages they want. I suppose at the first step we should provide very
> > > simple implementation, which will store MEK on every node and MEK will
> > > be extracted by administrator during cluster activation process. Once
> > > MEK is extracted from key store, we decrypt CEKs and destroy open MEK,
> > > leaving open only cache keys.
> > > 
> > > I think external storage is user's worry and we shouldn't give users
> > > built-in external storage like Oracle Wallet or Microsoft Azure Key
> > > Vault because it will increase Ignite's complexity too much.
> > > 
> > > And yes, we should to comply with the standards like PKCS#11.
> > > 
> > > > One more thing is how "node gets MEK from coordinator", if we send
> > > > cleartext MEK, such security becomes useless also.
> > > 
> > > Yeah, that's why we should use secured connection. As I know, we have
> > > SSL implementation over JDK implementation, am I right? But we must
> > > ensure to use latest SSL/TLS version.
> > > 
> > > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf

signature.asc
Description: This is a digitally signed message part

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Dmitriy Setrakyan]

I have looked at the design, but could not find anything about running SQL
queries against the encrypted data. Will it be supported?

D.

On Thu, Mar 1, 2018 at 8:05 PM, Nikolay Izhikov  wrote:

> Hell, Dima!
>
> Thank you for document!
>
> I'm ready to implement this feature with you.
>
> Igniters, please, share you thoughts about proposed design
>
> [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
>
> В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> > Hello, Igniters!
> >
> > I investigated the issue and wrote some details in a draft document
> > [1]. I think we should made IEP for TDE because it is a big change and
> > should be described in a single place, but not in a message
> > conversation.
> > Please, look it and write your thoughts. What is not understandable,
> > what should be detailed or described?
> >
> > > Where are we going to store keys (MEK) physically? Would it be PKCS#11
> > > storage? Where we will store passwords to unlock storage or it will be
> > > responibilty of user?
> >
> > I think we should provide interface for MEK storage to let users use
> > storages they want. I suppose at the first step we should provide very
> > simple implementation, which will store MEK on every node and MEK will
> > be extracted by administrator during cluster activation process. Once
> > MEK is extracted from key store, we decrypt CEKs and destroy open MEK,
> > leaving open only cache keys.
> >
> > I think external storage is user's worry and we shouldn't give users
> > built-in external storage like Oracle Wallet or Microsoft Azure Key
> > Vault because it will increase Ignite's complexity too much.
> >
> > And yes, we should to comply with the standards like PKCS#11.
> >
> > > One more thing is how "node gets MEK from coordinator", if we send
> > > cleartext MEK, such security becomes useless also.
> >
> > Yeah, that's why we should use secured connection. As I know, we have
> > SSL implementation over JDK implementation, am I right? But we must
> > ensure to use latest SSL/TLS version.
> >
> > [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf
>

Enabling swap space and Ignite Persistence

[Prachi Garg]

Engineers,

How does persistence and swap work when both are enabled? I was under the
impression that for a data region you can either have swap or persistence
configured at a time, but not both. Please clarify.

Thanks,
-Prachi

Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite

[techbysample]

Yury,

I have reviewed and removed unnecessary pom files.

In addition, I have updated my branch from master.

Thanks.

Best,
Turik




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts

[Valentin Kulichenko]

+1. Low level timeouts that we still have in discovery and communication
are very hard to explain and I doubt there is anyone who fully understands
how they currently work. They bring a lot of complexity and almost zero
value. Let's deprecate them and leave only failureDetectionTimeout plus
other high level settings that Alexey mentioned.

-Val

On Thu, Mar 1, 2018 at 6:06 AM, Ilya Kasnacheev 
wrote:

> I agree with you.
>
> I think we could restrict usage of e.g. setConnectTimeout/setSocketTimeout
> to people extending SPIs, since different implementations may need
> different values.
>
> However, for user configurations we should only expose timeouts we can
> explain, everything else should have reasonable values.
>
> --
> Ilya Kasnacheev
>
> 2018-03-01 17:01 GMT+03:00 Alexey Popov :
>
> > Hi Igniters,
> >
> > We often see similar questions from users and customers related to
> > IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts and
> > their
> > relations. And we see several side-effects after incorrect timeout
> > configuration.
> >
> > I tried to briefly describe these timeout settings (please see below) and
> > found out that the most of them do not have sense in terms of cluster
> > functions/operations and could not be explained to the users.
> >
> > I propose to deprecate most of them and leave only the timeouts we can
> > explain in common terms ( (setFailureDetectionTimeout, setNetworkTimeout,
> > setJoinTimeout and some others).
> >
> > Please let me know your thoughts.
> >
> > Thanks,
> > Alexey
> >
> > GLOBAL:
> >
> > IgniteConfiguration.setNetworkTimeout:
> > It is a global timeout for high-level operations where a network is
> > involved. For instance, IgniteMessaging delivery uses this timeout or
> > DiscoverySpi handshake.
> >
> > IgniteConfiguration.setFailureDetectionTimeout:
> > It is a global timeout for detecting failures at IgniteSpi
> implementations
> > (including DiscoverySpi and CommunicationSpi).
> > The failure detection algorithm actually limits a range of simple network
> > operations related to a single logical operation (for instance, a
> reliable
> > delivery of some DiscoverySpi message within a cluster).
> > Failure detection timeout is a cumulative timeout for a socket
> connection,
> > sending and receiving data bytes and all possible socket retries (if some
> > failure happens).
> > This timeout is intended to simplify the failure detection condition
> from a
> > user perspective.
> >
> > IgniteConfiguration.setClientFailureDetectionTimeout: - it is a special
> > case
> > for DiscoverySpi client-node Ignite.
> >
> > TCP DISCOVERY SPI:
> >
> > If you need more control over failure detection algorithm for
> > TcpDiscoverySpi you can explicitly use the following low-level options
> > (that
> > will disable failureDetectoinTimeout logic):
> >
> > 1. TcpDiscoverySpi.setConnectTimeout - socket connection timeout
> > 2. TcpDiscoverySpi.setReconnectCount - number of reconnect attempts used
> > when establishing connection with the remote node and sending messages to
> > it
> > 3. TcpDiscoverySpi.setSocketTimeout - socket write timeout. The write
> > operation will be repeated getReconnectCount() times if it exceeds this
> > timeout
> > 4. TcpDiscoverySpi.setAckTimeout - message acknowledgment timeout. If a
> > message acknowledgment is not received within this timeout, sending is
> > considered as failed and SPI will try to repeat send operation. It is
> > automatically doubled for simultaneous retries up to getMaxAckTimeout
> > value.
> > 5. TcpDiscoverySpi.setMaxAckTimeout - maximum connection timeout, if the
> > getAckTimeout reaches getMaxAckTimeout then SPI give up sending retries
> >
> > Another important TcpDiscoverySpi timeouts:
> >
> > TcpDiscoverySpi.setJoinTimeout - It is a timeout for join process when a
> > new/restarted node joins a cluster. The node tries to connect to all
> > available IP addresses provided by ipFinder within this timeout.
> > If the timeout is exceeded, the node will give up and throw an exception
> > from Ignition.start().
> >
> > TcpDiscoverySpi.setNetworkTimeout - timeout for high-level operations
> like
> > handshake. It looks like it should be deprecated and the
> > IgniteConfiguration.getNetworkTimeout should be used here.
> >
> > TCP COMMUNICATION SPI:
> >
> > If you need more control over failure detection algorithm for
> > TcpCommunicationSpi you can explicitly use the following low-level
> options
> > (that will disable failureDetectoinTimeout logic):
> >
> > 1. TcpCommunicationSpi.setConnectTimeout - socket connection timeout,
> will
> > be automatically doubled for simultaneous retries (up to
> getReconnectCount)
> > related to a single logical operation
> > 2. TcpCommunicationSpi.setMaxConnectTimeout - maximum connection
> timeout,
> > the higher limit of getReconnectCount-times doubled getConnectTimeout
> > 3. TcpCommunicationSpi.setReconnectCount - number of reconnect attempts
> > used
> > when establishing co

Re: Maven. Issues with flatten plugin

[Vyacheslav Daradur]

>> Anton, Vyacheslav — are you proposing assembly process shift to package 
>> phase, yet leaving the special profile for assembly itself?

I'm talking about compile and package processes like it has been until
2.1 release.

On Thu, Mar 1, 2018 at 6:00 PM, Dmitry Pavlov  wrote:
> Hi,
>
> I think we should solve initial problem: "We have to include all transitive
> dependencies into poms". I've faced with it during Direct IO plugin
> development.
>
> I am sure it is not best practice. Because 1st level dependency version
> change will require us to update 2nd level dependencies manually.
>
> So if we can find solution without disabling plugin, we should prefer it.
> If only disable may help we should be sure release installation into
> maven-central will be successful.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 1 мар. 2018 г. в 17:23, Petr Ivanov :
>
>> Anton, Vyacheslav — are you proposing assembly process shift to package
>> phase, yet leaving the special profile for assembly itself?
>>
>>
>>
>> > On 1 Mar 2018, at 14:36, Vyacheslav Daradur  wrote:
>> >
>> > I agree with Anton, especially with the following statement:
>> >>> Ignite should be assemblied using package phase,
>> >>> not install in case you want only to assembly it.
>> >
>> > Approach with "install" phase may lead to hidden issues in a
>> > development environment.
>> >
>> > On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov 
>> wrote:
>> >>> May be it will be enough to add a profile to parent pom.xml which
>> >> activates on deploy stage only.
>> >>
>> >> I think it will be enough to solve my issue.
>> >>
>> >>> What pom.xml becomes flattened, what is removed from it and how it
>> effects
>> >> other modules of the same project?
>> >>
>> >> pom.xml that are installed to local repository is flattened.
>> >>
>> >>> Yet, I still do not get a mechanism that causes a problem.
>> >>
>> >> If you want to reproduce it, just do the following steps:
>> >>
>> >> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
>> >>
>> >> 2. run:
>> >>
>> >> `~/src/ignite/modules/spark> mvn install -U
>> >> -Plgpl,examples,scala,-clean-libs,-release,ml
>> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>> >>
>> >> It will execute OK, because maven uses pom.xml from source.
>> >>
>> >> 3. run(only root dir is changed)
>> >>
>> >> `~/src/ignite/> mvn install -U
>> >> -Plgpl,examples,scala,-clean-libs,-release,ml
>> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>> >>
>> >> Test will fail with ClassNoDefFoundError because maven will use
>> >> pom.xml from local repository which is flattened.
>> >>
>> >> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
>> >> Retry step 3. It will succeed.
>> >>
>> >>
>> >> 2018-03-01 14:18 GMT+03:00 Petr Ivanov :
>> >>
>> >>> May be it will be enough to add a profile to parent pom.xml which
>> >>> activates on deploy stage only.
>> >>>
>> >>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
>> >>> becomes flattened, what is removed from it and how it effects other
>> modules
>> >>> of the same project?
>> >>>
>> >>>
>> >>>
>>  On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
>> 
>>  Andrey, thank you.
>> 
>>  If there are no other objections I will create ticket for pom
>> structure
>>  refactoring.
>> 
>>  Anton, can you help me with it as an Ignite Release Manager?
>> 
>>  2018-03-01 13:45 GMT+03:00 Andrey Novikov :
>> 
>> > Nikolay,
>> >
>> > I think it can be removed, if parent-pom will be installed(deployed).
>> >
>> > On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov > >
>> > wrote:
>> >> Hello, Andrey.
>> >>
>> >> Thanks for an answer.
>> >>
>> >>> As I remember we use `flatten-maven-plugin` for flattening and
>> >>> removing
>> > parent relationship in deployed artifacts
>> >>
>> >> So we need it only in `release` profile?
>> >>
>> >> I found some earlier discussion about plugin [1]
>> >>
>> >>> in first versions of build, version was stored in variable in
>> parent
>> >>> pom
>> >>
>> >> Got it. Do we need this plugin now?
>> >>
>> >> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> > com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>> >>
>> >> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>> >>> Nikolay,
>> >>>
>> >>> As I remember we use `flatten-maven-plugin` for flattening and
>> >>> removing parent relationship in deployed artifacts (parent pom does
>> >>> not deploy to repository and in first versions of build, version
>> was
>> >>> stored in variable in parent pom)
>> >>>
>> >>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <
>> nizhi...@apache.org>
>> > wrote:
>>  Hello, Petr.
>> 
>> > Can you describe yo

Re: Apache Ignite 2.4 release

[Denis Magda]

Hi Vladimir,

The regression, as well as the performance, is definitely bad. Please work
with Sergey to see how much time he needs to rerun the whole test set if
you merge the changes.

I would target the end of the next week as a release date if the vote goes
well. Probably, we can run the test suites over the weekend if your changes
get to the branch.

Nikolay Izhikov, follow up with Vladimir and Sergey tomorrow. If they
decide to merge the fixes then you are free to merge your fixed for Spark
examples.

--
Denis

On Thu, Mar 1, 2018 at 7:46 AM, Vladimir Ozerov 
wrote:

> Hi,
>
> We found one nasty regression in DROP COLUMN feature and fixed it today.
> Another problem is performance - we found that recent changes to TCP
> communication metrics caused considerable slowdown in in-memory mode. I
> almost fixed it and will merge the fix in the nearest day if benchmarks are
> ok.
>
> That said I think we will be able to start vote in the beginning of the
> next week.
>
> Any objections?
>
> Vladimir.
>
> On Mon, Feb 26, 2018 at 11:22 PM, Denis Magda  wrote:
>
> > Good question!
> >
> > We've been waiting for a sign-off from Alexey Goncharuk and Sergey Kozlov
> > who are benchmarking and trying to address performance issues in
> > coopeartion.
> >
> > Guys, please share your results and forecast.
> >
> > --
> > Denis
> >
> > On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn 
> > wrote:
> >
> > > Igniters,
> > >
> > > Any update on the 2.4 release status? Anything else to merge there?
> > >
> > > Pavel
> > >
> > > On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Alex,
> > > > >
> > > > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any
> dramatic
> > > > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the
> box
> > in
> > > > > 2.3, and it is unsafe in 2.4 as well.
> > > > >
> > > > > The very problem is that we claim ourselves to be ACID, while in
> > > reality
> > > > we
> > > > > are only "AI" out of the box, because durability is not guaranteed
> > due
> > > to
> > > > > zero backups and LOG_ONLY and consistency is not guaranteed due to
> > > > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim
> > > themselves
> > > > > to be ACID, so it is not valid to refer to their defaults.
> > > > >
> > > >
> > > > Vladimir,
> > > > Ignite can be fully ACID, but at the same time have non-ACID
> defaults,
> > as
> > > > long as we clearly document how to get ACID behavior. I do not see an
> > > issue
> > > > with it.
> > > >
> > > > D.
> > > >
> > >
> >
>

Re: Ignite Thin clients for Node.js, Python, PHP

[Pavel Tupitsyn]

Agree with Vladimir and Denis.

I don't think JSON has any place in the thin client protocol,
which is a binary protocol designed to be efficient, with clearly defined
data format.

We already have JSON in "REST" client.


On Thu, Mar 1, 2018 at 8:15 PM, Denis Magda  wrote:

> Totally share Vladimir's stance. Let's support the scope that already
> exists in the protocol and think about the future later. The users will
> definitely guide us to a right direction :)
>
> --
> Denis
>
> On Thu, Mar 1, 2018 at 7:12 AM, Vladimir Ozerov 
> wrote:
>
> > I would extract compute tasks into separate scope. It is better to keep
> > focus on protocol things and basic language support for now. Once we have
> > basic client API in production-ready state, we could consider adding
> > JavaScript to our core compute feature set and then extend it to the
> > clients (which should be trivial provided that core part is ready). We
> > should
> > be ready to spend considerable efforts to prior R&D because dynamic code
> > execution is not very simple thing, especially in terms of security,
> native
> > compilation, etc..
> >
> >
> >
> > On Thu, Mar 1, 2018 at 5:17 PM, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > With regards of thin clients for dynamically typed languages, I think
> > > Ignite needs two following features to shine:
> > >
> > > - Ability to pass JSON to such clients, turn JSON Objects into a
> > > BinaryObjects, which will give ability to index top-level keys in such
> > JSON
> > > with SQL Indexing. Of course this should be integrated with
> > QueryEntities.
> > > - Ability to pass JavaScript snippets to invoke() and affinityCall()
> > > families of calls. On Server node they should be interpreted by Nashorn
> > > (since we've upgraded to Java 8). We should also cache such snippets
> > > pre-interpreted, in this case it can be pretty fast since Nashorn
> compile
> > > to JVM bytecode.
> > >
> > > WDYT?
> > >
> > > --
> > > Ilya Kasnacheev
> > >
> > > 2018-02-20 0:23 GMT+03:00 Alexey Kosenchuk  > com
> > > >:
> > >
> > > > Hi All!
> > > >
> > > > Let us join the party, please ;)
> > > >
> > > > As we see, there is Binary Client Protocol to communicate with Ignite
> > > > cluster and a concept of Thin (lightweight) client.
> > > >
> > > > If there are no objections or duplicated plans, we would like to
> > develop
> > > > Thin Client libs for:
> > > > - Node.js
> > > > - Python
> > > > - PHP
> > > >
> > > > Please add us as contributors and provide access to the Ignite Jira
> > > > components.
> > > >
> > > > Usernames in the Apache Jira:
> > > > alexey.kosenchuk
> > > > ekaterina.vergizova
> > > > pavel.petroshenko
> > > >
> > > > Thanks!
> > > > -Alexey
> > > >
> > >
> >
>

Re: Ignite Thin clients for Node.js, Python, PHP

[Denis Magda]

Totally share Vladimir's stance. Let's support the scope that already
exists in the protocol and think about the future later. The users will
definitely guide us to a right direction :)

--
Denis

On Thu, Mar 1, 2018 at 7:12 AM, Vladimir Ozerov 
wrote:

> I would extract compute tasks into separate scope. It is better to keep
> focus on protocol things and basic language support for now. Once we have
> basic client API in production-ready state, we could consider adding
> JavaScript to our core compute feature set and then extend it to the
> clients (which should be trivial provided that core part is ready). We
> should
> be ready to spend considerable efforts to prior R&D because dynamic code
> execution is not very simple thing, especially in terms of security, native
> compilation, etc..
>
>
>
> On Thu, Mar 1, 2018 at 5:17 PM, Ilya Kasnacheev  >
> wrote:
>
> > With regards of thin clients for dynamically typed languages, I think
> > Ignite needs two following features to shine:
> >
> > - Ability to pass JSON to such clients, turn JSON Objects into a
> > BinaryObjects, which will give ability to index top-level keys in such
> JSON
> > with SQL Indexing. Of course this should be integrated with
> QueryEntities.
> > - Ability to pass JavaScript snippets to invoke() and affinityCall()
> > families of calls. On Server node they should be interpreted by Nashorn
> > (since we've upgraded to Java 8). We should also cache such snippets
> > pre-interpreted, in this case it can be pretty fast since Nashorn compile
> > to JVM bytecode.
> >
> > WDYT?
> >
> > --
> > Ilya Kasnacheev
> >
> > 2018-02-20 0:23 GMT+03:00 Alexey Kosenchuk  com
> > >:
> >
> > > Hi All!
> > >
> > > Let us join the party, please ;)
> > >
> > > As we see, there is Binary Client Protocol to communicate with Ignite
> > > cluster and a concept of Thin (lightweight) client.
> > >
> > > If there are no objections or duplicated plans, we would like to
> develop
> > > Thin Client libs for:
> > > - Node.js
> > > - Python
> > > - PHP
> > >
> > > Please add us as contributors and provide access to the Ignite Jira
> > > components.
> > >
> > > Usernames in the Apache Jira:
> > > alexey.kosenchuk
> > > ekaterina.vergizova
> > > pavel.petroshenko
> > >
> > > Thanks!
> > > -Alexey
> > >
> >
>

Re: Thin client: Java bindings

[Denis Magda]

Alexey, outstanding progress! Can't wait the client rolled out to Ignite
users.

Please create a doc JIRA ticket for 2.5 release and assign to Prachi for
review.

--
Denis

On Thu, Mar 1, 2018 at 4:56 AM, Alexey Kukushkin 
wrote:

> Igniters!
>
> Could you please help with the Java thin client code review:
>
> Documentation 
> Code 
>
> Thank you for the help.
>
> On Mon, Feb 5, 2018 at 5:45 PM, Alexey Kukushkin <
> kukushkinale...@gmail.com>
> wrote:
>
> > We also need asynchronous versions of all the thin client APIs. I created
> > JIRA-7623    to
> address
> > async APIs. Proposed design:
> > Async methods are named by appending "Async" to the corresponding
> > syncMethod, e.g. putAsync, clearAsync.
> > Async methods return java.util.concurrent.Future
> > Async methods are not cancellable (presently cancellation is not
> supported
> > by the thin client protocol)
> > Use thin client protocol's Request ID to map async requests and
> responses.
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
>
>
> --
> Best regards,
> Alexey
>

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Nikolay Izhikov]

Hell, Dima!

Thank you for document!

I'm ready to implement this feature with you.

Igniters, please, share you thoughts about proposed design

[1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf

В Чт, 01/03/2018 в 15:46 +0300, Дмитрий Рябов пишет:
> Hello, Igniters!
> 
> I investigated the issue and wrote some details in a draft document
> [1]. I think we should made IEP for TDE because it is a big change and
> should be described in a single place, but not in a message
> conversation.
> Please, look it and write your thoughts. What is not understandable,
> what should be detailed or described?
> 
> > Where are we going to store keys (MEK) physically? Would it be PKCS#11
> > storage? Where we will store passwords to unlock storage or it will be
> > responibilty of user?
> 
> I think we should provide interface for MEK storage to let users use
> storages they want. I suppose at the first step we should provide very
> simple implementation, which will store MEK on every node and MEK will
> be extracted by administrator during cluster activation process. Once
> MEK is extracted from key store, we decrypt CEKs and destroy open MEK,
> leaving open only cache keys.
> 
> I think external storage is user's worry and we shouldn't give users
> built-in external storage like Oracle Wallet or Microsoft Azure Key
> Vault because it will increase Ignite's complexity too much.
> 
> And yes, we should to comply with the standards like PKCS#11.
> 
> > One more thing is how "node gets MEK from coordinator", if we send
> > cleartext MEK, such security becomes useless also.
> 
> Yeah, that's why we should use secured connection. As I know, we have
> SSL implementation over JDK implementation, am I right? But we must
> ensure to use latest SSL/TLS version.
> 
> [1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf

signature.asc
Description: This is a digitally signed message part

Re: Apache Ignite 2.4 release

[Vladimir Ozerov]

Hi,

We found one nasty regression in DROP COLUMN feature and fixed it today.
Another problem is performance - we found that recent changes to TCP
communication metrics caused considerable slowdown in in-memory mode. I
almost fixed it and will merge the fix in the nearest day if benchmarks are
ok.

That said I think we will be able to start vote in the beginning of the
next week.

Any objections?

Vladimir.

On Mon, Feb 26, 2018 at 11:22 PM, Denis Magda  wrote:

> Good question!
>
> We've been waiting for a sign-off from Alexey Goncharuk and Sergey Kozlov
> who are benchmarking and trying to address performance issues in
> coopeartion.
>
> Guys, please share your results and forecast.
>
> --
> Denis
>
> On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > Any update on the 2.4 release status? Anything else to merge there?
> >
> > Pavel
> >
> > On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Alex,
> > > >
> > > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic
> > > > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box
> in
> > > > 2.3, and it is unsafe in 2.4 as well.
> > > >
> > > > The very problem is that we claim ourselves to be ACID, while in
> > reality
> > > we
> > > > are only "AI" out of the box, because durability is not guaranteed
> due
> > to
> > > > zero backups and LOG_ONLY and consistency is not guaranteed due to
> > > > PRIMARY_SYNC. Neither Cassandra, nor Mongo or any others claim
> > themselves
> > > > to be ACID, so it is not valid to refer to their defaults.
> > > >
> > >
> > > Vladimir,
> > > Ignite can be fully ACID, but at the same time have non-ACID defaults,
> as
> > > long as we clearly document how to get ACID behavior. I do not see an
> > issue
> > > with it.
> > >
> > > D.
> > >
> >
>

Re: Ignite Thin clients for Node.js, Python, PHP

[Vladimir Ozerov]

I would extract compute tasks into separate scope. It is better to keep
focus on protocol things and basic language support for now. Once we have
basic client API in production-ready state, we could consider adding
JavaScript to our core compute feature set and then extend it to the
clients (which should be trivial provided that core part is ready). We should
be ready to spend considerable efforts to prior R&D because dynamic code
execution is not very simple thing, especially in terms of security, native
compilation, etc..



On Thu, Mar 1, 2018 at 5:17 PM, Ilya Kasnacheev 
wrote:

> With regards of thin clients for dynamically typed languages, I think
> Ignite needs two following features to shine:
>
> - Ability to pass JSON to such clients, turn JSON Objects into a
> BinaryObjects, which will give ability to index top-level keys in such JSON
> with SQL Indexing. Of course this should be integrated with QueryEntities.
> - Ability to pass JavaScript snippets to invoke() and affinityCall()
> families of calls. On Server node they should be interpreted by Nashorn
> (since we've upgraded to Java 8). We should also cache such snippets
> pre-interpreted, in this case it can be pretty fast since Nashorn compile
> to JVM bytecode.
>
> WDYT?
>
> --
> Ilya Kasnacheev
>
> 2018-02-20 0:23 GMT+03:00 Alexey Kosenchuk  >:
>
> > Hi All!
> >
> > Let us join the party, please ;)
> >
> > As we see, there is Binary Client Protocol to communicate with Ignite
> > cluster and a concept of Thin (lightweight) client.
> >
> > If there are no objections or duplicated plans, we would like to develop
> > Thin Client libs for:
> > - Node.js
> > - Python
> > - PHP
> >
> > Please add us as contributors and provide access to the Ignite Jira
> > components.
> >
> > Usernames in the Apache Jira:
> > alexey.kosenchuk
> > ekaterina.vergizova
> > pavel.petroshenko
> >
> > Thanks!
> > -Alexey
> >
>

Re: Maven. Issues with flatten plugin

[Dmitry Pavlov]

Hi,

I think we should solve initial problem: "We have to include all transitive
dependencies into poms". I've faced with it during Direct IO plugin
development.

I am sure it is not best practice. Because 1st level dependency version
change will require us to update 2nd level dependencies manually.

So if we can find solution without disabling plugin, we should prefer it.
If only disable may help we should be sure release installation into
maven-central will be successful.

Sincerely,
Dmitriy Pavlov

чт, 1 мар. 2018 г. в 17:23, Petr Ivanov :

> Anton, Vyacheslav — are you proposing assembly process shift to package
> phase, yet leaving the special profile for assembly itself?
>
>
>
> > On 1 Mar 2018, at 14:36, Vyacheslav Daradur  wrote:
> >
> > I agree with Anton, especially with the following statement:
> >>> Ignite should be assemblied using package phase,
> >>> not install in case you want only to assembly it.
> >
> > Approach with "install" phase may lead to hidden issues in a
> > development environment.
> >
> > On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov 
> wrote:
> >>> May be it will be enough to add a profile to parent pom.xml which
> >> activates on deploy stage only.
> >>
> >> I think it will be enough to solve my issue.
> >>
> >>> What pom.xml becomes flattened, what is removed from it and how it
> effects
> >> other modules of the same project?
> >>
> >> pom.xml that are installed to local repository is flattened.
> >>
> >>> Yet, I still do not get a mechanism that causes a problem.
> >>
> >> If you want to reproduce it, just do the following steps:
> >>
> >> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
> >>
> >> 2. run:
> >>
> >> `~/src/ignite/modules/spark> mvn install -U
> >> -Plgpl,examples,scala,-clean-libs,-release,ml
> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> >>
> >> It will execute OK, because maven uses pom.xml from source.
> >>
> >> 3. run(only root dir is changed)
> >>
> >> `~/src/ignite/> mvn install -U
> >> -Plgpl,examples,scala,-clean-libs,-release,ml
> >> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> >> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> >>
> >> Test will fail with ClassNoDefFoundError because maven will use
> >> pom.xml from local repository which is flattened.
> >>
> >> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
> >> Retry step 3. It will succeed.
> >>
> >>
> >> 2018-03-01 14:18 GMT+03:00 Petr Ivanov :
> >>
> >>> May be it will be enough to add a profile to parent pom.xml which
> >>> activates on deploy stage only.
> >>>
> >>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
> >>> becomes flattened, what is removed from it and how it effects other
> modules
> >>> of the same project?
> >>>
> >>>
> >>>
>  On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
> 
>  Andrey, thank you.
> 
>  If there are no other objections I will create ticket for pom
> structure
>  refactoring.
> 
>  Anton, can you help me with it as an Ignite Release Manager?
> 
>  2018-03-01 13:45 GMT+03:00 Andrey Novikov :
> 
> > Nikolay,
> >
> > I think it can be removed, if parent-pom will be installed(deployed).
> >
> > On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov  >
> > wrote:
> >> Hello, Andrey.
> >>
> >> Thanks for an answer.
> >>
> >>> As I remember we use `flatten-maven-plugin` for flattening and
> >>> removing
> > parent relationship in deployed artifacts
> >>
> >> So we need it only in `release` profile?
> >>
> >> I found some earlier discussion about plugin [1]
> >>
> >>> in first versions of build, version was stored in variable in
> parent
> >>> pom
> >>
> >> Got it. Do we need this plugin now?
> >>
> >> [1] http://apache-ignite-developers.2346864.n4.nabble.
> > com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >>
> >> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> >>> Nikolay,
> >>>
> >>> As I remember we use `flatten-maven-plugin` for flattening and
> >>> removing parent relationship in deployed artifacts (parent pom does
> >>> not deploy to repository and in first versions of build, version
> was
> >>> stored in variable in parent pom)
> >>>
> >>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov <
> nizhi...@apache.org>
> > wrote:
>  Hello, Petr.
> 
> > Can you describe your problem
> 
>  In Ignite, maven doesn't include transitive dependencies to test
> > classpath.
>  So we have enlist all dependencies in project pom.xml.
> 
> > desired behaviour
> 
>  All I want is default maven behavior.
> 
>  Please, take a look at pom.xml [1] from line 144.
>  There is a long list of dependencies with `test` scope.
> >>>

Re: Thin client / Binary protocol: questions

[Alexey Kukushkin]

Denis and All,

The links you asked for:

   - Java thin client development branch
   
   - Code review in Upsource
   
   - Thin client documentation
   
  - Quick Start
  
  - API 
  - Security
  
  - High Availability
  

  - Monitoring
  


On Wed, Feb 28, 2018 at 10:44 PM, Denis Magda  wrote:

> Guys, please consider my five cents
>
> >>> - Name/password authentication.
>> >>> When will it be available in the spec?
>> >>> Is there any draft we can look at?
>> It is still under development. Will be available in several weeks. It is
>> better to skip this part for now.
>
>
> Let's give a reference to the branch where the guys can see those changes
> at the protocol level. My guess they want to know what to expect and how it
> can affect the design they've been working on. *Alexey Kukushkin*, please
> point out to your Java thin client branch were you already have this
> functionality embedded.
>
> >>> - SSL/TLS communication.
>> >>> When will it be available in the spec?
>
>
> This doc explains how SSL is enabled on the server side:
> https://apacheignite.readme.io/docs/ssltls
>
> As always, the thin client needs to establish the SSL tunneling first and
> then starts sending the protocol messages. Hope this is good for the
> beginning. Will be happy to answer more specific questions.
>
> >>>  - May we easily use 3rd-party components with the following licenses?
>> As a general rule of thumb it is not recommended to use any external
>> libraries. The whole Ignite core is built with almost no dependencies, not
>> to say about much less complex thin clients ans JDBC/ODBC drivers. IMO the
>> only possible place where external dependency might be required is SSL
>> support (e.g. we use OpenSSL for ODBC).
>
>
> I wouldn't discourage Alexey from using 3rd party libs taking Ignite core
> as an example. Ignite optional libs use 3rd party libs a lot. Take our REST
> protocol for instance that uses Jetty and not based on its own HTTP server
> implementation. So, I would alter the rule of thumb a little bit - if it
> takes weeks to develop a functionality already available in a 3rd party lib
> then let's go for the 3rd party.
>
> Here is a list of licenses that can be included in Ignite without any
> extra permissions or licensing concerns: https://www.apache.
> org/legal/resolved.html
>
> According to the doc you're free to use Apache 2.0 and MIT, but LGPL code
> can't be delivered within Ignite.
>
> --
> Denis
>
> On Wed, Feb 28, 2018 at 2:32 AM, Vladimir Ozerov 
> wrote:
>
>> Hi Alex,
>>
>> >>> Just to double-check - we should support the types from the spec only.
>> Right?
>> Please provide the list of types you are in doubt of
>>
>> >>> - Key-Value and SQL and Scan Queries.
>> >>>  The most of operations has "Flag" field in Request: "Pass 0 for
>> default, or 1 to keep the value in binary form."
>> >>>  What is it for?
>> Please see IgniteCache.withKeepBinary method. For SQL this flag has no
>> effect. For complex SCAN queries with filters this defines whether with
>> pass real objects or binary objects to the filter.
>>
>> >>>  Java and .net libs always pass 0. Why?
>> There is no Java client at the moment. As far as .NET, it doesn't support
>> binary mode at the moment.
>>
>> >>>  Why an app working remotely via the binary protocol should know the
>> exact platform where the node is running?
>> Because if node is started from .NET, you may execute both Java filters
>> and
>> .NET fitlers on it. This flag defines what kind of filter is passed.
>>
>> >>> - Binary Type operations (register/get type name, put/get type).
>> >>>  What are they for?
>> >>>  What is a use case?
>> Please get familiar with binary type concepts, especially binary metadata:
>> https://apacheignite.readme.io/docs#binaryobject-type-metadata
>>
>> >>>  - Just interesting - why it was decided that hash code should be also
>> calculated on a client side? Why it could not be returned by a server
>> side?
>> Because this is inefficient - we would have to rebuild binary objects on
>> the server side. Instead, it is better to implement pretty simple routine
>> for hashcode calculation inside every client.
>>
>> >>> - Name/password authentication.
>> >>> When will it be available in the spec?
>> >>> Is there any draft we can look at?
>> It is still under development. Will be available in several weeks. It is
>> better to skip this part for now.
>>
>> >>> - SSL/TLS communication.
>> >>> W

[GitHub] ignite pull request #3588: IGNITE-7789: fix IgniteClientReconnectApiExceptio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3588


---

Re: Maven. Issues with flatten plugin

[Petr Ivanov]

Anton, Vyacheslav — are you proposing assembly process shift to package phase, 
yet leaving the special profile for assembly itself?



> On 1 Mar 2018, at 14:36, Vyacheslav Daradur  wrote:
> 
> I agree with Anton, especially with the following statement:
>>> Ignite should be assemblied using package phase,
>>> not install in case you want only to assembly it.
> 
> Approach with "install" phase may lead to hidden issues in a
> development environment.
> 
> On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov  wrote:
>>> May be it will be enough to add a profile to parent pom.xml which
>> activates on deploy stage only.
>> 
>> I think it will be enough to solve my issue.
>> 
>>> What pom.xml becomes flattened, what is removed from it and how it effects
>> other modules of the same project?
>> 
>> pom.xml that are installed to local repository is flattened.
>> 
>>> Yet, I still do not get a mechanism that causes a problem.
>> 
>> If you want to reproduce it, just do the following steps:
>> 
>> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
>> 
>> 2. run:
>> 
>> `~/src/ignite/modules/spark> mvn install -U
>> -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>> 
>> It will execute OK, because maven uses pom.xml from source.
>> 
>> 3. run(only root dir is changed)
>> 
>> `~/src/ignite/> mvn install -U
>> -Plgpl,examples,scala,-clean-libs,-release,ml
>> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
>> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>> 
>> Test will fail with ClassNoDefFoundError because maven will use
>> pom.xml from local repository which is flattened.
>> 
>> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
>> Retry step 3. It will succeed.
>> 
>> 
>> 2018-03-01 14:18 GMT+03:00 Petr Ivanov :
>> 
>>> May be it will be enough to add a profile to parent pom.xml which
>>> activates on deploy stage only.
>>> 
>>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
>>> becomes flattened, what is removed from it and how it effects other modules
>>> of the same project?
>>> 
>>> 
>>> 
 On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
 
 Andrey, thank you.
 
 If there are no other objections I will create ticket for pom structure
 refactoring.
 
 Anton, can you help me with it as an Ignite Release Manager?
 
 2018-03-01 13:45 GMT+03:00 Andrey Novikov :
 
> Nikolay,
> 
> I think it can be removed, if parent-pom will be installed(deployed).
> 
> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov 
> wrote:
>> Hello, Andrey.
>> 
>> Thanks for an answer.
>> 
>>> As I remember we use `flatten-maven-plugin` for flattening and
>>> removing
> parent relationship in deployed artifacts
>> 
>> So we need it only in `release` profile?
>> 
>> I found some earlier discussion about plugin [1]
>> 
>>> in first versions of build, version was stored in variable in parent
>>> pom
>> 
>> Got it. Do we need this plugin now?
>> 
>> [1] http://apache-ignite-developers.2346864.n4.nabble.
> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>> 
>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>>> Nikolay,
>>> 
>>> As I remember we use `flatten-maven-plugin` for flattening and
>>> removing parent relationship in deployed artifacts (parent pom does
>>> not deploy to repository and in first versions of build, version was
>>> stored in variable in parent pom)
>>> 
>>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov 
> wrote:
 Hello, Petr.
 
> Can you describe your problem
 
 In Ignite, maven doesn't include transitive dependencies to test
> classpath.
 So we have enlist all dependencies in project pom.xml.
 
> desired behaviour
 
 All I want is default maven behavior.
 
 Please, take a look at pom.xml [1] from line 144.
 There is a long list of dependencies with `test` scope.
 Actually, all of them are available as a transitive dependency from
> `spark-core`.
 
 We doesn't have to enlist them in every other project that doesn't
> use `flatten-plugin`.
 
 [1] https://github.com/apache/ignite/blob/master/modules/
> spark/pom.xml#L144
 
 В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> Nikolay,
> 
> 
> Can you describe your problem and desired behaviour more
> thoroughly, please?
> 
> 
> 
>> On 28 Feb 2018, at 21:16, Nikolay Izhikov 
> wrote:
>> 
>> Hello, Igniters.
>> 
>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> As far as I can understand it minimize pom.xml before 

Re: Ignite Thin clients for Node.js, Python, PHP

[Ilya Kasnacheev]

With regards of thin clients for dynamically typed languages, I think
Ignite needs two following features to shine:

- Ability to pass JSON to such clients, turn JSON Objects into a
BinaryObjects, which will give ability to index top-level keys in such JSON
with SQL Indexing. Of course this should be integrated with QueryEntities.
- Ability to pass JavaScript snippets to invoke() and affinityCall()
families of calls. On Server node they should be interpreted by Nashorn
(since we've upgraded to Java 8). We should also cache such snippets
pre-interpreted, in this case it can be pretty fast since Nashorn compile
to JVM bytecode.

WDYT?

-- 
Ilya Kasnacheev

2018-02-20 0:23 GMT+03:00 Alexey Kosenchuk :

> Hi All!
>
> Let us join the party, please ;)
>
> As we see, there is Binary Client Protocol to communicate with Ignite
> cluster and a concept of Thin (lightweight) client.
>
> If there are no objections or duplicated plans, we would like to develop
> Thin Client libs for:
> - Node.js
> - Python
> - PHP
>
> Please add us as contributors and provide access to the Ignite Jira
> components.
>
> Usernames in the Apache Jira:
> alexey.kosenchuk
> ekaterina.vergizova
> pavel.petroshenko
>
> Thanks!
> -Alexey
>

[jira] [Created] (IGNITE-7860) JDBC thin driver: set default socket buffer sizes to greater value

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7860:
---

 Summary: JDBC thin driver: set default socket buffer sizes to 
greater value
 Key: IGNITE-7860
 URL: https://issues.apache.org/jira/browse/IGNITE-7860
 Project: Ignite
  Issue Type: Task
  Components: jdbc
Affects Versions: 2.3, 2.4
Reporter: Vladimir Ozerov
Assignee: Taras Ledkov
 Fix For: 2.5


Currently they are set to zero, meaning that OS will use it's default. But 
tyically these defaults are too pessimistic. Let's set them to 64Kb.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts

[Ilya Kasnacheev]

I agree with you.

I think we could restrict usage of e.g. setConnectTimeout/setSocketTimeout
to people extending SPIs, since different implementations may need
different values.

However, for user configurations we should only expose timeouts we can
explain, everything else should have reasonable values.

-- 
Ilya Kasnacheev

2018-03-01 17:01 GMT+03:00 Alexey Popov :

> Hi Igniters,
>
> We often see similar questions from users and customers related to
> IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts and
> their
> relations. And we see several side-effects after incorrect timeout
> configuration.
>
> I tried to briefly describe these timeout settings (please see below) and
> found out that the most of them do not have sense in terms of cluster
> functions/operations and could not be explained to the users.
>
> I propose to deprecate most of them and leave only the timeouts we can
> explain in common terms ( (setFailureDetectionTimeout, setNetworkTimeout,
> setJoinTimeout and some others).
>
> Please let me know your thoughts.
>
> Thanks,
> Alexey
>
> GLOBAL:
>
> IgniteConfiguration.setNetworkTimeout:
> It is a global timeout for high-level operations where a network is
> involved. For instance, IgniteMessaging delivery uses this timeout or
> DiscoverySpi handshake.
>
> IgniteConfiguration.setFailureDetectionTimeout:
> It is a global timeout for detecting failures at IgniteSpi implementations
> (including DiscoverySpi and CommunicationSpi).
> The failure detection algorithm actually limits a range of simple network
> operations related to a single logical operation (for instance, a reliable
> delivery of some DiscoverySpi message within a cluster).
> Failure detection timeout is a cumulative timeout for a socket connection,
> sending and receiving data bytes and all possible socket retries (if some
> failure happens).
> This timeout is intended to simplify the failure detection condition from a
> user perspective.
>
> IgniteConfiguration.setClientFailureDetectionTimeout: - it is a special
> case
> for DiscoverySpi client-node Ignite.
>
> TCP DISCOVERY SPI:
>
> If you need more control over failure detection algorithm for
> TcpDiscoverySpi you can explicitly use the following low-level options
> (that
> will disable failureDetectoinTimeout logic):
>
> 1. TcpDiscoverySpi.setConnectTimeout - socket connection timeout
> 2. TcpDiscoverySpi.setReconnectCount - number of reconnect attempts used
> when establishing connection with the remote node and sending messages to
> it
> 3. TcpDiscoverySpi.setSocketTimeout - socket write timeout. The write
> operation will be repeated getReconnectCount() times if it exceeds this
> timeout
> 4. TcpDiscoverySpi.setAckTimeout - message acknowledgment timeout. If a
> message acknowledgment is not received within this timeout, sending is
> considered as failed and SPI will try to repeat send operation. It is
> automatically doubled for simultaneous retries up to getMaxAckTimeout
> value.
> 5. TcpDiscoverySpi.setMaxAckTimeout - maximum connection timeout, if the
> getAckTimeout reaches getMaxAckTimeout then SPI give up sending retries
>
> Another important TcpDiscoverySpi timeouts:
>
> TcpDiscoverySpi.setJoinTimeout - It is a timeout for join process when a
> new/restarted node joins a cluster. The node tries to connect to all
> available IP addresses provided by ipFinder within this timeout.
> If the timeout is exceeded, the node will give up and throw an exception
> from Ignition.start().
>
> TcpDiscoverySpi.setNetworkTimeout - timeout for high-level operations like
> handshake. It looks like it should be deprecated and the
> IgniteConfiguration.getNetworkTimeout should be used here.
>
> TCP COMMUNICATION SPI:
>
> If you need more control over failure detection algorithm for
> TcpCommunicationSpi you can explicitly use the following low-level options
> (that will disable failureDetectoinTimeout logic):
>
> 1. TcpCommunicationSpi.setConnectTimeout - socket connection timeout, will
> be automatically doubled for simultaneous retries (up to getReconnectCount)
> related to a single logical operation
> 2. TcpCommunicationSpi.setMaxConnectTimeout - maximum connection timeout,
> the higher limit of getReconnectCount-times doubled getConnectTimeout
> 3. TcpCommunicationSpi.setReconnectCount - number of reconnect attempts
> used
> when establishing connection with the remote node and sending messages to
> it
>
> Another important TcpCommunicationSpi timeouts:
>
> TcpDiscoverySpi.setSocketWriteTimeout - timeout to send a message.
> TcpDiscoverySpi.setIdleConnectionTimeout - maximum idle connection timeout
> upon which a connection will be closed.
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Maven. Issues with flatten plugin

[Nikolay Izhikov]

Petr.

Thank you for trying!

Did you remove 'test' dependencies before running commands?

Because I commit in master only correct pom.xml for current build process,
of course.

But, to make things works I have to copy paste transitive dependencies from
spark-core.

1 марта 2018 г. 4:55 PM пользователь "Petr Ivanov" 
написал:

>
> >
> > I don't get what is the point.
> > Did you try to reproduce issue?
> > Or should I provide full traces to you?
> >
>
> My point in inability to fully understand and describe the problem in
> terms of mechanism which causes it.
> For now I can see only indirect guessing.
>
>
> And yes, I’ve run your commands and both times (in spark module directory
> and in root) it produces the same result:
>
> 1.
> ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn
> install -U -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 47.071 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
> [WARNING] The requested profile "ml" could not be activated because it
> does not exist.
>
> 2.
> ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v
> 1.8)" mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 53.468 s - in org.apache.ignite.testsuites.IgniteRDDTestSuite
> [WARNING] The requested profile "lgpl" could not be activated because it
> does not exist.
> [WARNING] The requested profile "examples" could not be activated because
> it does not exist.
> [WARNING] The requested profile "scala" could not be activated because it
> does not exist.
> [WARNING] The requested profile "ml" could not be activated because it
> does not exist.

IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts

[Alexey Popov]

Hi Igniters,

We often see similar questions from users and customers related to
IgniteConfiguration, TcpDiscoverySpi, TcpCommunicationSpi timeouts and their
relations. And we see several side-effects after incorrect timeout
configuration.

I tried to briefly describe these timeout settings (please see below) and
found out that the most of them do not have sense in terms of cluster
functions/operations and could not be explained to the users.

I propose to deprecate most of them and leave only the timeouts we can
explain in common terms ( (setFailureDetectionTimeout, setNetworkTimeout,
setJoinTimeout and some others).

Please let me know your thoughts.

Thanks,
Alexey

GLOBAL:

IgniteConfiguration.setNetworkTimeout:
It is a global timeout for high-level operations where a network is
involved. For instance, IgniteMessaging delivery uses this timeout or
DiscoverySpi handshake.

IgniteConfiguration.setFailureDetectionTimeout:
It is a global timeout for detecting failures at IgniteSpi implementations
(including DiscoverySpi and CommunicationSpi).
The failure detection algorithm actually limits a range of simple network
operations related to a single logical operation (for instance, a reliable
delivery of some DiscoverySpi message within a cluster).
Failure detection timeout is a cumulative timeout for a socket connection,
sending and receiving data bytes and all possible socket retries (if some
failure happens). 
This timeout is intended to simplify the failure detection condition from a
user perspective.

IgniteConfiguration.setClientFailureDetectionTimeout: - it is a special case
for DiscoverySpi client-node Ignite.

TCP DISCOVERY SPI:

If you need more control over failure detection algorithm for
TcpDiscoverySpi you can explicitly use the following low-level options (that
will disable failureDetectoinTimeout logic):

1. TcpDiscoverySpi.setConnectTimeout - socket connection timeout
2. TcpDiscoverySpi.setReconnectCount - number of reconnect attempts used
when establishing connection with the remote node and sending messages to it
3. TcpDiscoverySpi.setSocketTimeout - socket write timeout. The write
operation will be repeated getReconnectCount() times if it exceeds this
timeout
4. TcpDiscoverySpi.setAckTimeout - message acknowledgment timeout. If a
message acknowledgment is not received within this timeout, sending is
considered as failed and SPI will try to repeat send operation. It is
automatically doubled for simultaneous retries up to getMaxAckTimeout value.
5. TcpDiscoverySpi.setMaxAckTimeout - maximum connection timeout, if the
getAckTimeout reaches getMaxAckTimeout then SPI give up sending retries

Another important TcpDiscoverySpi timeouts:

TcpDiscoverySpi.setJoinTimeout - It is a timeout for join process when a
new/restarted node joins a cluster. The node tries to connect to all
available IP addresses provided by ipFinder within this timeout.
If the timeout is exceeded, the node will give up and throw an exception
from Ignition.start().

TcpDiscoverySpi.setNetworkTimeout - timeout for high-level operations like
handshake. It looks like it should be deprecated and the
IgniteConfiguration.getNetworkTimeout should be used here.

TCP COMMUNICATION SPI:

If you need more control over failure detection algorithm for
TcpCommunicationSpi you can explicitly use the following low-level options
(that will disable failureDetectoinTimeout logic):

1. TcpCommunicationSpi.setConnectTimeout - socket connection timeout, will
be automatically doubled for simultaneous retries (up to getReconnectCount)
related to a single logical operation 
2. TcpCommunicationSpi.setMaxConnectTimeout - maximum connection timeout,
the higher limit of getReconnectCount-times doubled getConnectTimeout
3. TcpCommunicationSpi.setReconnectCount - number of reconnect attempts used
when establishing connection with the remote node and sending messages to it

Another important TcpCommunicationSpi timeouts:

TcpDiscoverySpi.setSocketWriteTimeout - timeout to send a message.
TcpDiscoverySpi.setIdleConnectionTimeout - maximum idle connection timeout
upon which a connection will be closed.




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Maven. Issues with flatten plugin

[Petr Ivanov]

> 
> I don't get what is the point.
> Did you try to reproduce issue?
> Or should I provide full traces to you?
> 

My point in inability to fully understand and describe the problem in terms of 
mechanism which causes it.
For now I can see only indirect guessing.


And yes, I’ve run your commands and both times (in spark module directory and 
in root) it produces the same result:

1.
ignite (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" mvn install -U 
-Plgpl,examples,scala,-clean-libs,-release,ml 
-Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite 
-Dmaven.javadoc.skip=true -DfailIfNoTests=false
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.071 s 
- in org.apache.ignite.testsuites.IgniteRDDTestSuite
[WARNING] The requested profile "ml" could not be activated because it does not 
exist.

2.
ignite/modules/spark (master) $ JAVA_HOME="$(/usr/libexec/java_home -v 1.8)" 
mvn install -U -Plgpl,examples,scala,-clean-libs,-release,ml 
-Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite 
-Dmaven.javadoc.skip=true -DfailIfNoTests=false
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.468 s 
- in org.apache.ignite.testsuites.IgniteRDDTestSuite
[WARNING] The requested profile "lgpl" could not be activated because it does 
not exist.
[WARNING] The requested profile "examples" could not be activated because it 
does not exist.
[WARNING] The requested profile "scala" could not be activated because it does 
not exist.
[WARNING] The requested profile "ml" could not be activated because it does not 
exist.

[jira] [Created] (IGNITE-7859) SQL streaming support via native API

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7859:
---

 Summary: SQL streaming support via native API
 Key: IGNITE-7859
 URL: https://issues.apache.org/jira/browse/IGNITE-7859
 Project: Ignite
  Issue Type: Task
Reporter: Alexander Paschenko


In addition to streaming via thin JDBC driver, ability to run same {{SET 
STREAMING}} command should be added to cache API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3591: IGNITE-7253

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/3591

IGNITE-7253



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7253-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3591.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3591


commit 4477a14a7820a673e1d00040fbef25511e149d79
Author: Alexander Paschenko 
Date:   2018-02-28T09:16:48Z

IGNITE-7253 Initial streaming state

commit 3502928dafc2aa3a49e58ef703b6fc2d2f02458b
Author: Alexander Paschenko 
Date:   2018-02-28T12:36:39Z

IGNITE-7253 SET keyword

commit 6407a53d75d052e7d00d361e20372688acc20ed8
Author: Alexander Paschenko 
Date:   2018-02-28T15:57:20Z

IGNITE-7253 SET STREAMING params

commit 4945069f44d81038880f0444e85ae19b5d50b764
Author: Alexander Paschenko 
Date:   2018-03-01T09:13:46Z

IGNITE-7253 Removed streaming parameters from connection string

commit b7d9c7d0cc36f5019bbf200c0ef2ac5094fae01a
Author: Alexander Paschenko 
Date:   2018-03-01T13:49:49Z

IGNITE-7253 finishing.

commit 5ed144b471baec048d8f2423122c2ad73c96a43c
Author: Alexander Paschenko 
Date:   2018-03-01T13:51:05Z

IGNITE-7253 fixed comment.




---

[jira] [Created] (IGNITE-7858) "Cannot find cache" error for existing cache on unstable topology

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7858:


 Summary: "Cannot find cache" error for existing cache on unstable 
topology 
 Key: IGNITE-7858
 URL: https://issues.apache.org/jira/browse/IGNITE-7858
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: Alexey Kukushkin


Java thin client high availability test fails with "Cannot find cache" error 
for existing cache.

Reproducer:

git fetch professional
git checkout ignite-7421
Open org.apache.ignite.thinclient.system.ReliabilityTest and search for "TODO: 
fix CACHE_DOES_NOT_EXIST server error". Then remove the try/catch block leaving 
only assertion call.
Run test testFailover



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Maven. Issues with flatten plugin

[Nikolay Izhikov]

> Which pom.xml? From the module modules/spark?

Yes.

> As I understand maven, test will be executed using dependencies from the
reactor, not from local repository.

I don't get what is the point.
Did you try to reproduce issue?
Or should I provide full traces to you?

> What am I missing?

I see that whole build process is done in a very strange way.
Because of flatten plugin.

I described and able reproduce(both locally and on TC) specific issues that
turns whole pom.xml into the mess.
I'm pretty sure that source of issue is flatten plugin.
I'm not the first person who got this errors.

My propositions are:

1. Let's discuss how we should fix it.
2. Let's fix it!


2018-03-01 15:35 GMT+03:00 Petr Ivanov :

>
> > 2. run:
> >
> > `~/src/ignite/modules/spark> mvn install -U
> > -Plgpl,examples,scala,-clean-libs,-release,ml
> > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> > -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> >
> > It will execute OK, because maven uses pom.xml from source.
>
> Which pom.xml? From the module modules/spark?
> Also there is no profiles lgpl,examples,scala,-clean-libs,-release,ml in
> that module.
>
> >
> > 3. run(only root dir is changed)
> >
> > `~/src/ignite/> mvn install -U
> > -Plgpl,examples,scala,-clean-libs,-release,ml
> > -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> > -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> As I understand maven, test will be executed using dependencies from the
> reactor, not from local repository.
>
>
> What am I missing?

Re: Maven. Issues with flatten plugin

[Petr Ivanov]

> 2. run:
> 
> `~/src/ignite/modules/spark> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
> 
> It will execute OK, because maven uses pom.xml from source.

Which pom.xml? From the module modules/spark?
Also there is no profiles lgpl,examples,scala,-clean-libs,-release,ml in that 
module.

> 
> 3. run(only root dir is changed)
> 
> `~/src/ignite/> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`

As I understand maven, test will be executed using dependencies from the 
reactor, not from local repository.


What am I missing?

Re: Thin client: Java bindings

[Alexey Kukushkin]

Igniters!

Could you please help with the Java thin client code review:

Documentation 
Code 

Thank you for the help.

On Mon, Feb 5, 2018 at 5:45 PM, Alexey Kukushkin 
wrote:

> We also need asynchronous versions of all the thin client APIs. I created
> JIRA-7623    to address
> async APIs. Proposed design:
> Async methods are named by appending "Async" to the corresponding
> syncMethod, e.g. putAsync, clearAsync.
> Async methods return java.util.concurrent.Future
> Async methods are not cancellable (presently cancellation is not supported
> by the thin client protocol)
> Use thin client protocol's Request ID to map async requests and responses.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



-- 
Best regards,
Alexey

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Дмитрий Рябов]

Hello, Igniters!

I investigated the issue and wrote some details in a draft document
[1]. I think we should made IEP for TDE because it is a big change and
should be described in a single place, but not in a message
conversation.
Please, look it and write your thoughts. What is not understandable,
what should be detailed or described?

> Where are we going to store keys (MEK) physically? Would it be PKCS#11
> storage? Where we will store passwords to unlock storage or it will be
> responibilty of user?

I think we should provide interface for MEK storage to let users use
storages they want. I suppose at the first step we should provide very
simple implementation, which will store MEK on every node and MEK will
be extracted by administrator during cluster activation process. Once
MEK is extracted from key store, we decrypt CEKs and destroy open MEK,
leaving open only cache keys.

I think external storage is user's worry and we shouldn't give users
built-in external storage like Oracle Wallet or Microsoft Azure Key
Vault because it will increase Ignite's complexity too much.

And yes, we should to comply with the standards like PKCS#11.

> One more thing is how "node gets MEK from coordinator", if we send
> cleartext MEK, such security becomes useless also.

Yeah, that's why we should use secured connection. As I know, we have
SSL implementation over JDK implementation, am I right? But we must
ensure to use latest SSL/TLS version.

[1] https://1drv.ms/w/s!AqZdfua4UpmuhneoVhOCiXSUBGIf

[GitHub] ignite pull request #3536: IGNITE-7717 Update discovery caches on topologies...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3536


---

Re: Next Steps: GA Grid: Request to contribute GA library to Apache Ignite

[Yury Babak]

Turik,

I've made a review, please check it on the github, there is a question about
pom files there. Also please update your branch from master.

At this point we almost ready for merge.

Regards,
Yury



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-7857) Extend IgniteWalFlushMultiNodeFailoverAbstractSelfTest to cover MMAP mode

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-7857:


 Summary: Extend IgniteWalFlushMultiNodeFailoverAbstractSelfTest to 
cover MMAP mode
 Key: IGNITE-7857
 URL: https://issues.apache.org/jira/browse/IGNITE-7857
 Project: Ignite
  Issue Type: Improvement
Reporter: Ilya Lantukh
Assignee: Ilya Lantukh






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7856) ODBC: Support COPY command

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7856:
---

 Summary: ODBC: Support COPY command
 Key: IGNITE-7856
 URL: https://issues.apache.org/jira/browse/IGNITE-7856
 Project: Ignite
  Issue Type: Task
  Components: odbc
Reporter: Vladimir Ozerov
Assignee: Igor Sapego
 Fix For: 2.5


Need to do that in the same way as it is implemented in JDBC driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7855) ODBC: Supoprt streaming mode

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7855:
---

 Summary: ODBC: Supoprt streaming mode
 Key: IGNITE-7855
 URL: https://issues.apache.org/jira/browse/IGNITE-7855
 Project: Ignite
  Issue Type: Task
  Components: odbc
Reporter: Vladimir Ozerov
Assignee: Igor Sapego
 Fix For: 2.5


We need to support streaming mode in the same way it is done for JDBC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7854) Java thin client: add username/password authentication support

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7854:
---

 Summary: Java thin client: add username/password authentication 
support
 Key: IGNITE-7854
 URL: https://issues.apache.org/jira/browse/IGNITE-7854
 Project: Ignite
  Issue Type: Task
  Components: thin client
Reporter: Vladimir Ozerov
Assignee: Taras Ledkov
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7852) ODBC: Support username/password authentication

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7852:
---

 Summary: ODBC: Support username/password authentication
 Key: IGNITE-7852
 URL: https://issues.apache.org/jira/browse/IGNITE-7852
 Project: Ignite
  Issue Type: Task
  Components: odbc
Reporter: Vladimir Ozerov
Assignee: Igor Sapego
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7853) .NET thin client: supoprt username/password authentication

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7853:
---

 Summary: .NET thin client: supoprt username/password authentication
 Key: IGNITE-7853
 URL: https://issues.apache.org/jira/browse/IGNITE-7853
 Project: Ignite
  Issue Type: Task
  Components: thin client
Reporter: Vladimir Ozerov
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3586: IGNITE-7843: refresh index column ids on DROP COL...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3586


---

Re: Maven. Issues with flatten plugin

[Vyacheslav Daradur]

I agree with Anton, especially with the following statement:
>> Ignite should be assemblied using package phase,
>> not install in case you want only to assembly it.

Approach with "install" phase may lead to hidden issues in a
development environment.

On Thu, Mar 1, 2018 at 2:26 PM, Nikolay Izhikov  wrote:
>> May be it will be enough to add a profile to parent pom.xml which
> activates on deploy stage only.
>
> I think it will be enough to solve my issue.
>
>> What pom.xml becomes flattened, what is removed from it and how it effects
> other modules of the same project?
>
> pom.xml that are installed to local repository is flattened.
>
>> Yet, I still do not get a mechanism that causes a problem.
>
> If you want to reproduce it, just do the following steps:
>
> 1. remove spark dependencies with `scope=test` from `spark/pom.xml`
>
> 2. run:
>
> `~/src/ignite/modules/spark> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> It will execute OK, because maven uses pom.xml from source.
>
> 3. run(only root dir is changed)
>
> `~/src/ignite/> mvn install -U
> -Plgpl,examples,scala,-clean-libs,-release,ml
> -Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
> -Dmaven.javadoc.skip=true -DfailIfNoTests=false`
>
> Test will fail with ClassNoDefFoundError because maven will use
> pom.xml from local repository which is flattened.
>
> 4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
> Retry step 3. It will succeed.
>
>
> 2018-03-01 14:18 GMT+03:00 Petr Ivanov :
>
>> May be it will be enough to add a profile to parent pom.xml which
>> activates on deploy stage only.
>>
>> Yet, I still do not get a mechanism that causes a problem. What pom.xml
>> becomes flattened, what is removed from it and how it effects other modules
>> of the same project?
>>
>>
>>
>> > On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
>> >
>> > Andrey, thank you.
>> >
>> > If there are no other objections I will create ticket for pom structure
>> > refactoring.
>> >
>> > Anton, can you help me with it as an Ignite Release Manager?
>> >
>> > 2018-03-01 13:45 GMT+03:00 Andrey Novikov :
>> >
>> >> Nikolay,
>> >>
>> >> I think it can be removed, if parent-pom will be installed(deployed).
>> >>
>> >> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov 
>> >> wrote:
>> >>> Hello, Andrey.
>> >>>
>> >>> Thanks for an answer.
>> >>>
>>  As I remember we use `flatten-maven-plugin` for flattening and
>> removing
>> >> parent relationship in deployed artifacts
>> >>>
>> >>> So we need it only in `release` profile?
>> >>>
>> >>> I found some earlier discussion about plugin [1]
>> >>>
>>  in first versions of build, version was stored in variable in parent
>> pom
>> >>>
>> >>> Got it. Do we need this plugin now?
>> >>>
>> >>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> >> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>> >>>
>> >>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>>  Nikolay,
>> 
>>  As I remember we use `flatten-maven-plugin` for flattening and
>>  removing parent relationship in deployed artifacts (parent pom does
>>  not deploy to repository and in first versions of build, version was
>>  stored in variable in parent pom)
>> 
>>  On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov 
>> >> wrote:
>> > Hello, Petr.
>> >
>> >> Can you describe your problem
>> >
>> > In Ignite, maven doesn't include transitive dependencies to test
>> >> classpath.
>> > So we have enlist all dependencies in project pom.xml.
>> >
>> >> desired behaviour
>> >
>> > All I want is default maven behavior.
>> >
>> > Please, take a look at pom.xml [1] from line 144.
>> > There is a long list of dependencies with `test` scope.
>> > Actually, all of them are available as a transitive dependency from
>> >> `spark-core`.
>> >
>> > We doesn't have to enlist them in every other project that doesn't
>> >> use `flatten-plugin`.
>> >
>> > [1] https://github.com/apache/ignite/blob/master/modules/
>> >> spark/pom.xml#L144
>> >
>> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> >> Nikolay,
>> >>
>> >>
>> >> Can you describe your problem and desired behaviour more
>> >> thoroughly, please?
>> >>
>> >>
>> >>
>> >>> On 28 Feb 2018, at 21:16, Nikolay Izhikov 
>> >> wrote:
>> >>>
>> >>> Hello, Igniters.
>> >>>
>> >>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> >>> As far as I can understand it minimize pom.xml before it
>> >> installed in repository.
>> >>>
>> >>> It introduce some strange behavior in modules:
>> >>>
>> >>> I must to enlist all test dependencies in my module to get tests
>> >> work.
>> >>> Otherwise tests fails with ClassNoDefFoundError for transitive
>> >> dependencies.
>> >>> It h

Re: Maven. Issues with flatten plugin

[Nikolay Izhikov]

> May be it will be enough to add a profile to parent pom.xml which
activates on deploy stage only.

I think it will be enough to solve my issue.

> What pom.xml becomes flattened, what is removed from it and how it effects
other modules of the same project?

pom.xml that are installed to local repository is flattened.

> Yet, I still do not get a mechanism that causes a problem.

If you want to reproduce it, just do the following steps:

1. remove spark dependencies with `scope=test` from `spark/pom.xml`

2. run:

`~/src/ignite/modules/spark> mvn install -U
-Plgpl,examples,scala,-clean-libs,-release,ml
-Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
-Dmaven.javadoc.skip=true -DfailIfNoTests=false`

It will execute OK, because maven uses pom.xml from source.

3. run(only root dir is changed)

`~/src/ignite/> mvn install -U
-Plgpl,examples,scala,-clean-libs,-release,ml
-Dtest=org.apache.ignite.testsuites.IgniteRDDTestSuite
-Dmaven.javadoc.skip=true -DfailIfNoTests=false`

Test will fail with ClassNoDefFoundError because maven will use
pom.xml from local repository which is flattened.

4. remove flatten plugin from `parent/pom.xml`. Re Install parent pom.
Retry step 3. It will succeed.


2018-03-01 14:18 GMT+03:00 Petr Ivanov :

> May be it will be enough to add a profile to parent pom.xml which
> activates on deploy stage only.
>
> Yet, I still do not get a mechanism that causes a problem. What pom.xml
> becomes flattened, what is removed from it and how it effects other modules
> of the same project?
>
>
>
> > On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
> >
> > Andrey, thank you.
> >
> > If there are no other objections I will create ticket for pom structure
> > refactoring.
> >
> > Anton, can you help me with it as an Ignite Release Manager?
> >
> > 2018-03-01 13:45 GMT+03:00 Andrey Novikov :
> >
> >> Nikolay,
> >>
> >> I think it can be removed, if parent-pom will be installed(deployed).
> >>
> >> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov 
> >> wrote:
> >>> Hello, Andrey.
> >>>
> >>> Thanks for an answer.
> >>>
>  As I remember we use `flatten-maven-plugin` for flattening and
> removing
> >> parent relationship in deployed artifacts
> >>>
> >>> So we need it only in `release` profile?
> >>>
> >>> I found some earlier discussion about plugin [1]
> >>>
>  in first versions of build, version was stored in variable in parent
> pom
> >>>
> >>> Got it. Do we need this plugin now?
> >>>
> >>> [1] http://apache-ignite-developers.2346864.n4.nabble.
> >> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >>>
> >>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>  Nikolay,
> 
>  As I remember we use `flatten-maven-plugin` for flattening and
>  removing parent relationship in deployed artifacts (parent pom does
>  not deploy to repository and in first versions of build, version was
>  stored in variable in parent pom)
> 
>  On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov 
> >> wrote:
> > Hello, Petr.
> >
> >> Can you describe your problem
> >
> > In Ignite, maven doesn't include transitive dependencies to test
> >> classpath.
> > So we have enlist all dependencies in project pom.xml.
> >
> >> desired behaviour
> >
> > All I want is default maven behavior.
> >
> > Please, take a look at pom.xml [1] from line 144.
> > There is a long list of dependencies with `test` scope.
> > Actually, all of them are available as a transitive dependency from
> >> `spark-core`.
> >
> > We doesn't have to enlist them in every other project that doesn't
> >> use `flatten-plugin`.
> >
> > [1] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >
> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >> Nikolay,
> >>
> >>
> >> Can you describe your problem and desired behaviour more
> >> thoroughly, please?
> >>
> >>
> >>
> >>> On 28 Feb 2018, at 21:16, Nikolay Izhikov 
> >> wrote:
> >>>
> >>> Hello, Igniters.
> >>>
> >>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >>> As far as I can understand it minimize pom.xml before it
> >> installed in repository.
> >>>
> >>> It introduce some strange behavior in modules:
> >>>
> >>> I must to enlist all test dependencies in my module to get tests
> >> work.
> >>> Otherwise tests fails with ClassNoDefFoundError for transitive
> >> dependencies.
> >>> It happens in `spark` [2] and `spark_2.10` [3] modules.
> >>> Now, when I want to enable testing of Spark Examples I has to
> >> enlist same dependencies in `examples/pom.xml`
> >>>
> >>> It looks like a mess for me.
> >>>
> >>> Please, help me:
> >>>
> >>> 1. Am I miss something and can make pom.xml much clearer?
> >>>
> >>> 2. Why we need to minimize pom.xml? It looks like other Apache
> >> project doesn't do it [5].
> >>>
> >>> [1

Re: Maven. Issues with flatten plugin

[Anton Vinogradov]

 Nikolay,

Sure, I think we have to create IEP "Assembly and Release cleanup and
simplification".

I see some issues can be attached to this IEP
- Fabric word removal
- Flatten plugin removal (in possible)
- Ignite should be assemblied using package phase, not install in case you
want only to assembly it.
- Obsolete/unused/odd profiles removal
- scala 2.10 modules removal (if possible)
- Assembly simplification (get rid of ant scripts inside pom)
- Version update should be presented as sh script instead
of update-versions profile
- TeamCity repease procedure should be cleaned up (inc. RPM logic)
- etc.

2018-03-01 14:18 GMT+03:00 Petr Ivanov :

> May be it will be enough to add a profile to parent pom.xml which
> activates on deploy stage only.
>
> Yet, I still do not get a mechanism that causes a problem. What pom.xml
> becomes flattened, what is removed from it and how it effects other modules
> of the same project?
>
>
>
> > On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
> >
> > Andrey, thank you.
> >
> > If there are no other objections I will create ticket for pom structure
> > refactoring.
> >
> > Anton, can you help me with it as an Ignite Release Manager?
> >
> > 2018-03-01 13:45 GMT+03:00 Andrey Novikov :
> >
> >> Nikolay,
> >>
> >> I think it can be removed, if parent-pom will be installed(deployed).
> >>
> >> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov 
> >> wrote:
> >>> Hello, Andrey.
> >>>
> >>> Thanks for an answer.
> >>>
>  As I remember we use `flatten-maven-plugin` for flattening and
> removing
> >> parent relationship in deployed artifacts
> >>>
> >>> So we need it only in `release` profile?
> >>>
> >>> I found some earlier discussion about plugin [1]
> >>>
>  in first versions of build, version was stored in variable in parent
> pom
> >>>
> >>> Got it. Do we need this plugin now?
> >>>
> >>> [1] http://apache-ignite-developers.2346864.n4.nabble.
> >> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >>>
> >>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>  Nikolay,
> 
>  As I remember we use `flatten-maven-plugin` for flattening and
>  removing parent relationship in deployed artifacts (parent pom does
>  not deploy to repository and in first versions of build, version was
>  stored in variable in parent pom)
> 
>  On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov 
> >> wrote:
> > Hello, Petr.
> >
> >> Can you describe your problem
> >
> > In Ignite, maven doesn't include transitive dependencies to test
> >> classpath.
> > So we have enlist all dependencies in project pom.xml.
> >
> >> desired behaviour
> >
> > All I want is default maven behavior.
> >
> > Please, take a look at pom.xml [1] from line 144.
> > There is a long list of dependencies with `test` scope.
> > Actually, all of them are available as a transitive dependency from
> >> `spark-core`.
> >
> > We doesn't have to enlist them in every other project that doesn't
> >> use `flatten-plugin`.
> >
> > [1] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >
> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >> Nikolay,
> >>
> >>
> >> Can you describe your problem and desired behaviour more
> >> thoroughly, please?
> >>
> >>
> >>
> >>> On 28 Feb 2018, at 21:16, Nikolay Izhikov 
> >> wrote:
> >>>
> >>> Hello, Igniters.
> >>>
> >>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >>> As far as I can understand it minimize pom.xml before it
> >> installed in repository.
> >>>
> >>> It introduce some strange behavior in modules:
> >>>
> >>> I must to enlist all test dependencies in my module to get tests
> >> work.
> >>> Otherwise tests fails with ClassNoDefFoundError for transitive
> >> dependencies.
> >>> It happens in `spark` [2] and `spark_2.10` [3] modules.
> >>> Now, when I want to enable testing of Spark Examples I has to
> >> enlist same dependencies in `examples/pom.xml`
> >>>
> >>> It looks like a mess for me.
> >>>
> >>> Please, help me:
> >>>
> >>> 1. Am I miss something and can make pom.xml much clearer?
> >>>
> >>> 2. Why we need to minimize pom.xml? It looks like other Apache
> >> project doesn't do it [5].
> >>>
> >>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
> >> xml#L612
> >>> [2] https://github.com/apache/ignite/blob/master/modules/
> >> spark/pom.xml#L144
> >>> [3] https://github.com/apache/ignite/blob/master/modules/
> >> spark-2.10/pom.xml#L150
> >>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
> >> 08740066c64337d38cccd84991ac0912R155
> >>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
> >> 2.12/1.0.0/kafka_2.12-1.0.0.pom
> >>
> >>
> >>
>
>

Re: Maven. Issues with flatten plugin

[Petr Ivanov]

May be it will be enough to add a profile to parent pom.xml which activates on 
deploy stage only.

Yet, I still do not get a mechanism that causes a problem. What pom.xml becomes 
flattened, what is removed from it and how it effects other modules of the same 
project?



> On 1 Mar 2018, at 14:06, Nikolay Izhikov  wrote:
> 
> Andrey, thank you.
> 
> If there are no other objections I will create ticket for pom structure
> refactoring.
> 
> Anton, can you help me with it as an Ignite Release Manager?
> 
> 2018-03-01 13:45 GMT+03:00 Andrey Novikov :
> 
>> Nikolay,
>> 
>> I think it can be removed, if parent-pom will be installed(deployed).
>> 
>> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov 
>> wrote:
>>> Hello, Andrey.
>>> 
>>> Thanks for an answer.
>>> 
 As I remember we use `flatten-maven-plugin` for flattening and removing
>> parent relationship in deployed artifacts
>>> 
>>> So we need it only in `release` profile?
>>> 
>>> I found some earlier discussion about plugin [1]
>>> 
 in first versions of build, version was stored in variable in parent pom
>>> 
>>> Got it. Do we need this plugin now?
>>> 
>>> [1] http://apache-ignite-developers.2346864.n4.nabble.
>> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>>> 
>>> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
 Nikolay,
 
 As I remember we use `flatten-maven-plugin` for flattening and
 removing parent relationship in deployed artifacts (parent pom does
 not deploy to repository and in first versions of build, version was
 stored in variable in parent pom)
 
 On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov 
>> wrote:
> Hello, Petr.
> 
>> Can you describe your problem
> 
> In Ignite, maven doesn't include transitive dependencies to test
>> classpath.
> So we have enlist all dependencies in project pom.xml.
> 
>> desired behaviour
> 
> All I want is default maven behavior.
> 
> Please, take a look at pom.xml [1] from line 144.
> There is a long list of dependencies with `test` scope.
> Actually, all of them are available as a transitive dependency from
>> `spark-core`.
> 
> We doesn't have to enlist them in every other project that doesn't
>> use `flatten-plugin`.
> 
> [1] https://github.com/apache/ignite/blob/master/modules/
>> spark/pom.xml#L144
> 
> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> Nikolay,
>> 
>> 
>> Can you describe your problem and desired behaviour more
>> thoroughly, please?
>> 
>> 
>> 
>>> On 28 Feb 2018, at 21:16, Nikolay Izhikov 
>> wrote:
>>> 
>>> Hello, Igniters.
>>> 
>>> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>>> As far as I can understand it minimize pom.xml before it
>> installed in repository.
>>> 
>>> It introduce some strange behavior in modules:
>>> 
>>> I must to enlist all test dependencies in my module to get tests
>> work.
>>> Otherwise tests fails with ClassNoDefFoundError for transitive
>> dependencies.
>>> It happens in `spark` [2] and `spark_2.10` [3] modules.
>>> Now, when I want to enable testing of Spark Examples I has to
>> enlist same dependencies in `examples/pom.xml`
>>> 
>>> It looks like a mess for me.
>>> 
>>> Please, help me:
>>> 
>>> 1. Am I miss something and can make pom.xml much clearer?
>>> 
>>> 2. Why we need to minimize pom.xml? It looks like other Apache
>> project doesn't do it [5].
>>> 
>>> [1] https://github.com/apache/ignite/blob/master/parent/pom.
>> xml#L612
>>> [2] https://github.com/apache/ignite/blob/master/modules/
>> spark/pom.xml#L144
>>> [3] https://github.com/apache/ignite/blob/master/modules/
>> spark-2.10/pom.xml#L150
>>> [4] https://github.com/apache/ignite/pull/3590/files#diff-
>> 08740066c64337d38cccd84991ac0912R155
>>> [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
>> 2.12/1.0.0/kafka_2.12-1.0.0.pom
>> 
>> 
>> 

Re: Maven. Issues with flatten plugin

[Nikolay Izhikov]

Andrey, thank you.

If there are no other objections I will create ticket for pom structure
refactoring.

Anton, can you help me with it as an Ignite Release Manager?

2018-03-01 13:45 GMT+03:00 Andrey Novikov :

> Nikolay,
>
> I think it can be removed, if parent-pom will be installed(deployed).
>
> On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov 
> wrote:
> > Hello, Andrey.
> >
> > Thanks for an answer.
> >
> >> As I remember we use `flatten-maven-plugin` for flattening and removing
> parent relationship in deployed artifacts
> >
> > So we need it only in `release` profile?
> >
> > I found some earlier discussion about plugin [1]
> >
> >> in first versions of build, version was stored in variable in parent pom
> >
> > Got it. Do we need this plugin now?
> >
> > [1] http://apache-ignite-developers.2346864.n4.nabble.
> com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
> >
> > В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> >> Nikolay,
> >>
> >> As I remember we use `flatten-maven-plugin` for flattening and
> >> removing parent relationship in deployed artifacts (parent pom does
> >> not deploy to repository and in first versions of build, version was
> >> stored in variable in parent pom)
> >>
> >> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov 
> wrote:
> >> > Hello, Petr.
> >> >
> >> > > Can you describe your problem
> >> >
> >> > In Ignite, maven doesn't include transitive dependencies to test
> classpath.
> >> > So we have enlist all dependencies in project pom.xml.
> >> >
> >> > > desired behaviour
> >> >
> >> > All I want is default maven behavior.
> >> >
> >> > Please, take a look at pom.xml [1] from line 144.
> >> > There is a long list of dependencies with `test` scope.
> >> > Actually, all of them are available as a transitive dependency from
> `spark-core`.
> >> >
> >> > We doesn't have to enlist them in every other project that doesn't
> use `flatten-plugin`.
> >> >
> >> > [1] https://github.com/apache/ignite/blob/master/modules/
> spark/pom.xml#L144
> >> >
> >> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> >> > > Nikolay,
> >> > >
> >> > >
> >> > > Can you describe your problem and desired behaviour more
> thoroughly, please?
> >> > >
> >> > >
> >> > >
> >> > > > On 28 Feb 2018, at 21:16, Nikolay Izhikov 
> wrote:
> >> > > >
> >> > > > Hello, Igniters.
> >> > > >
> >> > > > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> >> > > > As far as I can understand it minimize pom.xml before it
> installed in repository.
> >> > > >
> >> > > > It introduce some strange behavior in modules:
> >> > > >
> >> > > > I must to enlist all test dependencies in my module to get tests
> work.
> >> > > > Otherwise tests fails with ClassNoDefFoundError for transitive
> dependencies.
> >> > > > It happens in `spark` [2] and `spark_2.10` [3] modules.
> >> > > > Now, when I want to enable testing of Spark Examples I has to
> enlist same dependencies in `examples/pom.xml`
> >> > > >
> >> > > > It looks like a mess for me.
> >> > > >
> >> > > > Please, help me:
> >> > > >
> >> > > > 1. Am I miss something and can make pom.xml much clearer?
> >> > > >
> >> > > > 2. Why we need to minimize pom.xml? It looks like other Apache
> project doesn't do it [5].
> >> > > >
> >> > > > [1] https://github.com/apache/ignite/blob/master/parent/pom.
> xml#L612
> >> > > > [2] https://github.com/apache/ignite/blob/master/modules/
> spark/pom.xml#L144
> >> > > > [3] https://github.com/apache/ignite/blob/master/modules/
> spark-2.10/pom.xml#L150
> >> > > > [4] https://github.com/apache/ignite/pull/3590/files#diff-
> 08740066c64337d38cccd84991ac0912R155
> >> > > > [5] http://central.maven.org/maven2/org/apache/kafka/kafka_
> 2.12/1.0.0/kafka_2.12-1.0.0.pom
> >> > >
> >> > >
>

Re: Maven. Issues with flatten plugin

[Andrey Novikov]

Nikolay,

I think it can be removed, if parent-pom will be installed(deployed).

On Thu, Mar 1, 2018 at 5:05 PM, Nikolay Izhikov  wrote:
> Hello, Andrey.
>
> Thanks for an answer.
>
>> As I remember we use `flatten-maven-plugin` for flattening and removing 
>> parent relationship in deployed artifacts
>
> So we need it only in `release` profile?
>
> I found some earlier discussion about plugin [1]
>
>> in first versions of build, version was stored in variable in parent pom
>
> Got it. Do we need this plugin now?
>
> [1] 
> http://apache-ignite-developers.2346864.n4.nabble.com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html
>
> В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
>> Nikolay,
>>
>> As I remember we use `flatten-maven-plugin` for flattening and
>> removing parent relationship in deployed artifacts (parent pom does
>> not deploy to repository and in first versions of build, version was
>> stored in variable in parent pom)
>>
>> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov  wrote:
>> > Hello, Petr.
>> >
>> > > Can you describe your problem
>> >
>> > In Ignite, maven doesn't include transitive dependencies to test classpath.
>> > So we have enlist all dependencies in project pom.xml.
>> >
>> > > desired behaviour
>> >
>> > All I want is default maven behavior.
>> >
>> > Please, take a look at pom.xml [1] from line 144.
>> > There is a long list of dependencies with `test` scope.
>> > Actually, all of them are available as a transitive dependency from 
>> > `spark-core`.
>> >
>> > We doesn't have to enlist them in every other project that doesn't use 
>> > `flatten-plugin`.
>> >
>> > [1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>> >
>> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> > > Nikolay,
>> > >
>> > >
>> > > Can you describe your problem and desired behaviour more thoroughly, 
>> > > please?
>> > >
>> > >
>> > >
>> > > > On 28 Feb 2018, at 21:16, Nikolay Izhikov  wrote:
>> > > >
>> > > > Hello, Igniters.
>> > > >
>> > > > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> > > > As far as I can understand it minimize pom.xml before it installed in 
>> > > > repository.
>> > > >
>> > > > It introduce some strange behavior in modules:
>> > > >
>> > > > I must to enlist all test dependencies in my module to get tests work.
>> > > > Otherwise tests fails with ClassNoDefFoundError for transitive 
>> > > > dependencies.
>> > > > It happens in `spark` [2] and `spark_2.10` [3] modules.
>> > > > Now, when I want to enable testing of Spark Examples I has to enlist 
>> > > > same dependencies in `examples/pom.xml`
>> > > >
>> > > > It looks like a mess for me.
>> > > >
>> > > > Please, help me:
>> > > >
>> > > > 1. Am I miss something and can make pom.xml much clearer?
>> > > >
>> > > > 2. Why we need to minimize pom.xml? It looks like other Apache project 
>> > > > doesn't do it [5].
>> > > >
>> > > > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
>> > > > [2] 
>> > > > https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>> > > > [3] 
>> > > > https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
>> > > > [4] 
>> > > > https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
>> > > > [5] 
>> > > > http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
>> > >
>> > >

Re: Maven. Issues with flatten plugin

[Nikolay Izhikov]

Hello, Andrey.

Thanks for an answer.

> As I remember we use `flatten-maven-plugin` for flattening and removing 
> parent relationship in deployed artifacts

So we need it only in `release` profile?

I found some earlier discussion about plugin [1]

> in first versions of build, version was stored in variable in parent pom

Got it. Do we need this plugin now?

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Difference-between-pom-xml-and-pom-installed-xml-td2171.html

В Чт, 01/03/2018 в 16:59 +0700, Andrey Novikov пишет:
> Nikolay,
> 
> As I remember we use `flatten-maven-plugin` for flattening and
> removing parent relationship in deployed artifacts (parent pom does
> not deploy to repository and in first versions of build, version was
> stored in variable in parent pom)
> 
> On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov  wrote:
> > Hello, Petr.
> > 
> > > Can you describe your problem
> > 
> > In Ignite, maven doesn't include transitive dependencies to test classpath.
> > So we have enlist all dependencies in project pom.xml.
> > 
> > > desired behaviour
> > 
> > All I want is default maven behavior.
> > 
> > Please, take a look at pom.xml [1] from line 144.
> > There is a long list of dependencies with `test` scope.
> > Actually, all of them are available as a transitive dependency from 
> > `spark-core`.
> > 
> > We doesn't have to enlist them in every other project that doesn't use 
> > `flatten-plugin`.
> > 
> > [1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> > 
> > В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> > > Nikolay,
> > > 
> > > 
> > > Can you describe your problem and desired behaviour more thoroughly, 
> > > please?
> > > 
> > > 
> > > 
> > > > On 28 Feb 2018, at 21:16, Nikolay Izhikov  wrote:
> > > > 
> > > > Hello, Igniters.
> > > > 
> > > > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> > > > As far as I can understand it minimize pom.xml before it installed in 
> > > > repository.
> > > > 
> > > > It introduce some strange behavior in modules:
> > > > 
> > > > I must to enlist all test dependencies in my module to get tests work.
> > > > Otherwise tests fails with ClassNoDefFoundError for transitive 
> > > > dependencies.
> > > > It happens in `spark` [2] and `spark_2.10` [3] modules.
> > > > Now, when I want to enable testing of Spark Examples I has to enlist 
> > > > same dependencies in `examples/pom.xml`
> > > > 
> > > > It looks like a mess for me.
> > > > 
> > > > Please, help me:
> > > > 
> > > > 1. Am I miss something and can make pom.xml much clearer?
> > > > 
> > > > 2. Why we need to minimize pom.xml? It looks like other Apache project 
> > > > doesn't do it [5].
> > > > 
> > > > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
> > > > [2] 
> > > > https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> > > > [3] 
> > > > https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
> > > > [4] 
> > > > https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
> > > > [5] 
> > > > http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
> > > 
> > > 

signature.asc
Description: This is a digitally signed message part

Re: Maven. Issues with flatten plugin

[Andrey Novikov]

Nikolay,

As I remember we use `flatten-maven-plugin` for flattening and
removing parent relationship in deployed artifacts (parent pom does
not deploy to repository and in first versions of build, version was
stored in variable in parent pom)

On Thu, Mar 1, 2018 at 4:57 PM, Nikolay Izhikov  wrote:
> Hello, Petr.
>
>> Can you describe your problem
>
> In Ignite, maven doesn't include transitive dependencies to test classpath.
> So we have enlist all dependencies in project pom.xml.
>
>> desired behaviour
>
> All I want is default maven behavior.
>
> Please, take a look at pom.xml [1] from line 144.
> There is a long list of dependencies with `test` scope.
> Actually, all of them are available as a transitive dependency from 
> `spark-core`.
>
> We doesn't have to enlist them in every other project that doesn't use 
> `flatten-plugin`.
>
> [1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>
> В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
>> Nikolay,
>>
>>
>> Can you describe your problem and desired behaviour more thoroughly, please?
>>
>>
>>
>> > On 28 Feb 2018, at 21:16, Nikolay Izhikov  wrote:
>> >
>> > Hello, Igniters.
>> >
>> > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
>> > As far as I can understand it minimize pom.xml before it installed in 
>> > repository.
>> >
>> > It introduce some strange behavior in modules:
>> >
>> > I must to enlist all test dependencies in my module to get tests work.
>> > Otherwise tests fails with ClassNoDefFoundError for transitive 
>> > dependencies.
>> > It happens in `spark` [2] and `spark_2.10` [3] modules.
>> > Now, when I want to enable testing of Spark Examples I has to enlist same 
>> > dependencies in `examples/pom.xml`
>> >
>> > It looks like a mess for me.
>> >
>> > Please, help me:
>> >
>> > 1. Am I miss something and can make pom.xml much clearer?
>> >
>> > 2. Why we need to minimize pom.xml? It looks like other Apache project 
>> > doesn't do it [5].
>> >
>> > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
>> > [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
>> > [3] 
>> > https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
>> > [4] 
>> > https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
>> > [5] 
>> > http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
>>
>>

Re: Maven. Issues with flatten plugin

[Nikolay Izhikov]

Hello, Petr.

> Can you describe your problem

In Ignite, maven doesn't include transitive dependencies to test classpath.
So we have enlist all dependencies in project pom.xml.

> desired behaviour

All I want is default maven behavior.

Please, take a look at pom.xml [1] from line 144.
There is a long list of dependencies with `test` scope.
Actually, all of them are available as a transitive dependency from 
`spark-core`.

We doesn't have to enlist them in every other project that doesn't use 
`flatten-plugin`.

[1] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144

В Чт, 01/03/2018 в 11:49 +0300, Petr Ivanov пишет:
> Nikolay,
> 
> 
> Can you describe your problem and desired behaviour more thoroughly, please?
> 
> 
> 
> > On 28 Feb 2018, at 21:16, Nikolay Izhikov  wrote:
> > 
> > Hello, Igniters.
> > 
> > We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> > As far as I can understand it minimize pom.xml before it installed in 
> > repository.
> > 
> > It introduce some strange behavior in modules:
> > 
> > I must to enlist all test dependencies in my module to get tests work.
> > Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
> > It happens in `spark` [2] and `spark_2.10` [3] modules.
> > Now, when I want to enable testing of Spark Examples I has to enlist same 
> > dependencies in `examples/pom.xml`
> > 
> > It looks like a mess for me.
> > 
> > Please, help me:
> > 
> > 1. Am I miss something and can make pom.xml much clearer?
> > 
> > 2. Why we need to minimize pom.xml? It looks like other Apache project 
> > doesn't do it [5].
> > 
> > [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
> > [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> > [3] 
> > https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150
> >  
> > [4] 
> > https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
> > [5] 
> > http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom
> 
> 

signature.asc
Description: This is a digitally signed message part

Re: Nodes which started in separate JVM couldn't stop properly (in tests)

[Vyacheslav Daradur]

Thank you, Dmitry!

I'll join this review soon.

On Thu, Mar 1, 2018 at 12:07 PM, Dmitry Pavlov  wrote:
> Hi Vyacheslav,
>
> I will take a look, but first of all I am going to review
> https://reviews.ignite.apache.org/ignite/review/IGNT-CR-502  - it is impact
> change in testing framework. Hope you also will join to this review .
>
> Sincerely,
> Dmitiry Pavlov
>
>
> чт, 1 мар. 2018 г. в 11:13, Vyacheslav Daradur :
>>
>> Hi, Dmitry, could you please review it, because you are one of the
>> most experienced people in the testing framework.
>>
>> Please see comment in Jira, because it is in pretty-format there.
>>
>> On Thu, Feb 22, 2018 at 11:56 AM, Vyacheslav Daradur
>>  wrote:
>> > Hi Igniters!
>> >
>> > I have investigated the issue [1] and found that stopping node in
>> > separate JVM may stuck thread or leave system process alive after test
>> > finished.
>> > The main reason is *StopGridTask* that we send from node in local JVM
>> > to node in separate JVM via remote computing.
>> > We send job synchronously to be sure that node will be stopped, but
>> > job calls synchronously *G.stop(igniteInstanceName, cancel))* with
>> > *cancel = false*, that means node must wait to compute jobs before it
>> > goes down what leads to some kind of deadlock. Using of *cancel =
>> > true* would solve the issue but may break some tests’ logic, for this
>> > reason, I've reworked the method’s synchronization logic [2].
>> >
>> > We have not noticed that before because we use only *stopAllGrids()*
>> > in out tests which stop local JVM without waiting for nodes in other
>> > JVMs.
>> > I believe this fix should reduce the number of flaky tests on
>> > TeamCity, especially which fails because of a cluster from the
>> > previous test has not been stopped properly.
>> >
>> > Ci.tests [3] look a bit better than in master.
>> > Please review prepared PR [2] and share your thoughts.
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-5910
>> > [2] https://github.com/apache/ignite/pull/2382
>> > [3] https://ci.ignite.apache.org/viewLog.html?buildId=1105939
>> >
>> >
>> > On Fri, Aug 4, 2017 at 11:41 AM, Vyacheslav Daradur
>> >  wrote:
>> >> Hi Igniters,
>> >>
>> >> Working on my task I found a bug at call the method #stopGrid(name),
>> >> it produced ClassCastException. I created a ticket[1].
>> >>
>> >> After it was fixed[2] I saw that nodes which was started in a separate
>> >> JVM
>> >> could stay in process of operation system.
>> >> It was fixed too, but not sure is it fixed in proper way or not.
>> >>
>> >> Could someone review it?
>> >>
>> >> [1] https://issues.apache.org/jira/browse/IGNITE-5910
>> >> [2] https://github.com/apache/ignite/pull/2382
>> >>
>> >> --
>> >> Best Regards, Vyacheslav D.
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.

Re: Nodes which started in separate JVM couldn't stop properly (in tests)

[Dmitry Pavlov]

Hi Vyacheslav,

I will take a look, but first of all I am going to review
https://reviews.ignite.apache.org/ignite/review/IGNT-CR-502  - it is impact
change in testing framework. Hope you also will join to this review .

Sincerely,
Dmitiry Pavlov


чт, 1 мар. 2018 г. в 11:13, Vyacheslav Daradur :

> Hi, Dmitry, could you please review it, because you are one of the
> most experienced people in the testing framework.
>
> Please see comment in Jira, because it is in pretty-format there.
>
> On Thu, Feb 22, 2018 at 11:56 AM, Vyacheslav Daradur
>  wrote:
> > Hi Igniters!
> >
> > I have investigated the issue [1] and found that stopping node in
> > separate JVM may stuck thread or leave system process alive after test
> > finished.
> > The main reason is *StopGridTask* that we send from node in local JVM
> > to node in separate JVM via remote computing.
> > We send job synchronously to be sure that node will be stopped, but
> > job calls synchronously *G.stop(igniteInstanceName, cancel))* with
> > *cancel = false*, that means node must wait to compute jobs before it
> > goes down what leads to some kind of deadlock. Using of *cancel =
> > true* would solve the issue but may break some tests’ logic, for this
> > reason, I've reworked the method’s synchronization logic [2].
> >
> > We have not noticed that before because we use only *stopAllGrids()*
> > in out tests which stop local JVM without waiting for nodes in other
> > JVMs.
> > I believe this fix should reduce the number of flaky tests on
> > TeamCity, especially which fails because of a cluster from the
> > previous test has not been stopped properly.
> >
> > Ci.tests [3] look a bit better than in master.
> > Please review prepared PR [2] and share your thoughts.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5910
> > [2] https://github.com/apache/ignite/pull/2382
> > [3] https://ci.ignite.apache.org/viewLog.html?buildId=1105939
> >
> >
> > On Fri, Aug 4, 2017 at 11:41 AM, Vyacheslav Daradur 
> wrote:
> >> Hi Igniters,
> >>
> >> Working on my task I found a bug at call the method #stopGrid(name),
> >> it produced ClassCastException. I created a ticket[1].
> >>
> >> After it was fixed[2] I saw that nodes which was started in a separate
> JVM
> >> could stay in process of operation system.
> >> It was fixed too, but not sure is it fixed in proper way or not.
> >>
> >> Could someone review it?
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-5910
> >> [2] https://github.com/apache/ignite/pull/2382
> >>
> >> --
> >> Best Regards, Vyacheslav D.
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Maven. Issues with flatten plugin

[Petr Ivanov]

Nikolay,


Can you describe your problem and desired behaviour more thoroughly, please?



> On 28 Feb 2018, at 21:16, Nikolay Izhikov  wrote:
> 
> Hello, Igniters.
> 
> We have `flatten-maven-plugin` in `parent/pom.xml` [1]
> As far as I can understand it minimize pom.xml before it installed in 
> repository.
> 
> It introduce some strange behavior in modules:
> 
> I must to enlist all test dependencies in my module to get tests work.
> Otherwise tests fails with ClassNoDefFoundError for transitive dependencies.
> It happens in `spark` [2] and `spark_2.10` [3] modules.
> Now, when I want to enable testing of Spark Examples I has to enlist same 
> dependencies in `examples/pom.xml`
> 
> It looks like a mess for me.
> 
> Please, help me:
> 
> 1. Am I miss something and can make pom.xml much clearer?
> 
> 2. Why we need to minimize pom.xml? It looks like other Apache project 
> doesn't do it [5].
> 
> [1] https://github.com/apache/ignite/blob/master/parent/pom.xml#L612
> [2] https://github.com/apache/ignite/blob/master/modules/spark/pom.xml#L144
> [3] 
> https://github.com/apache/ignite/blob/master/modules/spark-2.10/pom.xml#L150 
> [4] 
> https://github.com/apache/ignite/pull/3590/files#diff-08740066c64337d38cccd84991ac0912R155
> [5] 
> http://central.maven.org/maven2/org/apache/kafka/kafka_2.12/1.0.0/kafka_2.12-1.0.0.pom

[GitHub] ignite pull request #3451: IGNITE-7366 Service reassignment with merge excha...

[xtern]

GitHub user xtern reopened a pull request:

https://github.com/apache/ignite/pull/3451

IGNITE-7366 Service reassignment with merge exchanges.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xtern/ignite IGNITE-7366

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3451


commit 2398cf3a5c9a91eaf15336cd5699a2f83e588bbc
Author: pereslegin-pa 
Date:   2018-03-01T07:56:09Z

IGNITE-7366 Fixed ready topology await on merge exchanges.




---

[GitHub] ignite pull request #3451: Service reassign with merge exchanges (TC run).

[xtern]

Github user xtern closed the pull request at:

https://github.com/apache/ignite/pull/3451


---

Re: Nodes which started in separate JVM couldn't stop properly (in tests)

[Vyacheslav Daradur]

Hi, Dmitry, could you please review it, because you are one of the
most experienced people in the testing framework.

Please see comment in Jira, because it is in pretty-format there.

On Thu, Feb 22, 2018 at 11:56 AM, Vyacheslav Daradur
 wrote:
> Hi Igniters!
>
> I have investigated the issue [1] and found that stopping node in
> separate JVM may stuck thread or leave system process alive after test
> finished.
> The main reason is *StopGridTask* that we send from node in local JVM
> to node in separate JVM via remote computing.
> We send job synchronously to be sure that node will be stopped, but
> job calls synchronously *G.stop(igniteInstanceName, cancel))* with
> *cancel = false*, that means node must wait to compute jobs before it
> goes down what leads to some kind of deadlock. Using of *cancel =
> true* would solve the issue but may break some tests’ logic, for this
> reason, I've reworked the method’s synchronization logic [2].
>
> We have not noticed that before because we use only *stopAllGrids()*
> in out tests which stop local JVM without waiting for nodes in other
> JVMs.
> I believe this fix should reduce the number of flaky tests on
> TeamCity, especially which fails because of a cluster from the
> previous test has not been stopped properly.
>
> Ci.tests [3] look a bit better than in master.
> Please review prepared PR [2] and share your thoughts.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5910
> [2] https://github.com/apache/ignite/pull/2382
> [3] https://ci.ignite.apache.org/viewLog.html?buildId=1105939
>
>
> On Fri, Aug 4, 2017 at 11:41 AM, Vyacheslav Daradur  
> wrote:
>> Hi Igniters,
>>
>> Working on my task I found a bug at call the method #stopGrid(name),
>> it produced ClassCastException. I created a ticket[1].
>>
>> After it was fixed[2] I saw that nodes which was started in a separate JVM
>> could stay in process of operation system.
>> It was fixed too, but not sure is it fixed in proper way or not.
>>
>> Could someone review it?
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-5910
>> [2] https://github.com/apache/ignite/pull/2382
>>
>> --
>> Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.