Re: Migration to JUnit 5
Ivan, Indeed junit5 has a lot of powerful features which can improve testing process. But first we should go through a migration process. There are several items which looks quite challenging. 1. Test suites support. Correct me if I am missed it, but I have not found a concept of test suites similar to junit3/4 ones. CI in Ignite heavily depends on test suites. Is there an alternative in junit5? 2. The majority of our tests extend GridAbstractTest which in fact is a core class in Ignite testing. Writing a test without extending it is not a good idea. Currently it employs number of junit4 concepts, e.g. test rules which as I saw are not supported in junit5. So, it sounds that some changes in GridAbstractTest need to be done. During migration from junit 3 to 4 GridAbstractTest used kind of mimicry, it can be used as a base class for junit3 and junit4 tests at the same time. How can we address transitional period now? 3. Also we have bunch of tests using our home-brewed parametrization. You can find them by searching usages of ConfigVariationsTestSuiteBuilder. This part was rather tricky during migration to junit4. Do we have a plan for all these items? Couple of side notes. Regarding dependencies minimization. Actually, I think it is important for junit itself as a library. Many libraries try to minimize dependency. In Ignite we do so as well. But in my opinion it is not the case in context of libraries used during testing. If we have useful, stable and widely-used test library which can improve our processes why should not we use it? Regarding removing leftovers left after junit 3->4 migration. Actually, I think that GridAbstractTest and GridCommonAbstractTest can be refactored in order to simplify further development and migration to new testing framework. For example, there are a lot of instance methods which can be turned to static methods. Various start/stopGrid methods fall into this category. They can be extracted into some utility class and imported statically. Perhaps, after number of such refactoring we will be able to write tests without extending GridCommonAbstractTest. пт, 22 февр. 2019 г. в 18:33, Ivan Fedotov : > > Hi Ivan! > > Junit5 differs from JUnit4 not so strong as 4 from 3 version. Of course, > we can use AssertJ and other libraries, but it is more comfortable to > use functionality from the box. Moreover, the JUnit team provides strong > support for its products and it is the core JUnit principle - minimize > third-party dependency [1]. > > According to Parameterized tests, it has several cons in JUnit4: > 1. Test classes use fields and constructors to define parameters, which > make tests more verbose > 2. It requires a separate test class for each method being tested. > In JUnit5 it has a simplified parameter syntax and supports multiple > data-set source types, including CSV and annotation > > Impact on daily test development does not so differ from development on > JUnit4. We also can use annotations to mark methods as tests, but some main > annotations have > different names - you can see it in the ticket description [2]. You have to > use those annotations and different import, but these are minor changes. > We can change suites from static to dynamic tests [3], but I am not sure > that it is necessary. If you have any arguments in favor of dynamic tests, > I am ready to discuss them. > > Now I see that changes in GridAbstractTest are not required. Only > improvements in JUnit 3->4 migration, which were given in IEP. Other JUnit5 > features we can use with additional imports. The problem can appear with > dynamic tests because we can not launch static and dynamic under one > surefire version. I made a preliminary migration on examples module, you > can take a look on it [4], but now it is still in work. > > I tried to find some other JUnit5 features and added them to IEP. If I miss > something, please, let me now, we will also take it into account. > > [1] https://github.com/junit-team/junit5/wiki/Core-Principles > [2] https://issues.apache.org/jira/browse/IGNITE-10958 > [3] https://www.baeldung.com/junit5-dynamic-tests > [4] https://github.com/apache/ignite/pull/5888 > > > чт, 21 февр. 2019 г. в 18:45, Павлухин Иван : > > > Hi Ivan, > > > > Thank you for your efforts! > > > > I checked a section "Motivation" in IEP and I think that we should add > > more details there. You provided mostly examples of more convenient > > assertions. But there are other options to deal with it. E.g. AssertJ > > library [1] (I think that we can consider it even after migration to > > junit5). It would be great if we can describe some junit5 features > > which can make our life simpler and there is no alternative in junit4. > > E.g. we have the similar Parameterized concept in junit4, so it does > > not look as a big win here. > > > > Also, an impact on everyday development should be estimated. As I > > know, junit5 has a compatibility layer which allows to migrate from > >
[jira] [Created] (IGNITE-11398) Remove leftover @RunWith(JUnit4.class)
Ivan Pavlukhin created IGNITE-11398: --- Summary: Remove leftover @RunWith(JUnit4.class) Key: IGNITE-11398 URL: https://issues.apache.org/jira/browse/IGNITE-11398 Project: Ignite Issue Type: Task Reporter: Ivan Pavlukhin During migration to junit4 transitional construction `@RunWith(JUnit4.class)` was used. And it was forgotten in couple of places. Let's clean up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: SQL: INSERT with hidden columns _key, _val and check the type of input objects
Vladimir, Ok, agreed, let's not boil the ocean...at least for now ;) -- Denis Magda On Sat, Feb 23, 2019 at 12:50 AM Vladimir Ozerov wrote: > Denis, > > Yes, this is what my answer was about - you cannot have SQL without > defining fields in advance. Because it breaks a lot of standard SQL > invariants and virtually makes the whole language unusable. For instance, > think of product behavior in the following cases: > 1) User queries an empty cache with a query "SELECT a FROM table" - what > should happen - exception or empty result? How would I know whether field > "a" will appear in future? > 2) User executed a command "ALTER TABLE ... ADD COLUMN b" - how can I > understand whether it is possible or not to add a column without strict > schema? > 3) "ALTER TABLE ... DROP COLUMN c" - what should happen if user will add an > object with field "c" after that? > 4) User connects to Ignite from Tableau and navigates through schema - what > should be shown? > > That is, you cannot have SQL without schema because it is at the very heart > of the technology. But you can have schema-less noSQL database. > > Let's do not invent a hybrid with tons of corner cases and separate > learning curve. It should be enough just to rethink and simplify our > configuration - reshape QueryEntity, deprecate all SQL annotations, allow > only one table per cache, allow to define SQL script to be executed on > cache start or so. > > As far as schemaless - it is viable approach for sure, but should be > considered either outside of SQL (e.g. a kind of predicate/criteria API > which can be merged with ScanQuery) or as a special datatype in SQL > ecosystem (like is is done with JSON in many RDBMS databases). > > Vladimir. > > > > > On Fri, Feb 22, 2019 at 11:01 PM Denis Magda wrote: > > > Vladimir, > > > > That's understood. I'm just thinking of a use case different from the DDL > > approach where the schema is defined initially. Let's say that someone > > configured caches with CacheConfiguration and now puts an Object in the > > cache. For that person, it would be helpful to skip the Annotations or > > QueryEntities approaches for queryable fields definitions (not even > > indexes). For instance, the person might simply query some fields with > the > > primary index in the WHERE clause and this shouldn't require any extra > > settings. Yes, it's clear that it might be extremely challenging to > support > > but imagine how usable the API could become if we can get rid of > > Annotations and QueryEntities. > > > > Basically, my idea is that all of the objects and their fields stored in > > the caches should be visible to SQL w/o extra settings. If someone wants > to > > create indexes then use DDL which was designed for this. > > > > > > - > > Denis > > > > > > On Fri, Feb 22, 2019 at 2:27 AM Vladimir Ozerov > > wrote: > > > > > Denis, > > > > > > SQL is a language with strict schema what was one of significant > factors > > of > > > it's worldwide success. I doubt we will ever have SQL without > > > configuration/definiton, because otherwise it will be not SQL, but > > > something else (e.g. document-oriented, JSON, whatever). > > > > > > On Fri, Feb 22, 2019 at 1:52 AM Denis Magda wrote: > > > > > > > Folks, > > > > > > > > Do we want to preserve the annotation-based configuration? There are > > too > > > > many ways to configure SQL indexes/fields. > > > > > > > > For instance, if our new SQL API could see and access all of the > fields > > > > out-of-the-box (without any extra settings) and DDL will be used to > > > define > > > > indexed fields then that would be a huge usability improvement. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Thu, Feb 21, 2019 at 5:27 AM Taras Ledkov > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > Lets discuss SQL DML (INSERT/UPDATE) current behavior specific: > > > > > > > > > > Ignite doesn't check a type of input objects when hidden columns > > _key, > > > > > _value is used in a DML statements. > > > > > I describe the current behavior for example: > > > > > > > > > > 1. Cache configuration: 'setIndexedTypes(PersonKey.class, > > > > Person.class))' > > > > > 2. PersonKey type contains 'int id' field. > > > > > 3. SQL statement: 'INSERT INTO test (_val, _key) VALUES (?, ?)' > > > > > > > > > > Cases: > > > > > 1. Invalid value object type: > > > > > - Any value object may be passed as a query parameter > > > > > - Query is executed without an error and returns '1' (one row > > updated); > > > > > - There is not inserted row at the 'SELECT * FROM test' results. > > > > > - cache.get(key) returns inserted object; > > > > > > > > > > 2. Invalid key object type: > > > > > 2.1 Non-primitive object is passed and binary representation > doesn't > > > > > contain 'id' field. > > > > > - Query is executed without error and returns '1' (one row > updated); > > > > > - The inserted row is available by 'SELECT *' and the row contains > > id = > > > > > null; > > > > > 2.2
Re: Help Wanted: Java 11 support for Camel & Cassandra integrations
Igor, would you be able to help with the Cassandra support? -- Denis Magda On Sat, Feb 23, 2019 at 1:03 AM Roman Shtykh wrote: > Dmitriy, I will have a look at the Camel Streamer. > > -- Roman > > > On Tuesday, February 19, 2019, 2:26:26 a.m. GMT+9, Dmitriy Pavlov < > dpav...@apache.org> wrote: > > Hi Igniters, > > I've noticed we have a couple more Java11 related failures: Tests here > didn't start at all, so assistance here for finding out > - reasons and > - how to fix it > are appreciated. > > https://issues.apache.org/jira/browse/IGNITE-11190 Fix Apache Ignite > tests > of Camel Streamer under Java 11 > - failed to receive a response from the streamer. > > https://issues.apache.org/jira/browse/IGNITE-11273 > IgniteCassandraStoreTestSuite failed under Java 11 > - failed to start Cassandra instance. > > Sincerely, > Dmitriy Pavlov >
[jira] [Created] (IGNITE-11397) Binary mode for Ignite Queue
Uday Kale created IGNITE-11397: -- Summary: Binary mode for Ignite Queue Key: IGNITE-11397 URL: https://issues.apache.org/jira/browse/IGNITE-11397 Project: Ignite Issue Type: New Feature Components: binary, data structures Reporter: Uday Kale Assignee: Uday Kale Fix For: 2.8 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Help Wanted: Java 11 support for Camel & Cassandra integrations
Dmitriy, I will have a look at the Camel Streamer. -- Roman On Tuesday, February 19, 2019, 2:26:26 a.m. GMT+9, Dmitriy Pavlov wrote: Hi Igniters, I've noticed we have a couple more Java11 related failures: Tests here didn't start at all, so assistance here for finding out - reasons and - how to fix it are appreciated. https://issues.apache.org/jira/browse/IGNITE-11190 Fix Apache Ignite tests of Camel Streamer under Java 11 - failed to receive a response from the streamer. https://issues.apache.org/jira/browse/IGNITE-11273 IgniteCassandraStoreTestSuite failed under Java 11 - failed to start Cassandra instance. Sincerely, Dmitriy Pavlov
Re: SQL: INSERT with hidden columns _key, _val and check the type of input objects
Denis, Yes, this is what my answer was about - you cannot have SQL without defining fields in advance. Because it breaks a lot of standard SQL invariants and virtually makes the whole language unusable. For instance, think of product behavior in the following cases: 1) User queries an empty cache with a query "SELECT a FROM table" - what should happen - exception or empty result? How would I know whether field "a" will appear in future? 2) User executed a command "ALTER TABLE ... ADD COLUMN b" - how can I understand whether it is possible or not to add a column without strict schema? 3) "ALTER TABLE ... DROP COLUMN c" - what should happen if user will add an object with field "c" after that? 4) User connects to Ignite from Tableau and navigates through schema - what should be shown? That is, you cannot have SQL without schema because it is at the very heart of the technology. But you can have schema-less noSQL database. Let's do not invent a hybrid with tons of corner cases and separate learning curve. It should be enough just to rethink and simplify our configuration - reshape QueryEntity, deprecate all SQL annotations, allow only one table per cache, allow to define SQL script to be executed on cache start or so. As far as schemaless - it is viable approach for sure, but should be considered either outside of SQL (e.g. a kind of predicate/criteria API which can be merged with ScanQuery) or as a special datatype in SQL ecosystem (like is is done with JSON in many RDBMS databases). Vladimir. On Fri, Feb 22, 2019 at 11:01 PM Denis Magda wrote: > Vladimir, > > That's understood. I'm just thinking of a use case different from the DDL > approach where the schema is defined initially. Let's say that someone > configured caches with CacheConfiguration and now puts an Object in the > cache. For that person, it would be helpful to skip the Annotations or > QueryEntities approaches for queryable fields definitions (not even > indexes). For instance, the person might simply query some fields with the > primary index in the WHERE clause and this shouldn't require any extra > settings. Yes, it's clear that it might be extremely challenging to support > but imagine how usable the API could become if we can get rid of > Annotations and QueryEntities. > > Basically, my idea is that all of the objects and their fields stored in > the caches should be visible to SQL w/o extra settings. If someone wants to > create indexes then use DDL which was designed for this. > > > - > Denis > > > On Fri, Feb 22, 2019 at 2:27 AM Vladimir Ozerov > wrote: > > > Denis, > > > > SQL is a language with strict schema what was one of significant factors > of > > it's worldwide success. I doubt we will ever have SQL without > > configuration/definiton, because otherwise it will be not SQL, but > > something else (e.g. document-oriented, JSON, whatever). > > > > On Fri, Feb 22, 2019 at 1:52 AM Denis Magda wrote: > > > > > Folks, > > > > > > Do we want to preserve the annotation-based configuration? There are > too > > > many ways to configure SQL indexes/fields. > > > > > > For instance, if our new SQL API could see and access all of the fields > > > out-of-the-box (without any extra settings) and DDL will be used to > > define > > > indexed fields then that would be a huge usability improvement. > > > > > > - > > > Denis > > > > > > > > > On Thu, Feb 21, 2019 at 5:27 AM Taras Ledkov > > wrote: > > > > > > > Hi, > > > > > > > > Lets discuss SQL DML (INSERT/UPDATE) current behavior specific: > > > > > > > > Ignite doesn't check a type of input objects when hidden columns > _key, > > > > _value is used in a DML statements. > > > > I describe the current behavior for example: > > > > > > > > 1. Cache configuration: 'setIndexedTypes(PersonKey.class, > > > Person.class))' > > > > 2. PersonKey type contains 'int id' field. > > > > 3. SQL statement: 'INSERT INTO test (_val, _key) VALUES (?, ?)' > > > > > > > > Cases: > > > > 1. Invalid value object type: > > > > - Any value object may be passed as a query parameter > > > > - Query is executed without an error and returns '1' (one row > updated); > > > > - There is not inserted row at the 'SELECT * FROM test' results. > > > > - cache.get(key) returns inserted object; > > > > > > > > 2. Invalid key object type: > > > > 2.1 Non-primitive object is passed and binary representation doesn't > > > > contain 'id' field. > > > > - Query is executed without error and returns '1' (one row updated); > > > > - The inserted row is available by 'SELECT *' and the row contains > id = > > > > null; > > > > 2.2 Non-primitive object is passed and binary representation contains > > > > 'id' field. > > > > - The inserted row is available by 'SELECT *' and the row contains > > > > expected 'id' field; > > > > - The cache entry cannot be gathered by 'cache.get' operation with > the > > > > corresponding 'PersonKey(id)' (keys differ). > > > > > > > > I propose to check type of the user's input object. > > > >
Re: [DISCUSSION] Service Grid Roadmap
Denis, thank you for ranging. I am working on service hot-redeployment via DeploymentSpi [1]. This allows us using the feature without changing public API. [1] https://issues.apache.org/jira/browse/IGNITE-11384 On Thu, Feb 7, 2019 at 11:53 PM Denis Magda wrote: > > Vyacheslav, > > Such tasks aggregation makes sense for me. If to range by priority I would > focus on hot redeployment, metrics and failover. > > > - > Denis > > > On Thu, Feb 7, 2019 at 12:48 PM Vyacheslav Daradur > wrote: > > > Igniters, > > > > While working on the redesign of the Service Grid component I've > > looked at the component as end-user and I've faced with usability > > issues. > > > > I'd suggest updating IEP-17 [1] or create a new one to specify streams > > of improvements: > > > > --- Features > > - IGNITE-8367 Implement hot redeployment of services [2] > > - IGNITE-6629 Make service automatic redeployment configurable in > > ServiceConfiguration > > - IGNITE- Service versioning > > > > --- Failovers: > > - IGNITE-8365 Introduce service failure events > > - IGNITE-4235 Can't get user exception if was on remote service (also, > > Service Grid should handle errors on services #execute and #cancel) > > - IGNITE-5560 A failed service must be redeployed when possible > > > > --- Usability increasing: > > - IGNITE- Thin client for Service Grid > > - IGNITE- Provide ability to manage deployment using script e.g. > > “control.sh” > > > > --- Monitoring: > > - IGNITE- Introduce some kind of services metrics > > - IGNITE- Monitoring tools > > > > --- Performance: > > - IGNITE- Improve load-balancing [3] > > > > I believe these streams are basic to do Service Grid even better for > > our end-users. Work within the streams may be paralleled as Service > > Grid redesign phase 1 has been merged. > > > > Thoughts? > > > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-17%3A+Oil+Change+in+Service+Grid > > [2] > > http://apache-ignite-developers.2346864.n4.nabble.com/Services-hot-redeployment-td40562.html > > [3] > > http://apache-ignite-developers.2346864.n4.nabble.com/IgniteServices-serviceProxy-and-local-services-td40608.html > > > > -- > > Best Regards, Vyacheslav D. > > -- Best Regards, Vyacheslav D.
Re: [Discussion] Persistence compatibility framework refactoring
Pavel,
>> A new version of Ignite has released and a developer should update
>> compatibility tests to run it against the new version.
Now I think that understand a problem which you are trying to solve,
but I'd suggest another solution:
In unit-testing world such problems usually is solved similar way as
parameterized tests in JUnit.
>From my point of view we should implement test-runner supported
annotation, for example:
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.FIELD})
@Inherited
public @interface RunWithIgnite {
IgniteVersion since() default IgniteVersion.ver2_1;
IgniteVersion until();
}
So, a developer will be able use annotation to run a test with
different Ignite's versions.
I believe that it is much flexible solution in comparison with new
framework since we are still able to run nodes of directly defined
version.
>> I still don't understand why this version was explicitly added and
>> not just inherited from ignite-indexing. If we delegate all dependency
>> management to Maven itself it will simplify tests development. Tests will
>> look much simpler.
At the current implementation, we build a classpath based on statement
"full current classpath with replaced ignite-core artifacts". In you
example, test relates to "ignite-indexing" module which changed H2
dependency version in one of previous version, so the dependency had
to be excluded directly in tests.
If it's a problem for us, this can be solved easily with a new
approach to build class path based on statement "only ignite-core
module + compatibility module". Look at
"IgniteCompatibilityAbstractTest#getDependencies" for more details.
>> Suppose you have a closure that invokes some deprecated method which will
>> be removed in next release. After method removal, such tests will be
>> broken. At the current framework, I can't see a resolution of that problem.
There is not such problem in the current framework, you described an
issue which will be in a new framework without restricting "compatible
version" for test.
Anyway see my suggestion about the annotation above, it solves
described issue too.
In summary, all described issues (if they are issues really) can be
easily solved in the current framework. I see no reason in a new
framework with restricting current API.
What do you think?
On Wed, Feb 20, 2019 at 5:00 PM Dmitriy Pavlov wrote:
>
> Just a reminder: Veto is valid with justification. I see no reasons either
> (need some time to dive into details), but it is not a reason for a veto.
>
> ср, 20 февр. 2019 г. в 16:39, Anton Vinogradov :
>
> > Dmitriy,
> >
> > Please stop chilling community :)
> > No one makes final decisions here.
> > Please make sure you've got my position before chilling :)
> >
> > I see no arguments to replace one solution with another.
> > In case no real reasons will be provided this merge will gain my Veto, just
> > a warning...
> >
> > On Wed, Feb 20, 2019 at 4:37 PM Nikolay Izhikov
> > wrote:
> >
> > > Dmitriy,
> > >
> > > > So I suggest chilling a couple of days without any discussion.
> > >
> > > Why you trying to stop regular, positive dev-list discussion?
> > >
> > > I think, we should have a strong reason to throw away some subsystem and
> > > completely rewrite it.
> > > We will damage product robustness with it.
> > >
> > > >> 1. Automatic tests scaling for new versions.
> > > >> 2. Automatic dependency management.
> > >
> > > This features needed by Ignite, for sure.
> > > Pavel, can we extend current solution with new framework features?
> > >
> > >
> > >
> > > ср, 20 февр. 2019 г. в 16:23, Dmitriy Pavlov :
> > >
> > > > Hi, please do not rush to make any final decision. I think we all need
> > > some
> > > > time to think about these 2 solutions.
> > > >
> > > > So I suggest chilling a couple of days without any discussion.
> > > >
> > > > Later we can come back to this topic and dive into it once again, maybe
> > > we
> > > > can merge it into one taking the best parts of both.
> > > >
> > > > ср, 20 февр. 2019 г. в 16:07, Anton Vinogradov :
> > > >
> > > > > >> 1. Automatic tests scaling for new versions.
> > > > > >> 2. Automatic dependency management.
> > > > > Let's just improve the current solution.
> > > > > See no problems here.
> > > > >
> > > > > You'll have my *Veto* on merge without real reasons to replace
> > solution
> > > > > instead of improvement.
> > > > >
> > > > > On Wed, Feb 20, 2019 at 3:55 PM Pavel Kovalenko
> > > > > wrote:
> > > > >
> > > > > > Anton,
> > > > > >
> > > > > > In my first message, I already noticed the usability problems of
> > the
> > > > > > current framework and showed how they can be solved using the new
> > > > > > framework.
> > > > > > It gives us 2 major advantages:
> > > > > > 1. Automatic tests scaling for new versions.
> > > > > > 2. Automatic dependency management.
> > > > > > Described approach simplifies tests writing and supporting.
> > > > > >
> > > > > > I don't see now how we can avoid
