from:"Vladimir Ozerov \(JIRA\)"

[jira] [Created] (IGNITE-1292) Platforms .Net: Move handle registry to Ignite.

2015-08-24 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1292:
---

 Summary: Platforms .Net: Move handle registry to Ignite.
 Key: IGNITE-1292
 URL: https://issues.apache.org/jira/browse/IGNITE-1292
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1780) .Net: DateTime is seialized incorrectly.

2015-10-23 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1780:
---

 Summary: .Net: DateTime is seialized incorrectly.
 Key: IGNITE-1780
 URL: https://issues.apache.org/jira/browse/IGNITE-1780
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


Add the following test to PortableSelfTest.cs:
{code}
[Test]
public void TestWriteDate()
{
DateTime time = DateTime.Now;
DateTime timeUtc = DateTime.UtcNow;

Assert.AreEqual(_marsh.Unmarshal(_marsh.Marshal(time)), time);
Assert.AreEqual(_marsh.Unmarshal(_marsh.Marshal(timeUtc)), 
timeUtc);
}
{code}

Observe that it fails becuase we loose DateTimeKind. 

DateTime must be serialized in portable form only in two cases:
1) IPortableWriter.WriteDate()
2) IPortableWriter.WriteDateArray()

In all other cases it should be serialized in some other form. E.g., we can 
introduce new wrapper type ".NET-specific" and wrap DateTime, Collections, 
Arrays (IGNITE-1779) in it. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1768) .Net: Review portable configuration on Java side.

2015-10-22 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1768:
---

 Summary: .Net: Review portable configuration on Java side.
 Key: IGNITE-1768
 URL: https://issues.apache.org/jira/browse/IGNITE-1768
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


I am concerned why for some objects we pass assembly + type, while for others 
type only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1770) Portables: implement constant-time field lookup.

2015-10-22 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1770:
---

 Summary: Portables: implement constant-time field lookup.
 Key: IGNITE-1770
 URL: https://issues.apache.org/jira/browse/IGNITE-1770
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5


See 
https://cwiki.apache.org/confluence/display/IGNITE/Portable+object+constant-time+field+lookup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1803) Optimize random portable field lookup.

2015-10-28 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1803:
---

 Summary: Optimize random portable field lookup.
 Key: IGNITE-1803
 URL: https://issues.apache.org/jira/browse/IGNITE-1803
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


This ticket should be implemented after IGNITE-1770 is ready because it relies 
on constant-time lookup feature availability.

The following optimizations should be performed:
1) Create method "PortableFieldReader createFieldReader()" on PortableObject 
interface. This will minimize garbage and let us get rid from unnecessary 
header parsing.

2) Field position is calculated as follows: 
- Lookup ID mapper for type, get ID;
- Lookup schemas for type;
- Lookup schema for schema ID;
- Lookup field offset for field ID inside schema.
4 (!!!) map lookups! It should work as follows:
- Lookup ID mapper and schemas for type;
- Lookup field offset using (schema ID + field ID).
The latter lookup should be performed from highly optimized long->int map, not 
from conventional HashMap.

3) Normal field scan is faster than constant-time lookup when amount of fields 
is about 10. We need to have two field search strategies and pick the most 
performant one based on field count (which can be derived in advance from 
object footer size).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1779) .Net: Evaluate BinaryFormatter overhead for generic collections.

2015-10-23 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1779:
---

 Summary: .Net: Evaluate BinaryFormatter overhead for generic 
collections.
 Key: IGNITE-1779
 URL: https://issues.apache.org/jira/browse/IGNITE-1779
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


It seems that BinaryFormatter is extrmly inefficient. I tried to write a 
single Guid with and it resulted in >100 bytes comparing to 16 bytes when 
written in portable form. 
If we notice the same overhead for collections/arrays which are fairly often 
used types, we will have to revert to our initial approach when collections 
were serialized using our own mechanics.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1759) .Net: Guid read/write is not platform-safe.

2015-10-21 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1759:
---

 Summary: .Net: Guid read/write is not platform-safe.
 Key: IGNITE-1759
 URL: https://issues.apache.org/jira/browse/IGNITE-1759
 Project: Ignite
  Issue Type: Bug
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


Two problems here:
1) We do not count for endiannes, so byte shifts might be invalid.
2) Guid doesn't have fixed layout. For this reason there are no guarantees that 
it will take 16 bytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1902) NET: Adopt new marshalling flags.

2015-11-13 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1902:
---

 Summary: NET: Adopt new marshalling flags.
 Key: IGNITE-1902
 URL: https://issues.apache.org/jira/browse/IGNITE-1902
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


As a part of IGNITE-1816 I had to revisit our flags. Here are how they look now:

{code}
/** Flag: user type. */
public static final short FLAG_USR_TYP = 0x0001;

/** Flag: schema exists. */
public static final short FLAG_HAS_SCHEMA = 0x0002;

/** Flag indicating that object has raw data. */
public static final short FLAG_HAS_RAW = 0x0004;

/** Flag: offsets take 1 byte. */
public static final short FLAG_OFFSET_ONE_BYTE = 0x0008;

/** Flag: offsets take 2 bytes. */
public static final short FLAG_OFFSET_TWO_BYTES = 0x0010;

/** Flag: compact footer, no field IDs. */
public static final short FLAG_COMPACT_FOOTER = 0x0020;
{code}

The following changes must be made to CPP:
1) When schema exists (i.e. at least on non-raw fieid was written), HAS_ SCHEMA 
flag is set to true. This is the inversed version of what previously was 
"RAW_ONLY".
2) When raw data exists, HAS_RAW flag is set to true. This flag should be used 
to determine location of raw data. 
*DO NOT USE % for this anymore!*
3) For now COMPACT_FOOTER should always be 0. If you receive an object with 
this flag set to 1, an exception must be thrown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1917) Portables: micro optimizations.

2015-11-15 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1917:
---

 Summary: Portables: micro optimizations.
 Key: IGNITE-1917
 URL: https://issues.apache.org/jira/browse/IGNITE-1917
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1882) Rename BinaryTypeIdMapper to BinaryIdMapper

2015-11-10 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1882:
---

 Summary: Rename BinaryTypeIdMapper to BinaryIdMapper
 Key: IGNITE-1882
 URL: https://issues.apache.org/jira/browse/IGNITE-1882
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 1.5


BinaryTypeIdMapper is wrong name because it suggests type ID mapping, while it 
maps both type and field IDs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1815) Portables: optionally do not write field IDs to object footer.

2015-10-30 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1815:
---

 Summary: Portables: optionally do not write field IDs to object 
footer.
 Key: IGNITE-1815
 URL: https://issues.apache.org/jira/browse/IGNITE-1815
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


1) Implement compact schema write mode - instead of writting it in object 
footer, we can store field IDs in REPLICATED metadata cache.
When object is written in compact format, 0x10h flag must be set.
2) Add optional global switch on configuration to choose between compact 
(default) and non-compact modes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1816) Portables: optionally do not write field IDs to object footer.

2015-10-30 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1816:
---

 Summary: Portables: optionally do not write field IDs to object 
footer.
 Key: IGNITE-1816
 URL: https://issues.apache.org/jira/browse/IGNITE-1816
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


1) Implement compact schema write mode - instead of writting it in object 
footer, we can store field IDs in REPLICATED metadata cache.
When object is written in compact format, 0x10h flag must be set.
2) Add optional global switch on configuration to choose between compact 
(default) and non-compact modes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1838) Implement fast-path deserialization of common types in PortableObjectImpl.

2015-11-02 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1838:
---

 Summary: Implement fast-path deserialization of common types in 
PortableObjectImpl.
 Key: IGNITE-1838
 URL: https://issues.apache.org/jira/browse/IGNITE-1838
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 1.5


See PortableObjectImpl and PortableObjectOffheapImpl, method "fieldByOrder". 
We need to add fast-path deserializtion of the following types:

DECIMAL
STRING
UUID
DATE
TIMESTAMP




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1846) CPP: Adopt portable API changes.

2015-11-03 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1846:
---

 Summary: CPP: Adopt portable API changes.
 Key: IGNITE-1846
 URL: https://issues.apache.org/jira/browse/IGNITE-1846
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5


Important API changes will be introduced as a part of IGNITE-950. 
Once these changes are in place, we need to port them to CPP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1845) .Net: Adopt poratble API changes.

2015-11-03 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1845:
---

 Summary: .Net: Adopt poratble API changes.
 Key: IGNITE-1845
 URL: https://issues.apache.org/jira/browse/IGNITE-1845
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5


Important API changes will be introduced as a part of IGNITE-950. 
Once these changes are in place, we need to port them to .Net.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1847) Portables: add "field" method to PortableMetadata.

2015-11-03 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1847:
---

 Summary: Portables: add "field" method to PortableMetadata.
 Key: IGNITE-1847
 URL: https://issues.apache.org/jira/browse/IGNITE-1847
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


1) Add "PortableField field()" method to PortableMetadata. Currently it is 
located on "PortableObject", which is wrong.
2) PortableObject "field" and "hasField" methods must go through cached 
PortableFields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1862) CPP: Fix string handling

2015-11-05 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1862:
---

 Summary: CPP: Fix string handling
 Key: IGNITE-1862
 URL: https://issues.apache.org/jira/browse/IGNITE-1862
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


Currently we write strings using 2 bytes for each simbol. This way we are 
trying to mimic UTF16, which is natural for Java. 

Instead of doing this weird things, we should:
1) Tell user that we expect UTF8 string.
2) Write strings as plain char*.

Once it is done, we will be able to remove "convertStrings" property from 
PortableMarshaller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1694) .Net: More efficient way to call native methods.

2015-10-15 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1694:
---

 Summary: .Net: More efficient way to call native methods.
 Key: IGNITE-1694
 URL: https://issues.apache.org/jira/browse/IGNITE-1694
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.6


Currently we invoke native platform using 
"Marshal.GetDelegateForFunctionPointer". 

This is the least efficient way to call external code. What can be done to 
improve it:
1) Add [SuppressUnmanagedCodeSecurity] attribute.
2) Use P/Invoke instead. We do not use it now because we unpack native library 
in runtime into some temp folder, while PInvoke searches for libraries in 
predefined file system places. Need to evaluate whether we can switch back to 
P/Invoke somehow.

See 
http://ybeernet.blogspot.ru/2011/03/techniques-of-calling-unmanaged-code.html 
for more info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1693) .Net: Rename IMessage.Send(IEnumerable, object) method.

2015-10-15 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1693:
---

 Summary: .Net: Rename IMessage.Send(IEnumerable, object) method.
 Key: IGNITE-1693
 URL: https://issues.apache.org/jira/browse/IGNITE-1693
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


As some of integral .Net types implement "IEnumerable", we should rename the 
method
{code}
IMessage.Send(IEnumerable, object);
{code} 

to 

{code}
IMessage.SendAll(IEnumerable, object);
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1638) IGFS: Review IgfsImpl.checkConflictWithPrimary() method.

2015-10-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1638:
---

 Summary: IGFS: Review IgfsImpl.checkConflictWithPrimary() method.
 Key: IGNITE-1638
 URL: https://issues.apache.org/jira/browse/IGNITE-1638
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


This method looks very suspicious to me. It checks whether secondary file 
system contains any file under the same path as IGFS primary direcotry. If yes 
- an exception is thrown.
But why? What is the use case we are trying to resolve with this?

We need to re-think this approach and decide whether to leave it or remove it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1637) IGFS: Consistent properties propagation.

2015-10-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1637:
---

 Summary: IGFS: Consistent properties propagation.
 Key: IGNITE-1637
 URL: https://issues.apache.org/jira/browse/IGNITE-1637
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


IGFS has methods accepting properties map. E.g., create, append, mkdirs. 
How we should apply these properties? Two strategies are possible:
1) Apply there properties only to leaf node, and set defaults to parent nodes.
2) Apply there properties to all created nodes.

Lets take HDFS as a reference for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1639) IGFS: IgfsMetaManager() contains duplicated "delete" method.

2015-10-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1639:
---

 Summary: IGFS: IgfsMetaManager() contains duplicated "delete" 
method.
 Key: IGNITE-1639
 URL: https://issues.apache.org/jira/browse/IGNITE-1639
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.6


Our metamanager has two "delete" methods which do roughly the same thing: 
delete leafs and update parent listing. But one method remove collection of 
leafs, while another remove only single leaf.

Do we really need to have them as separate methods? Looks like them can be 
merged into one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1636) CPP: Add read/write methods with iterators for maps.

2015-10-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1636:
---

 Summary: CPP: Add read/write methods with iterators for maps.
 Key: IGNITE-1636
 URL: https://issues.apache.org/jira/browse/IGNITE-1636
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


This needs to be done in the same way as we did it for collections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1643) Cleanup platform communication protocol.

2015-10-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1643:
---

 Summary: Cleanup platform communication protocol.
 Key: IGNITE-1643
 URL: https://issues.apache.org/jira/browse/IGNITE-1643
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5


At the moment we have lots of .Net-specific objects travelling over a wire. 
E.g., user's .Net job is wrapped in some holder, then serialized and passed to 
Java, then Java pass this object over wire, etc..

This approach have two drawbacks:
1) Adds unnecessary overhead on wrapping.
2) Java have no chance to understand what it is passing. 

If we unwrap all .Net objects and pass them to Java as is, we can achieve 
slightly better performance. But what is *much more* important - we will be 
able to unleash possibilities for very cool transparent interoperability 
between platforms. E.g., .Net will be able to execute Java services, Java node 
will be able to execute .Net/CPP jobs if it has matching portable class 
implemented, etc..

Alternatively we can leave wrappers as is, but implement them in Java/CPP as 
well. This is not very good idea because of performance overhead.

The feature *must be implemented in 1.5* because now we are free to change 
protocol as we want.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1650) Add ability to specify thread pool for IgniteFuture listen/chain methods.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1650:
---

 Summary: Add ability to specify thread pool for IgniteFuture 
listen/chain methods.
 Key: IGNITE-1650
 URL: https://issues.apache.org/jira/browse/IGNITE-1650
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.6


Closures passed to IgniteFuture listen() and chain() methods are executed 
either in the same thread if future is completed, or in a completion thread 
(usually this is a thread from one of Ignite pools).

This enforces restrictions on what user can do in closures. He cannot use call 
operations, he cannot call any Ignite operations. Otherwise deadlocks or 
starvation could occur.

To fix that we should allow user to pass optional thread pool where passed 
closure should be executed. This already done in Java 8 CompletableFuture. We 
should do almost the same.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1651) Add version to portable object header.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1651:
---

 Summary: Add version to portable object header.
 Key: IGNITE-1651
 URL: https://issues.apache.org/jira/browse/IGNITE-1651
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1655) Decouple Date and Timestamp types.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1655:
---

 Summary: Decouple Date and Timestamp types.
 Key: IGNITE-1655
 URL: https://issues.apache.org/jira/browse/IGNITE-1655
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


Date and Timestamp types must be decoupled from each other:
1) New type ID must be introduced for Timestamp
2) New read/write methods should be added to portable readers, writers and 
builder.
3) Configuration parameter "isUseTimestamp" should be removed.
4) .Net DateTime type should be re-mapped to Timestamp.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1654) .Net: Ensure consistent "keepPortable" behavior.

2015-10-12 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1654:
---

 Summary: .Net: Ensure consistent "keepPortable" behavior.
 Key: IGNITE-1654
 URL: https://issues.apache.org/jira/browse/IGNITE-1654
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


We need to ensure that "keep portable" concept is
a) Consistent;
b) Exists for all logical units where it is necessary.

Lets split "keep portable" behavior in two scenarios:
1) Client mode. This is when client (caller) operates on portable objects. 
Example: ICache.Get(...);
2) Server mode. This is when some server side object (caller) operates on 
portable objects. Example: a remote service which should not deserialize passed 
arguments and operate on portables instead.

Looks like both cases can be handled using the same "keep portable" flag on 
related logical unit.

Logical units affected:
1) Compute - at the very least closures can be executed in this mode;
2) Cache;
3) Data streamer;
4) Data structures: atomic stamped, atmoc reference;
5) Cache events;
6) Messages;
7) Services.

Probably we should introduce a kind of "IKeepPortableEnabled" (or so) 
interface, which will have a method "T WithKeepPortable()". This way users will 
be able to easily understand that "keep portable" semantics is applicable.

Lets split it into several children tickets if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1626) .Net: Create NuGet package.

2015-10-07 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1626:
---

 Summary: .Net: Create NuGet package.
 Key: IGNITE-1626
 URL: https://issues.apache.org/jira/browse/IGNITE-1626
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.5
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: ignite-1.5


*Overview*
To boost usage of the product we need to distribute it using NuGet, which is 
tightly integrated with Visual Studio.
To achieve this several technical, infrastructure and marketing tasks must be 
completed.

*Technical tasks*
- Apache Ignite.NET heavily depends on relative positions of compiled Java 
classes. We must be able to apply different classpath search strategies 
depending on packaging. This includes normal search in exploded build 
directory, search in NuGet package, search in local Maven repo.
- We need a way to select classpath search strategy. E.g. we can add special 
marker resource to NuGet package so that DLL understands that it is 
NuGet-based. More investigation here is reuqired.
- Minimal set of required JARs should be defined. Currently we have over >100Mb 
of Java libraries shipped with Ignite build. NuGet have limitation of 30Mb per 
package. Only vital subset of JARs should be shipped.

*Infrastructure tasks*
- INFRA team must be asked about a place where NuGet package could be stored. 
- Separate build plan should be created, producing NuGet artifacts.

*Marketing tasks*
- We need to produce clean, selling and intriguing header and description for 
our package and attach logo.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1629) .Net: Introduce native logging facility.

2015-10-07 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1629:
---

 Summary: .Net: Introduce native logging facility.
 Key: IGNITE-1629
 URL: https://issues.apache.org/jira/browse/IGNITE-1629
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: ignite-1.6


This is pretty serious usability issue. Currently Ignite produces logs using 
Java "log4j" library. While naural for Java environment, this is somewhat alien 
for Windows users. 

We need to investigate ability to hack into normal .Net logging frameworks. 
This include both native Windows APIs (e.g. events), and widely-used .Net 
loggers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1630) .Net: Create LINQ adapter for queries.

2015-10-07 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1630:
---

 Summary: .Net: Create LINQ adapter for queries.
 Key: IGNITE-1630
 URL: https://issues.apache.org/jira/browse/IGNITE-1630
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.6


*NOTE*: Is targeted for 1.6, but might be moved forward in case of tight 
release schedule.

SQL queries are one of the most frequently used features in data grids. And 
.Net comes with a very nice LINQ concept. 

We need to investigate and idea of own LINQ adapter which will delegate to 
Ignite native query API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1628) .Net: Ensure that .Net works on Mono platform.

2015-10-07 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1628:
---

 Summary: .Net: Ensure that .Net works on Mono platform.
 Key: IGNITE-1628
 URL: https://issues.apache.org/jira/browse/IGNITE-1628
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.6


*NOTE*: Targeted for 1.6, but could be moved even further in case of tight 
release schedule.

*Goal*
As Microsoft is moving .Net towards open standards, Mono receives more an more 
attention. 

We need to ensure that our product works fine on this platform. This includes 
both Windows, Linux and (possibly) Mac environments.

*Tasks*
- Ensure that Ignite works with Mono on Windows. This is the easiest task 
because we only need to re-compile .Net part.
- Ensure that Ignite works with Mono on Linux. This will require alternate 
build procedure because CPP recompilation will be required as well.
- Create relevant TC builds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1663) .Net: Check why node ID is nullable in IEvents and IMessaging interfaces.

2015-10-13 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1663:
---

 Summary: .Net: Check why node ID is nullable in IEvents and 
IMessaging interfaces.
 Key: IGNITE-1663
 URL: https://issues.apache.org/jira/browse/IGNITE-1663
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


It is not clear why we operate on node as "Guid?". Are there any use cases when 
it could be null?
If no, we must change signature to "Guid".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1646) .Net: IgniteGuid must be serializable and portable.

2015-10-09 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1646:
---

 Summary: .Net: IgniteGuid must be serializable and portable.
 Key: IGNITE-1646
 URL: https://issues.apache.org/jira/browse/IGNITE-1646
 Project: Ignite
  Issue Type: Bug
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: 1.5


1) User must be able to pass IgniteGuid over a wire using serializable 
semantics when serialized manually by user.
2) On the other hand, it must be portable for correct passing between Java and 
.Net.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1950) Binary format: Arrays and collections use handles differently in Java and platforms.

2015-11-19 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1950:
---

 Summary: Binary format: Arrays and collections use handles 
differently in Java and platforms.
 Key: IGNITE-1950
 URL: https://issues.apache.org/jira/browse/IGNITE-1950
 Project: Ignite
  Issue Type: Bug
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5


In Java arrays and collections could use handles. In .Net and CPP they can't. 
We need to provide unified approach for all platforms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1325) Platforms: introduce extensions framework.

2015-08-28 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1325:
---

 Summary: Platforms: introduce extensions framework.
 Key: IGNITE-1325
 URL: https://issues.apache.org/jira/browse/IGNITE-1325
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Blocker
 Fix For: ignite-1.4


We need to introduce several methods which will allow craeting extensions 
without any changes in CPP (common) code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1331) Platforms: consistent naming for "op" methods.

2015-08-31 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1331:
---

 Summary: Platforms: consistent naming for "op" methods.
 Key: IGNITE-1331
 URL: https://issues.apache.org/jira/browse/IGNITE-1331
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: ignite-1.4


Currently "op" methods which are invoked from native platforms have consufing 
names. We need to make them clean and consistent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1303) Platforms: abstract-out platform context.

2015-08-26 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1303:
---

 Summary: Platforms: abstract-out platform context.
 Key: IGNITE-1303
 URL: https://issues.apache.org/jira/browse/IGNITE-1303
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


Need to introduce context interface thus simplifying code move from GridGain.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1310) Platforms: move cluster group to Ignite.

2015-08-27 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1310:
---

 Summary: Platforms: move cluster group to Ignite.
 Key: IGNITE-1310
 URL: https://issues.apache.org/jira/browse/IGNITE-1310
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1336) Platforms: Create JNI callback extensions.

2015-08-31 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1336:
---

 Summary: Platforms: Create JNI callback extensions.
 Key: IGNITE-1336
 URL: https://issues.apache.org/jira/browse/IGNITE-1336
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1341) Platforms: refactor exceptions.

2015-09-01 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1341:
---

 Summary: Platforms: refactor exceptions.
 Key: IGNITE-1341
 URL: https://issues.apache.org/jira/browse/IGNITE-1341
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


1) Move exceptions from "platforms" to "core".
2) PlatformNativeException: move from "compute" to base package.
3) PlatormExtendedException: incorporate into regular exception hierarchy.
4) PlatformContext.createNativeException must return PlatformNativeException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1375) Move CPP tests for "src/test" folder.

2015-09-04 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1375:
---

 Summary: Move CPP tests for "src/test" folder.
 Key: IGNITE-1375
 URL: https://issues.apache.org/jira/browse/IGNITE-1375
 Project: Ignite
  Issue Type: Sub-task
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: ignite-1.4






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1382) Consider using "JavaCritical_" prefix to speedup short Java -> platform callbacks.

2015-09-07 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1382:
---

 Summary: Consider using "JavaCritical_" prefix to speedup short 
Java -> platform callbacks.
 Key: IGNITE-1382
 URL: https://issues.apache.org/jira/browse/IGNITE-1382
 Project: Ignite
  Issue Type: Task
  Components: clients
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


This could improve performance of various native methods such as 
"destroy"-family and "reallocate".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1362) Update notifier causes Ignite instance leak.

2015-09-03 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1362:
---

 Summary: Update notifier causes Ignite instance leak.
 Key: IGNITE-1362
 URL: https://issues.apache.org/jira/browse/IGNITE-1362
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


IgniteKernal has private non-static anonymous class for GridTimerTask which 
forces üpdate notify. 
As a result IgniteKernal instance leaks to it and stays in memory for a long 
time even after node is stopped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1364) Move C++ platform to Ignite.

2015-09-03 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1364:
---

 Summary: Move C++ platform to Ignite.
 Key: IGNITE-1364
 URL: https://issues.apache.org/jira/browse/IGNITE-1364
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


This involves:
1) Moving code base;
2) Moving examples;
3) Fixing doxygen docs generation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1360) Extend platform processor lifecycle.

2015-09-02 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1360:
---

 Summary: Extend platform processor lifecycle.
 Key: IGNITE-1360
 URL: https://issues.apache.org/jira/browse/IGNITE-1360
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Currently platform processor lifecycle is the same as for all other processors 
wiht 4 callbacks: onStart, onKernalStart, onKernalStop, onStop.

This appears to be not enough for platforms. IN particular, it doesn't let us 
flush all write-behind store data correctly and doesn't allow for normal 
AFTER_NODE_STOP lifecycle callback.
Also it makes platfomr initialization more painful because some callbacks might 
reach the platform when it is not initialized yet.

To mitigate this problem we should extend platform processor with two more 
callbacks: "onBeforeStart" and "onAfterStop". Careful management of these 
callbacks will let us get rid of all aforementioned problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1356) Move platforms Java public API to Ignite.

2015-09-02 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1356:
---

 Summary: Move platforms Java public API to Ignite.
 Key: IGNITE-1356
 URL: https://issues.apache.org/jira/browse/IGNITE-1356
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1353) PortableContext.typeId() incorrectly resolves type ID for predefined system types.

2015-09-02 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1353:
---

 Summary: PortableContext.typeId() incorrectly resolves type ID for 
predefined system types.
 Key: IGNITE-1353
 URL: https://issues.apache.org/jira/browse/IGNITE-1353
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


See PortableContext.typeId() method.
First it checks whether type is "system", and only then try picking predefiend 
ID. As a result, predefined types like IgniteBiTuple are written with wrong IDs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1359) Inject platform processor into Ignite lifecycle.

2015-09-02 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1359:
---

 Summary: Inject platform processor into Ignite lifecycle.
 Key: IGNITE-1359
 URL: https://issues.apache.org/jira/browse/IGNITE-1359
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


Processor creation sequence should be as follows:
1) Ask plugins for their processor. Stop in case of non-null result.
2) Check for PlatformConfiguration. If not null - create fully-fledged 
processor through reflection if it is present in classpath. Stop if created.
3) Return no-op processor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1357) Move Java bootstrap logic to Ignite.

2015-09-02 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1357:
---

 Summary: Move Java bootstrap logic to Ignite.
 Key: IGNITE-1357
 URL: https://issues.apache.org/jira/browse/IGNITE-1357
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1396) Remove platforms from public API for ignite-1.4.

2015-09-10 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1396:
---

 Summary: Remove platforms from public API for ignite-1.4.
 Key: IGNITE-1396
 URL: https://issues.apache.org/jira/browse/IGNITE-1396
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


It appears that adding platforms in the current release will shift release date 
without real necessity. 
Lets hide platforms API from release for now and re-add it in 1.5.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1477) Hadoop file system must throw detailed error message if communication with IGFS failed.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1477:
---

 Summary: Hadoop file system must throw detailed error message if 
communication with IGFS failed.
 Key: IGNITE-1477
 URL: https://issues.apache.org/jira/browse/IGNITE-1477
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.4






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1422) .Net: Marshalling can be optimized for fixed-length objects.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1422:
---

 Summary: .Net: Marshalling can be optimized for fixed-length 
objects.
 Key: IGNITE-1422
 URL: https://issues.apache.org/jira/browse/IGNITE-1422
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


Sometimes we can guess user object length in advance:
1) Strict rule: it is "reflective" and contains only fixed-length fields.
2) Speculative rule: It is "marshal aware", contains only fixed-length fields 
and there is only one metadata "path".

In these cases we can do the following:
1) No tracking of length and raw offset.
2) Header can be written as one memcpy() rather than as multiple int/byte/bool 
writes.

We need to have a fallback strategy for a cases when new metadata path appears 
or user write raw data. It is very easy: if expected and actual lengths differ 
at the end, set proper values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1421) .Net: Optimize writes of ultra-dense structures.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1421:
---

 Summary: .Net: Optimize writes of ultra-dense structures.
 Key: IGNITE-1421
 URL: https://issues.apache.org/jira/browse/IGNITE-1421
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


There is a case when we can dramatically increase marshal/unmarshal performance:
1) Type is a structure;
2) It contains only integer/float/double primitive types with well-defined;
memory layout which is consistent with our serialization protocol;
3) We are sure that there are no gaps in it's memory layout.
4) User writes it as "marshal-aware" and fields write order is consistent with 
memory layout.

In this case we can copy the whole structure into our stream with a single 
memcpy() operation. And we can read it from the stream as easy as [pointer 
dereference + position shift] (provided that target is on the stack).





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1410) .Net: Implement IEvents.RecordLocal.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1410:
---

 Summary: .Net: Implement IEvents.RecordLocal.
 Key: IGNITE-1410
 URL: https://issues.apache.org/jira/browse/IGNITE-1410
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1487) Exception is always thrown on normal code path when connecting to IGFS from file system.

2015-09-15 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1487:
---

 Summary: Exception is always thrown on normal code path when 
connecting to IGFS from file system.
 Key: IGNITE-1487
 URL: https://issues.apache.org/jira/browse/IGNITE-1487
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


See HadoopIgfsWrapper.delegate(). It first tries to get default Ignite 
instance. As it doesn't exist in the most cases, an exception is thrown. We 
catch and ignore it. As a result, we throw exception on normal and pretty hot 
code path.

Proposed fix: check for grid existence using IgnitionEx.state().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1413) .Net: Get rid of set -> map conversion in PlatformDotNetCacheStore.writeAll

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1413:
---

 Summary: .Net: Get rid of set -> map conversion in 
PlatformDotNetCacheStore.writeAll
 Key: IGNITE-1413
 URL: https://issues.apache.org/jira/browse/IGNITE-1413
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


This is a legacy artifact from a veeery old API which is currently removed. 
Just pass set without unnecessary conversions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1414) .Net: support async/await in the API via extension methods

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1414:
---

 Summary: .Net: support async/await in the API via extension methods
 Key: IGNITE-1414
 URL: https://issues.apache.org/jira/browse/IGNITE-1414
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


Currently to get awaitable target (Task) for async operation, user have to do 
the following (pseudocode):
{code}
target.ExecuteOperation();

IFuture fut = target.CurrentFuture();

Task task = fut.ToTask();
{code}

This is too complex. We need to think how to minimize this boilerplate code. 
Extensions + lambdas appears to be good candidates for this. E.g.:
{code}
Task task = target.DoAsync(t => t.Execute());
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1433) .Net: Add GridException.JavaStackTrace

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1433:
---

 Summary: .Net: Add GridException.JavaStackTrace
 Key: IGNITE-1433
 URL: https://issues.apache.org/jira/browse/IGNITE-1433
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


Propagate java stack trace as a string in ExceptionUtils.GetException and write 
it to a new field in IgniteException class.

This will simplify debugging for us both locally and when getting error reports 
from clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1441) CPP: Support filter in SCAN queries.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1441:
---

 Summary: CPP: Support filter in SCAN queries.
 Key: IGNITE-1441
 URL: https://issues.apache.org/jira/browse/IGNITE-1441
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1448) CPP: Implement cache iterators.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1448:
---

 Summary: CPP: Implement cache iterators.
 Key: IGNITE-1448
 URL: https://issues.apache.org/jira/browse/IGNITE-1448
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1424) .Net: Add tests for BIG_ENDIANs.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1424:
---

 Summary: .Net: Add tests for BIG_ENDIANs.
 Key: IGNITE-1424
 URL: https://issues.apache.org/jira/browse/IGNITE-1424
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Minor
 Fix For: ignite-1.5


As all our hardware uses LITTLE_ENDIAN byte order, BIG_ENDIAN was never tested 
properly. We do not know, whether it works. We do not know it's performance 
characteristics. 
Need to add ability to override endian final flags in both .Net and Java (e.g. 
through a system property or through "Ex" configuration) and then run unit 
tests with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1438) CPP: Implement transactions API.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1438:
---

 Summary: CPP: Implement transactions API.
 Key: IGNITE-1438
 URL: https://issues.apache.org/jira/browse/IGNITE-1438
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1444) CPP: Implement cache expiry policy.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1444:
---

 Summary: CPP: Implement cache expiry policy.
 Key: IGNITE-1444
 URL: https://issues.apache.org/jira/browse/IGNITE-1444
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1442) CPP: Implement cache async mode.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1442:
---

 Summary: CPP: Implement cache async mode.
 Key: IGNITE-1442
 URL: https://issues.apache.org/jira/browse/IGNITE-1442
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1426) .Net: Improve/add forked tests

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1426:
---

 Summary: .Net: Improve/add forked tests
 Key: IGNITE-1426
 URL: https://issues.apache.org/jira/browse/IGNITE-1426
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


Run forked tests from a separate directory to better cover assembly loading 
code and examples scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1434) Platforms: Ensure that @IgniteInstanceResource is set only on setters, not fields.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1434:
---

 Summary: Platforms: Ensure that @IgniteInstanceResource is set 
only on setters, not fields.
 Key: IGNITE-1434
 URL: https://issues.apache.org/jira/browse/IGNITE-1434
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


Currently @IgniteInstanceResource is set on fields. And then we have to deal 
with complex initiailzation logic. 
Instead, we must set this annotation on setters which will perform 
initialization immediately.
Furthermore, when possible we must avoid injection in favor of direct setter 
calls.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1437) CPP: Implement SQL fields query.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1437:
---

 Summary: CPP: Implement SQL fields query.
 Key: IGNITE-1437
 URL: https://issues.apache.org/jira/browse/IGNITE-1437
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1440) CPP: Implement cache store.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1440:
---

 Summary: CPP: Implement cache store.
 Key: IGNITE-1440
 URL: https://issues.apache.org/jira/browse/IGNITE-1440
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1439) CPP: Implement futures.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1439:
---

 Summary: CPP: Implement futures.
 Key: IGNITE-1439
 URL: https://issues.apache.org/jira/browse/IGNITE-1439
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1445) CPP: Implement cache invoke.

2015-09-11 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1445:
---

 Summary: CPP: Implement cache invoke.
 Key: IGNITE-1445
 URL: https://issues.apache.org/jira/browse/IGNITE-1445
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


This will require well-defined architecture for generic callback calls from 
Java core to CPP.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1466) CPP: Implement portable objects support.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1466:
---

 Summary: CPP: Implement portable objects support.
 Key: IGNITE-1466
 URL: https://issues.apache.org/jira/browse/IGNITE-1466
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


We need to support portable objects in the same way as it is done in .Net and 
Java: if some flag ("keepPortable") is set, then we should return not real 
object, but rather a wrapper around it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1469) CPP: Support decimals in marshaller.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1469:
---

 Summary: CPP: Support decimals in marshaller.
 Key: IGNITE-1469
 URL: https://issues.apache.org/jira/browse/IGNITE-1469
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


We need to support decimals in CPP. 

Counterparts:
Java -> BigDecimal
.Net -> decimal



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1470) CPP: Support string arrays in marshaller.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1470:
---

 Summary: CPP: Support string arrays in marshaller.
 Key: IGNITE-1470
 URL: https://issues.apache.org/jira/browse/IGNITE-1470
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


String arrays are written in a special way in Java/.Net. We need to support 
them in CPP as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1473) CPP: Add failed keys for CachePartialUpdateException.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1473:
---

 Summary: CPP: Add failed keys for CachePartialUpdateException.
 Key: IGNITE-1473
 URL: https://issues.apache.org/jira/browse/IGNITE-1473
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1474) CPP: Support nested collection reads/writes.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1474:
---

 Summary: CPP: Support nested collection reads/writes.
 Key: IGNITE-1474
 URL: https://issues.apache.org/jira/browse/IGNITE-1474
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Currently we do not support nested container writes. E.g. vector -> 
Person -> vector.

We must get rid of this restriction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1465) SecondaryFileSystemProvider must throw more informative exception if secondary cfg URL is not resolved.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1465:
---

 Summary: SecondaryFileSystemProvider must throw more informative 
exception if secondary cfg URL is not resolved.
 Key: IGNITE-1465
 URL: https://issues.apache.org/jira/browse/IGNITE-1465
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.4


If SecondaryFileSystemProvider failed to resolve secondary file system URI, the 
following exception is thrown: "java.lang.IllegalArgumentException: Failed to 
resolve secondary file system configuration path: [path]"

We need to give user possible solutions:
1) Check if file exists (either absolute or relative to IGNITE_HOME)
2) Check if there is enough access rights.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1472) CPP: Support enums.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1472:
---

 Summary: CPP: Support enums.
 Key: IGNITE-1472
 URL: https://issues.apache.org/jira/browse/IGNITE-1472
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Needed to be able to interact with Java/.Net enums. Probably this can be done 
with some extension structure which will provide type ID for the enum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1475) CPP: Implement shortcuts for array/collection/dictionary methods in reader/writer.

2015-09-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1475:
---

 Summary: CPP: Implement shortcuts for array/collection/dictionary 
methods in reader/writer.
 Key: IGNITE-1475
 URL: https://issues.apache.org/jira/browse/IGNITE-1475
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Currently user is forced to use our reader/writer wrappers for collections. 
While really required in general case, this is not needed for the most common 
data types: vectors, sets, maps, etc.

We need to define shortcut methods in reader/writer .h files to support it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1567) Optimize portable object fields read/write in .Net.

2015-09-29 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1567:
---

 Summary: Optimize portable object fields read/write in .Net.
 Key: IGNITE-1567
 URL: https://issues.apache.org/jira/browse/IGNITE-1567
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1572) .Net: Check if we can get rid of "WritePortableOrSerializable" method.

2015-09-29 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1572:
---

 Summary: .Net: Check if we can get rid of 
"WritePortableOrSerializable" method.
 Key: IGNITE-1572
 URL: https://issues.apache.org/jira/browse/IGNITE-1572
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


We have lots of places where "WritePortableOrSerializable" routine is invoked. 
It is not clear why do we need it provided that we already handle 
[Serializable] classes properly. 
Looks like we are able to get rid of it safely, aren't we? Need to double-check.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1580) CPP: Review Ignition interface.

2015-09-30 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1580:
---

 Summary: CPP: Review Ignition interface.
 Key: IGNITE-1580
 URL: https://issues.apache.org/jira/browse/IGNITE-1580
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


It is necessary to review Ignition interface and understand whether it is 
compliant with common CPP practices. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1581) Synchronous cache operation can deadlock on async semaphore.

2015-09-30 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1581:
---

 Summary: Synchronous cache operation can deadlock on async 
semaphore.
 Key: IGNITE-1581
 URL: https://issues.apache.org/jira/browse/IGNITE-1581
 Project: Ignite
  Issue Type: Task
  Components: cache
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Problem can be reproduced as follows:
1) Set CacheConfiguration.setMaxConcurrentAsyncOperations() to a very small 
value, e.g. 3.
2) T1: Start PESSIMISTIC/REPEATABLE_READ tx and call Cache.get(1);
3) Start 3 other threads, initiate PESSIMISTIC/REPEATABLE_READ tx and call 
Cache.get(1) on them. They will stuck as expected.
4) T1: Call Cache.get(1) again. Instead of getting already locked value, thread 
is stuck on async semaphore acquire:

{code}
at sun.misc.Unsafe.park(Unsafe.java:-1)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
  at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4329)
  at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOp(GridCacheAdapter.java:4203)
  at 
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.getAllAsync(GridDhtColocatedCache.java:211)
  at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync(GridCacheAdapter.java:4609)
  at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4556)
  at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1569)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1615) .Net: Perform AtomicLong.get() without JNI if possible.

2015-10-05 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1615:
---

 Summary: .Net: Perform AtomicLong.get() without JNI if possible.
 Key: IGNITE-1615
 URL: https://issues.apache.org/jira/browse/IGNITE-1615
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


Variables with atomic/interlocked semantics are frequently used in mostly-read 
scenarios. E.g. in spin-loops, non-blocking alogrithms, as regular volatiles, 
etc..

With current implementation we perform JNI call on every read which is too 
expensive, especially with poor Java performance when performing (native -> 
JVM) transition.

We can optimize it with the following non-blocking algorithm:
1) Add atomic "stamp" field.
2) Add atomic "cached" field.
3) On any update:
- Do the update;
- Atomically increment the stamp;
4) On any read:
- Read stamp (oldStamp);
- Read cached value;
- Read stamp again (newStamp);
- if (oldStamp == newStamp == cache.stamp()), return cached value.
- Otherwise perform real read through JNI and update cached value with a pair 
of (readValue + oldStamp);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1511) Revisit IGFS delete worker.

2015-09-18 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1511:
---

 Summary: Revisit IGFS delete worker.
 Key: IGNITE-1511
 URL: https://issues.apache.org/jira/browse/IGNITE-1511
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Looks like our IGFS delete worker is broken:
1) It doesn't account of locked files
2) It first removes data and then removes meta. Looks like we should do that in 
the opposite order to prevent orphaned data blocks in case very close 
concurrent write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1509) .Net: User must operate on IEnumerable in events API rather than on arrays.

2015-09-18 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1509:
---

 Summary: .Net: User must operate on IEnumerable in events API 
rather than on arrays.
 Key: IGNITE-1509
 URL: https://issues.apache.org/jira/browse/IGNITE-1509
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: ignite-1.5


Currently we expose arrays for event groups. As they are mutable, we have to 
copy them every time user use them. 
To avoid this we should switch to IEnumerables and leave shortcut methods with 
"params[]" for convenience.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1512) Rework IGFS remove operation.

2015-09-18 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1512:
---

 Summary: Rework IGFS remove operation.
 Key: IGNITE-1512
 URL: https://issues.apache.org/jira/browse/IGNITE-1512
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


Currently it goes through "move" on it's final stage. This causes redundant 
code dependencies and unnecessary check. Let's implement it as a separate 
method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1539) Marshaller performance micro-optimizations.

2015-09-24 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1539:
---

 Summary: Marshaller performance micro-optimizations.
 Key: IGNITE-1539
 URL: https://issues.apache.org/jira/browse/IGNITE-1539
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
 Fix For: ignite-1.5


The main goal of marshalling is to write an object to byte[]/ByteBuffer, or 
read from it. Currently each read/write is surrounded by quite a few additional 
actions which can be avoided either in general, or in some specific cases.

Looks like we can apply lots of micro-optimizations to our marshallers:
1) Do not perform HashMap lookups for each field ID.
2) Pre-calculate field IDs for reflection-based types.
3) Pre-calculate metadata for reflection-based types and do not track it.
4) Avoid "raw" mode checks for reflection-based types.
5) Merge small writes into bigger ones where possible. E.g. WRITE_INT_FIELD_ID 
+ WRITE_INT_FIELD_LEN can be replaced with a single WRITE_LONG when field 
length is constant.
6) Memory bounds checks can be avoided in lots cases if we read/write object 
part with pre-defined length.
7) For objects having only fixed-length fields, we can pre-calculate a kind of 
template seriously minimizing writes.
8) Anything else?

Applying some of these optimizations in .Net shows very promising results with 
up to 40% throughput increase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1549) Optimize portable object fields write in non-raw mode.

2015-09-25 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1549:
---

 Summary: Optimize portable object fields write in non-raw mode.
 Key: IGNITE-1549
 URL: https://issues.apache.org/jira/browse/IGNITE-1549
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: ignite-1.5


Currently we write user fields as follows:
0 ,, 3 - field ID;
4 - field type;
5 ..8 - field len;
9 .. - the field itself.

It can be optimized as follows:
1) Field len usually can be inferred from type. E.g., for int it is 4.
2) Frequently used constants can be written as separate types. E.g. INT - 
normal int, INT_0 - zero, etc.
3) Last, but not least, values should be encoded using "variable bytes" (and 
possibly ZigZag) algorithm. This will give us 2 bytes economy for ints and 
longs on average (I assume here that longs are usually bigger than 4 bytes, 
e.g. timestamps).

*New types will be introduced:*
1) Booleans: BOOL_FALSE, BOOL_TRUE;
2) Bytes: BYTE_C0 => zero, BYTE_C1 => 1, BYTE_C1N => -1;
3) Shorts, chars: SHORT_C0, SHORT_C1, SHORT_C1N;
4) Ints: INT_C0, INT_C1, INT_C1N, INT_1 - int which fits into 1 byte, INT_1N - 
same for negative value, INT_2, INT_2N, INT_3, INT_3N, INT_3, INT_3N, INT_4, 
INT_4N.
5) Longs: same as ints, but have only 2, 4, 6 and 8 byte count discriminators 
to avoid excessive calculations.

It means that instead of 6 integer types previously, we will have 2 + 3 + 3 + 3 
+ 11 + 11 = 32 types.

To avoid excessive switches or (even worse) array/map lookups to understand 
what the type is, we can divide all types space (256) into two parts: optimized 
and non-optimized. Optimized space will have the MSB set to 1, and mentioned 
~30 optimized types (or some of them) are located there.

For floats and doubles we simply infer length. 

For primitive arrays we do not write field length and then arrya length, but 
only array length.

*Expected compaction*:
bool: 10 -> 5 bytes (50%);
byte: 10 -> 5-6 bytes (45%);
short, char: 11 -> 5-7 bytes, 7 on average (35%);
int: 13 -> 5-9 bytes, 7 on average (45%).
long: 17 -> 5-13 bytes, 11 on average (35%).
float: 13 -> 9 bytes (30%);
double: 17 -> 13 bytes (25%);

*Expected CPU overhead on writes:*
Bool, float, double: -
Byte, short, char: zero check, sign check;
Int, long: two (shift + OR)s to understand bytes count, if small - "zero" and 
"one" checks, if big - sign check,

*Expected CPU overhead on reads:*
One additional branch between optimzied and non-optimized spaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1550) Optimize "direct" message serialization.

2015-09-25 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1550:
---

 Summary: Optimize "direct" message serialization.
 Key: IGNITE-1550
 URL: https://issues.apache.org/jira/browse/IGNITE-1550
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: ignite-1.5


*Overview*
Ignite use "direct" serialization to speed up marshalling of internal classes 
avoiding byte array copying.
This mechanism can be improved significantly on class-by-class basis reducing 
message size by 10-30% in different cases.

*Implementation*
1) Definte the list of possible optimizations. At the very least it includes:
- Var-length integer encoding; 
- Special cases (constants, positive-only values, low amount of significant 
bits, etc.);
- Efficient writes of "nulls";
- Write groups of relates fields with boolean switch if they are either null or 
not-null at the same time.

2) Determine what to optimize. This can be done on class-by-class basis for 
each of ~180 messages. Better approach will be to enable tracking of written 
values inside DirectMessageWriter. Then we should run Ignite with differnet 
payloads and identify fields which are good candidates for optimization.

3) Implement each optimziation.

*Risks*
Backward compatibility will be broken. We must either inform users about it, or 
support ability to use old non-optimized protocol version somehow.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-1548) Optimize portable object header.

2015-09-25 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-1548:
---

 Summary: Optimize portable object header.
 Key: IGNITE-1548
 URL: https://issues.apache.org/jira/browse/IGNITE-1548
 Project: Ignite
  Issue Type: Task
  Components: general
Affects Versions: 1.1.4
Reporter: Vladimir Ozerov
Priority: Blocker
 Fix For: ignite-1.5


Currently portable object header takes 18 bytes:
1 - object type
1 - user flag
4 - type ID
4 - hash code
4 - length
4 - raw offset.

The following optimizations can be applied:
1) User flag can be easilty merged into object type. (-1 byte)
2) In most cases only length or raw-offset exists, but not both. Remove one of 
them and encode state into the object type. (-4 bytes)
3) Length usually fit in 1-2 bytes. Encode it with "variable bytes" algo (-2-3 
bytes).

As a result we will shrink header size from 18 to 10-11 bytes. Encoded states 
will require 6 additional object types:
104 => system, with length 
105 => system, with raw offset
106 => system, with both
107, 108, 109 - the same, but for user types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2102) .NET: Simplify query examples

2015-12-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2102:
---

 Summary: .NET: Simplify query examples
 Key: IGNITE-2102
 URL: https://issues.apache.org/jira/browse/IGNITE-2102
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: 1.6


Simply move IGNITE-2097 to .NET once it is ready.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2103) CPP: Simplify query examples.

2015-12-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2103:
---

 Summary: CPP: Simplify query examples.
 Key: IGNITE-2103
 URL: https://issues.apache.org/jira/browse/IGNITE-2103
 Project: Ignite
  Issue Type: Task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: 1.6


Simply move IGNITE-2097 to .NET once it is ready.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2101) NPE when trying to submit Hadoop job to a node where Hadoop module is not loaded.

2015-12-08 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2101:
---

 Summary: NPE when trying to submit Hadoop job to a node where 
Hadoop module is not loaded.
 Key: IGNITE-2101
 URL: https://issues.apache.org/jira/browse/IGNITE-2101
 Project: Ignite
  Issue Type: Bug
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: 1.6


See 
http://apache-ignite-users.70518.x6.nabble.com/NPE-issue-with-trying-to-submit-Hadoop-MapReduce-td2146.html

When Hadoop module is not loaded, HadoopNoopProcessor is started on the node. 
And it's method nextJobId() returns null. This null is then propagated to the 
client and NPE is raised. There is no way for users to underastand what is 
wrong.

Solution should be fairly simple: just throw an exception with sensible message 
from HadoopNoopProcessor.nextJobId() method. It should be propagated smoothly 
to the client. 
Also we need to add tests for this scenario.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2115) Improve "peerClassLoadingEnabled=true" handling in Hadoop module.

2015-12-09 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2115:
---

 Summary: Improve "peerClassLoadingEnabled=true" handling in Hadoop 
module.
 Key: IGNITE-2115
 URL: https://issues.apache.org/jira/browse/IGNITE-2115
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: 1.6


We have very subtle logic for Hadoop module loading. Even if it is in 
classpath, if "peerClassloadingEnabledFlag" is set, module is not loaded. No 
warning or errors appear. 

We should:
1) Print warning if this flag prevents loading of Hadoop module;
2) Remove "peerClassLoadingEnabled=true" from IGFS example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2122) .NET: Add generic Read/WriteCollection and Read/WriteDictionary methods.

2015-12-09 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2122:
---

 Summary: .NET: Add generic Read/WriteCollection and 
Read/WriteDictionary methods.
 Key: IGNITE-2122
 URL: https://issues.apache.org/jira/browse/IGNITE-2122
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2121) .NET: Port correct collections handling logic.

2015-12-09 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2121:
---

 Summary: .NET: Port correct collections handling logic.
 Key: IGNITE-2121
 URL: https://issues.apache.org/jira/browse/IGNITE-2121
 Project: Ignite
  Issue Type: Sub-task
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2120) Rename Binarylizable -> Binarizable

2015-12-09 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2120:
---

 Summary: Rename Binarylizable -> Binarizable
 Key: IGNITE-2120
 URL: https://issues.apache.org/jira/browse/IGNITE-2120
 Project: Ignite
  Issue Type: Task
  Components: general, interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Priority: Critical
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2153) .NET: Package descriptions are missing in doxygen-generated files.

2015-12-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2153:
---

 Summary: .NET: Package descriptions are missing in 
doxygen-generated files.
 Key: IGNITE-2153
 URL: https://issues.apache.org/jira/browse/IGNITE-2153
 Project: Ignite
  Issue Type: Bug
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-2152) .NET: Introduction page is missing.

2015-12-14 Thread Vladimir Ozerov (JIRA)

Vladimir Ozerov created IGNITE-2152:
---

 Summary: .NET: Introduction page is missing.
 Key: IGNITE-2152
 URL: https://issues.apache.org/jira/browse/IGNITE-2152
 Project: Ignite
  Issue Type: Bug
  Components: interop
Affects Versions: ignite-1.4
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 1.5






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 1017 matches

Mail list logo