[jira] [Created] (ARROW-1761) Multi argument operator kernel behavior for decimal columns

[Phillip Cloud (JIRA)]

Phillip Cloud created ARROW-1761:


 Summary: Multi argument operator kernel behavior for decimal 
columns
 Key: ARROW-1761
 URL: https://issues.apache.org/jira/browse/ARROW-1761
 Project: Apache Arrow
  Issue Type: Bug
  Components: C++, Java - Vectors
Affects Versions: 0.7.1
Reporter: Phillip Cloud
Assignee: Phillip Cloud
Priority: Major


This is a JIRA to discuss the behavior of operator kernels that require more 
than one decimal column input where the column types have a different {{scale}} 
parameter.

For example:

{code}
a: decimal(12, 2)
b: decimal(10, 3)

c = a + b
{code}

Arithmetic is the primary use case, but anything that needs to efficiently 
operate on decimal columns with different scales would require this 
functionality.

I imagine that @jacques-n and folks at Dremio have thought about and solved the 
problem in Java. If so, we should consider implementing this behavior in C++. 
Otherwise, I'll do a bit of reading and digging to see how existing systems 
efficiently handle this problem.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Arrow sync today

[Siddharth Teotia]

There was no meeting today.

On Wed, Nov 1, 2017 at 10:14 AM, Li Jin  wrote:

> I wasn't able to join the chat room so not sure what's going on. Did we
> have the meeting?
> On Wed, Nov 1, 2017 at 12:55 PM Bryan Cutler  wrote:
>
> > Sorry, I won't be able to make today's call either.  I have been working
> on
> > ARROW-1047 to make a generic message interface for stream format in Java.
> > I'm just curious about where we are at with the Java refactoring. Thanks!
> >
> > On Nov 1, 2017 9:07 AM, "Uwe L. Korn"  wrote:
> >
> > > Due to a public holiday in Germany today I'm also unable to join. Short
> > > heads-up from me : i'm still working on packaging problems with the
> > Python
> > > wheels and on selective categorical conversion for Arrow-> Pandas.
> > >
> > > Uwe
> > >
> > > > Am 01.11.2017 um 16:43 schrieb Wes McKinney :
> > > >
> > > > I am not able to attend today’s Arrow sync. Others are free to meet
> and
> > > > relay notes to the mailing list
> > >
> > >
> >
>

Re: Arrow sync today

[Li Jin]

I wasn't able to join the chat room so not sure what's going on. Did we
have the meeting?
On Wed, Nov 1, 2017 at 12:55 PM Bryan Cutler  wrote:

> Sorry, I won't be able to make today's call either.  I have been working on
> ARROW-1047 to make a generic message interface for stream format in Java.
> I'm just curious about where we are at with the Java refactoring. Thanks!
>
> On Nov 1, 2017 9:07 AM, "Uwe L. Korn"  wrote:
>
> > Due to a public holiday in Germany today I'm also unable to join. Short
> > heads-up from me : i'm still working on packaging problems with the
> Python
> > wheels and on selective categorical conversion for Arrow-> Pandas.
> >
> > Uwe
> >
> > > Am 01.11.2017 um 16:43 schrieb Wes McKinney :
> > >
> > > I am not able to attend today’s Arrow sync. Others are free to meet and
> > > relay notes to the mailing list
> >
> >
>

Re: JDBC Adapter for Apache-Arrow

[Julian Hyde]

http://lmgtfy.com/?q=unsubscribe+apache+arrow 
 

> On Oct 31, 2017, at 5:20 PM, 丁锦祥  wrote:
> 
> unsubscribe
> 
> On Tue, Oct 31, 2017 at 4:28 PM, Julian Hyde  wrote:
> 
>> Yeah, I agree, it should be an interface defined as part of Arrow. Not
>> driver-specific.
>> 
>>> On Oct 31, 2017, at 1:37 PM, Laurent Goujon  wrote:
>>> 
>>> I really like Julian's idea of unwrapping Arrow objects out of the JDBC
>>> ResultSet, but I wonder if the unwrap class has to be specific to the
>>> driver and if an interface can be designed to be used by multiple
>> drivers:
>>> for drivers based on Arrow, it means you could totally skip the
>>> serialization/deserialization from/to JDBC records.
>>> If such an interface exists, I would propose to add it to the Arrow
>>> project, with Arrow product/projects in charge of adding support for it
>> in
>>> their own JDBC driver.
>>> 
>>> Laurent
>>> 
>>> On Tue, Oct 31, 2017 at 1:18 PM, Atul Dambalkar <
>> atul.dambal...@xoriant.com>
>>> wrote:
>>> 
 Thanks for your thoughts Julian. I think, adding support for Arrow
>> objects
 for Avatica Remote Driver (AvaticaToArrowConverter) can be certainly
>> taken
 up as another activity. And you are right, we will have to look at
>> specific
 JDBC driver to really optimize it individually.
 
 I would be curious if there are any further inputs/comments from other
>> Dev
 folks, on the JDBC adapter aspect.
 
 -Atul
 
 -Original Message-
 From: Julian Hyde [mailto:jh...@apache.org]
 Sent: Tuesday, October 31, 2017 11:12 AM
 To: dev@arrow.apache.org
 Subject: Re: JDBC Adapter for Apache-Arrow
 
 Sorry I didn’t read your email thoroughly enough. I was talking about
>> the
 inverse (JDBC reading from Arrow) whereas you are talking about Arrow
 reading from JDBC. Your proposal makes perfect sense.
 
 JDBC is quite a chatty interface (a call for every column of every row,
 plus an occasional call to find out whether values are null, and objects
 such as strings and timestamps become a Java heap object) so for
>> specific
 JDBC drivers it may be possible to optimize. For example, the Avatica
 remove driver receives row sets in an RPC response in protobuf format.
>> It
 may be useful if the JDBC driver were able to expose a direct path from
 protobuf to Arrow. "ResultSet.unwrap(AvaticaToArrowConverter.class)”
 might be one way to achieve this.
 
 Julian
 
 
 
 
> On Oct 31, 2017, at 10:41 AM, Atul Dambalkar <
>> atul.dambal...@xoriant.com>
 wrote:
> 
> Hi Julian,
> 
> Thanks for your response. If I understand correctly (looking at other
 adapters), Calcite-Arrow adapter would provide SQL front end for
>> in-memory
 Arrow data objects/structures. So from that perspective, are you
>> suggesting
 building the Calcite-Arrow adapter?
> 
> In this case, what we are saying is to provide a mechanism for upstream
 apps to be able to get/create Arrow objects/structures from a relational
 database. This would also mean converting row like data from a SQL
>> Database
 to columnar Arrow data structures. The utility may be, can make use of
 JDBC's MetaData features to figure out the underlying DB schema and
>> define
 Arrow columnar schema. Also underlying database in this case would be
>> any
 relational DB and hence would be persisted to the disk, but the Arrow
 objects being in-memory can be ephemeral.
> 
> Please correct me if I am missing anything.
> 
> -Atul
> 
> -Original Message-
> From: Julian Hyde [mailto:jhyde.apa...@gmail.com]
> Sent: Monday, October 30, 2017 7:50 PM
> To: dev@arrow.apache.org
> Subject: Re: JDBC Adapter for Apache-Arrow
> 
> How about writing an Arrow adapter for Calcite? I think it amounts to
 the same thing - you would inherit Calcite’s SQL parser and Avatica JDBC
 stack.
> 
> Would this database be ephemeral (i.e. would the data go away when you
 close the connection)? If not, how would you know where to load the data
 from?
> 
> Julian
> 
>> On Oct 30, 2017, at 6:17 PM, Atul Dambalkar <
>> atul.dambal...@xoriant.com>
 wrote:
>> 
>> Hi all,
>> 
>> I wanted to open up a conversation here regarding developing a
 Java-based JDBC Adapter for Apache Arrow. I have had a preliminary
 discussion with Wes McKinney and Siddharth Teotia on this a couple weeks
 earlier.
>> 
>> Basically at a high level (over-simplified) this adapter/API will
>> allow
 upstream apps to query RDBMS data over JDBC and get the JDBC objects
 converted to Arrow in-memory (JVM) objects/structures. The upstream
>> utility
 can then work with Arrow objects/structures with usual performance
 benefits. 

Re: Arrow sync today

[Siddharth Teotia]

I have joined the meeting here -- https://meet.google.com/vtm-teks-phx
I don't see anybody. Can someone please send out the correct link?

On Wed, Nov 1, 2017 at 8:43 AM, Wes McKinney  wrote:

> I am not able to attend today’s Arrow sync. Others are free to meet and
> relay notes to the mailing list
>

Arrow sync today

[Wes McKinney]

I am not able to attend today’s Arrow sync. Others are free to meet and
relay notes to the mailing list