[arrow] branch master updated: ARROW-6083: [Java] Refactor Jdbc adapter consume logic
This is an automated email from the ASF dual-hosted git repository.
emkornfield pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 5daba72 ARROW-6083: [Java] Refactor Jdbc adapter consume logic
5daba72 is described below
commit 5daba72aef8810940b5bfadb807851f87c219020
Author: tianchen
AuthorDate: Sun Aug 11 20:06:13 2019 -0700
ARROW-6083: [Java] Refactor Jdbc adapter consume logic
Related to [ARROW-6083](https://issues.apache.org/jira/browse/ARROW-6083).
Jdbc adapter read from ResultSet looks like:
while (rs.next()) {
for (int i = 1; i <= columnCount; i++)
{ jdbcToFieldVector( rs, i, rs.getMetaData().getColumnType(i), rowCount,
root.getVector(rsmd.getColumnName(i)), config); }
rowCount++;
}
And in jdbcToFieldVector has lots of switch-case, that is to see, for every
single value from ResultSet we have to do lots of analyzing conditions.
I think we could optimize this using consumer/delegate like avro adapter.
Closes #4978 from tianchen92/ARROW-6083 and squashes the following commits:
6acff423d fix potential leak
85e0ebd82 resolve comments
1318574e7 remove addNull API
fa13e5971 add constructor with no Calendar
2b0aa78d6 resolve comments
aa5c9d927 ARROW-6083: Refactor Jdbc adapter consume logic
Authored-by: tianchen
Signed-off-by: Micah Kornfield
---
.../arrow/adapter/jdbc/JdbcToArrowUtils.java | 416 +++--
.../arrow/adapter/jdbc/consumer/ArrayConsumer.java | 68
.../adapter/jdbc/consumer/BigIntConsumer.java | 52 +++
.../adapter/jdbc/consumer/BinaryConsumer.java | 99 +
.../arrow/adapter/jdbc/consumer/BitConsumer.java | 52 +++
.../arrow/adapter/jdbc/consumer/BlobConsumer.java | 58 +++
.../arrow/adapter/jdbc/consumer/ClobConsumer.java | 90 +
.../jdbc/consumer/CompositeJdbcConsumer.java | 58 +++
.../arrow/adapter/jdbc/consumer/DateConsumer.java | 64
.../adapter/jdbc/consumer/DecimalConsumer.java | 53 +++
.../adapter/jdbc/consumer/DoubleConsumer.java | 52 +++
.../arrow/adapter/jdbc/consumer/FloatConsumer.java | 52 +++
.../arrow/adapter/jdbc/consumer/IntConsumer.java | 52 +++
.../arrow/adapter/jdbc/consumer/JdbcConsumer.java | 39 ++
.../adapter/jdbc/consumer/SmallIntConsumer.java| 52 +++
.../arrow/adapter/jdbc/consumer/TimeConsumer.java | 64
.../adapter/jdbc/consumer/TimestampConsumer.java | 64
.../adapter/jdbc/consumer/TinyIntConsumer.java | 52 +++
.../adapter/jdbc/consumer/VarCharConsumer.java | 76
19 files changed, 1151 insertions(+), 362 deletions(-)
diff --git
a/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java
b/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java
index 4d7f6ea..3ccdfad 100644
---
a/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java
+++
b/java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java
@@ -20,14 +20,7 @@ package org.apache.arrow.adapter.jdbc;
import static org.apache.arrow.vector.types.FloatingPointPrecision.DOUBLE;
import static org.apache.arrow.vector.types.FloatingPointPrecision.SINGLE;
-import java.io.ByteArrayInputStream;
import java.io.IOException;
-import java.io.InputStream;
-import java.math.BigDecimal;
-import java.nio.charset.StandardCharsets;
-import java.sql.Array;
-import java.sql.Blob;
-import java.sql.Clob;
import java.sql.Date;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
@@ -43,6 +36,24 @@ import java.util.Locale;
import java.util.Map;
import java.util.TimeZone;
+import org.apache.arrow.adapter.jdbc.consumer.ArrayConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.BigIntConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.BinaryConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.BitConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.BlobConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.ClobConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.CompositeJdbcConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.DateConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.DecimalConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.DoubleConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.FloatConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.IntConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.JdbcConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.SmallIntConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.TimeConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.TimestampConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.TinyIntConsumer;
+import org.apache.arrow.adapter.jdbc.consumer.VarCharConsumer;
import org.apache.arrow.memory.RootAllocator;
[arrow] branch master updated: ARROW-6197: [GLib] Add garrow_decimal128_rescale()
This is an automated email from the ASF dual-hosted git repository.
shiro pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 803c818 ARROW-6197: [GLib] Add garrow_decimal128_rescale()
803c818 is described below
commit 803c818e1f50161c2e0962ac5437d2b29125edc7
Author: Sutou Kouhei
AuthorDate: Mon Aug 12 06:59:53 2019 +0900
ARROW-6197: [GLib] Add garrow_decimal128_rescale()
Closes #5057 from kou/glib-decimal128-rescale and squashes the following
commits:
487b167a4 Add garrow_decimal128_rescale()
Authored-by: Sutou Kouhei
Signed-off-by: Yosuke Shiro
---
c_glib/arrow-glib/decimal128.cpp | 33 -
c_glib/arrow-glib/decimal128.h | 6 ++
c_glib/test/test-decimal128.rb | 18 +-
3 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/c_glib/arrow-glib/decimal128.cpp b/c_glib/arrow-glib/decimal128.cpp
index 32bdf5f..92d5ebd 100644
--- a/c_glib/arrow-glib/decimal128.cpp
+++ b/c_glib/arrow-glib/decimal128.cpp
@@ -420,7 +420,7 @@ garrow_decimal128_divide(GArrowDecimal128 *left,
arrow_decimal_left->Divide(*arrow_decimal_right,
_result_raw,
_remainder_raw);
- if (garrow_error_check(error, status, "[decimal][divide]")) {
+ if (garrow_error_check(error, status, "[decimal128][divide]")) {
if (remainder) {
auto arrow_remainder =
std::make_shared(arrow_remainder_raw);
@@ -436,6 +436,37 @@ garrow_decimal128_divide(GArrowDecimal128 *left,
}
}
+/**
+ * garrow_decimal128_rescale:
+ * @decimal: A #GArrowDecimal128.
+ * @original_scale: A scale to be converted from.
+ * @new_scale: A scale to be converted to.
+ * @error: (nullable): Return location for a #GError or %NULL.
+ *
+ * Returns: (nullable) (transfer full): The rescaled decimal or %NULL on error.
+ *
+ * Since: 0.15.0
+ */
+GArrowDecimal128 *
+garrow_decimal128_rescale(GArrowDecimal128 *decimal,
+ gint32 original_scale,
+ gint32 new_scale,
+ GError **error)
+{
+ auto arrow_decimal = garrow_decimal128_get_raw(decimal);
+ arrow::Decimal128 arrow_rescaled_decimal_raw;
+ auto status = arrow_decimal->Rescale(original_scale,
+ new_scale,
+ _rescaled_decimal_raw);
+ if (garrow_error_check(error, status, "[decimal128][rescale]")) {
+auto arrow_rescaled_decimal =
+ std::make_shared(arrow_rescaled_decimal_raw);
+return garrow_decimal128_new_raw(_rescaled_decimal);
+ } else {
+return NULL;
+ }
+}
+
G_END_DECLS
GArrowDecimal128 *
diff --git a/c_glib/arrow-glib/decimal128.h b/c_glib/arrow-glib/decimal128.h
index e7601a4..7079737 100644
--- a/c_glib/arrow-glib/decimal128.h
+++ b/c_glib/arrow-glib/decimal128.h
@@ -72,5 +72,11 @@ GArrowDecimal128 *garrow_decimal128_divide(GArrowDecimal128
*left,
GArrowDecimal128 *right,
GArrowDecimal128 **remainder,
GError **error);
+GARROW_AVAILABLE_IN_0_15
+GArrowDecimal128 *
+garrow_decimal128_rescale(GArrowDecimal128 *decimal,
+ gint32 original_scale,
+ gint32 new_scale,
+ GError **error);
G_END_DECLS
diff --git a/c_glib/test/test-decimal128.rb b/c_glib/test/test-decimal128.rb
index de9453c..0e4bc82 100644
--- a/c_glib/test/test-decimal128.rb
+++ b/c_glib/test/test-decimal128.rb
@@ -101,7 +101,7 @@ class TestDecimal128 < Test::Unit::TestCase
decimal1 = Arrow::Decimal128.new(23423445)
decimal2 = Arrow::Decimal128.new(0)
message =
- "[decimal][divide]: Invalid: Division by 0 in Decimal128"
+ "[decimal128][divide]: Invalid: Division by 0 in Decimal128"
assert_raise(Arrow::Error::Invalid.new(message)) do
decimal1.divide(decimal2)
end
@@ -203,4 +203,20 @@ class TestDecimal128 < Test::Unit::TestCase
decimal >= decimal
])
end
+
+ def test_rescale
+decimal = Arrow::Decimal128.new(10)
+assert_equal(Arrow::Decimal128.new(1000),
+ decimal.rescale(1, 3))
+ end
+
+ def test_rescale_fail
+decimal = Arrow::Decimal128.new(10)
+message =
+ "[decimal128][rescale]: Invalid: " +
+ "Rescaling decimal value would cause data loss"
+assert_raise(Arrow::Error::Invalid.new(message)) do
+ decimal.rescale(1, -1)
+end
+ end
end
[arrow] branch master updated (34dd3ed -> 102ba09)
This is an automated email from the ASF dual-hosted git repository.
shiro pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git.
from 34dd3ed ARROW-6097: [Java] Avro adapter implement unions type
add 102ba09 ARROW-6196: [Ruby] Add support for building
Arrow::TimeNNArray by .new
No new revisions were added by this update.
Summary of changes:
ruby/red-arrow/lib/arrow/loader.rb | 2 ++
...compression-type.rb => time32-array-builder.rb} | 26 ---
...compression-type.rb => time64-array-builder.rb} | 26 ---
...st-time32-data-type.rb => test-time32-array.rb} | 38 --
...st-time32-data-type.rb => test-time64-array.rb} | 38 --
5 files changed, 72 insertions(+), 58 deletions(-)
copy ruby/red-arrow/lib/arrow/{compression-type.rb => time32-array-builder.rb}
(67%)
copy ruby/red-arrow/lib/arrow/{compression-type.rb => time64-array-builder.rb}
(67%)
copy ruby/red-arrow/test/{test-time32-data-type.rb => test-time32-array.rb}
(56%)
copy ruby/red-arrow/test/{test-time32-data-type.rb => test-time64-array.rb}
(56%)
