[
https://issues.apache.org/jira/browse/BEAM-13242?focusedWorklogId=682846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-682846
]
ASF GitHub Bot logged work on BEAM-13242:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Nov/21 18:25
Start Date: 17/Nov/21 18:25
Worklog Time Spent: 10m
Work Description: kennknowles commented on a change in pull request
#15987:
URL: https://github.com/apache/beam/pull/15987#discussion_r751524811
##########
File path:
sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java
##########
@@ -266,8 +266,8 @@ private FixedPrecisionNumeric(
@Override
public BigDecimal toInputType(BigDecimal base) {
checkArgument(
- base == null || (base.precision() == precision && base.scale() ==
scale),
- "Expected BigDecimal base to be null or have precision = %s (was
%s), scale = %s (was %s)",
+ base == null || (base.precision() <= precision && base.scale() <=
scale),
Review comment:
Incidentally it took me a minute to understand what is happening here
because I don't know why it is called `base`. This is converting the value
returned by JDBC to the value expected by Beam schema, right?
I am having trouble wrapping my head around all the cases here, or coming up
with the simple math. The inefficient but simple check here is `base.round(new
MathContext(precision)).compareTo(base) == 0`.
If the scales are equal, then lower precision is fine, certainly.
But you could even have "1000 x 10^-3" (precision 3, scale 3) and it can
safely coerce to "1 x 10^0" (precision 1, scale 0). Am I getting this wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 682846)
Time Spent: 1h 10m (was: 1h)
> IllegalArgumentException while reading a Numeric column with a fixed precision
> ------------------------------------------------------------------------------
>
> Key: BEAM-13242
> URL: https://issues.apache.org/jira/browse/BEAM-13242
> Project: Beam
> Issue Type: Bug
> Components: io-java-jdbc
> Affects Versions: 2.33.0
> Environment: openjdk version "11.0.10" 2021-01-19
> macOS 11.1
> Reporter: Denis
> Priority: P2
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> *Error*
>
> I am trying to read a table in a Postgresql database. One of columns has a
> type definition
> {code:java}
> Numeric(15, 2){code}
>
> At runtime I am getting:
> {code:java}
> Caused by: java.lang.IllegalArgumentException: Expected BigDecimal base to be
> null or have precision = 15 (was 6), scale = 2 (was 2) at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:477)
> at
> org.apache.beam.sdk.io.jdbc.LogicalTypes$FixedPrecisionNumeric.toInputType(LogicalTypes.java:268)
> at
> org.apache.beam.sdk.io.jdbc.LogicalTypes$FixedPrecisionNumeric.toInputType(LogicalTypes.java:246)
> at
> org.apache.beam.sdk.io.jdbc.SchemaUtil.lambda$createLogicalTypeExtractor$ca0ab2ec$1(SchemaUtil.java:289)
> at
> org.apache.beam.sdk.io.jdbc.SchemaUtil$BeamRowMapper.mapRow(SchemaUtil.java:354)
> at
> org.apache.beam.sdk.io.jdbc.SchemaUtil$BeamRowMapper.mapRow(SchemaUtil.java:332)
> at
> org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.processElement(JdbcIO.java:1172)
> {code}
>
> It seems that when JdbcIO reads a numeric column it might have smaller
> precision/scale than defined in DDL.
>
> *How to reproduce*
>
> {code:java}
> import org.apache.beam.sdk.io.jdbc.JdbcIO
> import org.apache.beam.sdk.testing.PAssert
> import org.apache.beam.sdk.testing.TestPipeline
> import org.apache.beam.sdk.transforms.MapElements
> import org.apache.beam.sdk.transforms.ProcessFunction
> import org.apache.beam.sdk.values.TypeDescriptors
> import org.junit.AfterClass
> import org.junit.BeforeClass
> import org.junit.Test
> import org.testcontainers.containers.PostgreSQLContainer
> import java.math.BigDecimal
> import java.sql.Connection
> import java.sql.DriverManager
> class PostgresNumericBugTest {
> companion object {
> val pgContainer = PostgreSQLContainer<Nothing>("postgres:14.0")
> @BeforeClass
> @JvmStatic
> fun start() {
> pgContainer.start()
> getConnection().use {
> val stmt = it.createStatement()
> stmt.execute(
> """create table account (
> id integer not null primary key,
> balance numeric(15, 2)
> )
> """.trimIndent()
> )
> stmt.execute("insert into account(id, balance) values(1,
> 5755.94)")
> }
> }
> @AfterClass
> @JvmStatic
> fun stop() {
> pgContainer.stop()
> }
> private fun getConnection(): Connection {
> return DriverManager.getConnection(
> pgContainer.jdbcUrl, pgContainer.username,
> pgContainer.password
> )
> }
> private fun getDataSourceConfiguration():
> JdbcIO.DataSourceConfiguration =
> JdbcIO.DataSourceConfiguration.create(
> pgContainer.driverClassName,
> pgContainer.jdbcUrl
> )
> .withUsername(pgContainer.username)
> .withPassword(pgContainer.password)
> }
> @Test
> fun readNumeric() {
> val pipeline =
> TestPipeline.create().enableAbandonedNodeEnforcement(false)
> val pBalance = pipeline
> .apply(
> "Read Account from DB",
> JdbcIO
> .readRows()
> .withQuery("select balance from account")
> .withDataSourceConfiguration(getDataSourceConfiguration())
> )
> .apply(
> "Get Balance",
>
> MapElements.into(TypeDescriptors.bigdecimals()).via(ProcessFunction {
> it.getDecimal(0) })
> )
> PAssert.that(pBalance).containsInAnyOrder(BigDecimal("5755.94"))
> pipeline.run()
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
