[
https://issues.apache.org/jira/browse/FLINK-32565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanyu Zheng updated FLINK-32565:
--------------------------------
Description:
We are undertaking a task that requires casting from the DOUBLE type to BYTES
In particular, we have a INTEGER 1234. Our current approach is to convert this
INTEGER to BYTES using the following SQL query:
{code:java}
SELECT CAST(1234 as BYTES);{code}
{{ }}
However, we encounter an issue when executing this query, potentially due to an
error in the conversion between INTEGER and BYTES. Our goal is to identify and
correct this issue so that our query can execute successfully. The tasks
involved are:
# Investigate and pinpoint the specific reason for the conversion failure from
INTEGER to BYTES.
# Design and implement a solution that enables our query to function correctly.
# Test this solution across all required scenarios to ensure its robustness.
see also:
1. PostgreSQL: PostgreSQL supports casting from NUMBER types (INTEGER, BIGINT,
DECIMAL, etc.) to BYTES type (BYTEA). In PostgreSQL, you can use CAST or
TO_BINARY function for performing the conversion. URL:
[https://www.postgresql.org/docs/current/sql-expressions.html#SQL-SYNTAX-TYPE-CASTS]
2. MySQL: MySQL supports casting from NUMBER types (INTEGER, BIGINT, DECIMAL,
etc.) to BYTES type (BINARY or BLOB). In MySQL, you can use CAST or CONVERT
functions for performing the conversion. URL:
[https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html]
3. Microsoft SQL Server: SQL Server supports casting from NUMBER types (INT,
BIGINT, NUMERIC, etc.) to BYTES type (VARBINARY or IMAGE). You can use CAST or
CONVERT functions for performing the conversion. URL:
[https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql]
4. Oracle Database: Oracle supports casting from NUMBER types (NUMBER, INTEGER,
FLOAT, etc.) to BYTES type (RAW). You can use UTL_RAW.CAST_TO_RAW function for
performing the conversion. URL:
[https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/TO_BINARY_DOUBLE.html]
for the problem of bytes order may arise (little vs big endian).
1. Apache Hadoop: Hadoop, being an open-source framework, has to deal with byte
order issues across different platforms and architectures. The Hadoop File
System (HDFS) uses a technique called "sequence files," which include metadata
to describe the byte order of the data. This metadata ensures that data is read
and written correctly, regardless of the endianness of the platform.
2. Apache Avro: Avro is a data serialization system used by various big data
frameworks like Hadoop and Apache Kafka. Avro uses a compact binary encoding
format that includes a marker for the byte order. This allows Avro to handle
endianness issues seamlessly when data is exchanged between systems with
different byte orders.
3. Apache Parquet: Parquet is a columnar storage format used in big data
processing frameworks like Apache Spark. Parquet uses a little-endian format
for encoding numeric values, which is the most common format on modern systems.
When reading or writing Parquet data, data processing engines typically handle
any necessary byte order conversions transparently.
4. Apache Spark: Spark is a popular big data processing engine that can handle
data on distributed systems. It relies on the underlying data formats it reads
(e.g., Avro, Parquet, ORC) to manage byte order issues. These formats are
designed to handle byte order correctly, ensuring that Spark can handle data
correctly on different platforms.
5. Google Cloud BigQuery: BigQuery is a serverless data warehouse offered by
Google Cloud. When dealing with binary data and endianness, BigQuery relies on
the data encoding format. For example, when loading data in Avro or Parquet
formats, these formats already include byte order information, allowing
BigQuery to handle data across different platforms correctly.
was:
We are undertaking a task that requires casting from the DOUBLE type to BYTES
In particular, we have a INTEGER 1234. Our current approach is to convert this
INTEGER to BYTES using the following SQL query:
{code:java}
SELECT CAST(1234 as BYTES);{code}
{{ }}
However, we encounter an issue when executing this query, potentially due to an
error in the conversion between INTEGER and BYTES. Our goal is to identify and
correct this issue so that our query can execute successfully. The tasks
involved are:
# Investigate and pinpoint the specific reason for the conversion failure from
INTEGER to BYTES.
# Design and implement a solution that enables our query to function correctly.
# Test this solution across all required scenarios to ensure its robustness.
see also:
1. PostgreSQL: PostgreSQL supports casting from NUMBER types (INTEGER, BIGINT,
DECIMAL, etc.) to BYTES type (BYTEA). In PostgreSQL, you can use CAST or
TO_BINARY function for performing the conversion. URL:
[https://www.postgresql.org/docs/current/sql-expressions.html#SQL-SYNTAX-TYPE-CASTS]
2. MySQL: MySQL supports casting from NUMBER types (INTEGER, BIGINT, DECIMAL,
etc.) to BYTES type (BINARY or BLOB). In MySQL, you can use CAST or CONVERT
functions for performing the conversion. URL:
[https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html]
3. Microsoft SQL Server: SQL Server supports casting from NUMBER types (INT,
BIGINT, NUMERIC, etc.) to BYTES type (VARBINARY or IMAGE). You can use CAST or
CONVERT functions for performing the conversion. URL:
[https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql]
4. Oracle Database: Oracle supports casting from NUMBER types (NUMBER, INTEGER,
FLOAT, etc.) to BYTES type (RAW). You can use UTL_RAW.CAST_TO_RAW function for
performing the conversion. URL:
[https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/TO_BINARY_DOUBLE.html]
> Support cast from NUMBER to BYTES
> ---------------------------------
>
> Key: FLINK-32565
> URL: https://issues.apache.org/jira/browse/FLINK-32565
> Project: Flink
> Issue Type: Sub-task
> Reporter: Hanyu Zheng
> Assignee: Hanyu Zheng
> Priority: Major
> Labels: pull-request-available
>
> We are undertaking a task that requires casting from the DOUBLE type to BYTES
> In particular, we have a INTEGER 1234. Our current approach is to convert
> this INTEGER to BYTES using the following SQL query:
> {code:java}
> SELECT CAST(1234 as BYTES);{code}
> {{ }}
> However, we encounter an issue when executing this query, potentially due to
> an error in the conversion between INTEGER and BYTES. Our goal is to identify
> and correct this issue so that our query can execute successfully. The tasks
> involved are:
> # Investigate and pinpoint the specific reason for the conversion failure
> from INTEGER to BYTES.
> # Design and implement a solution that enables our query to function
> correctly.
> # Test this solution across all required scenarios to ensure its robustness.
>
> see also:
> 1. PostgreSQL: PostgreSQL supports casting from NUMBER types (INTEGER,
> BIGINT, DECIMAL, etc.) to BYTES type (BYTEA). In PostgreSQL, you can use CAST
> or TO_BINARY function for performing the conversion. URL:
> [https://www.postgresql.org/docs/current/sql-expressions.html#SQL-SYNTAX-TYPE-CASTS]
> 2. MySQL: MySQL supports casting from NUMBER types (INTEGER, BIGINT, DECIMAL,
> etc.) to BYTES type (BINARY or BLOB). In MySQL, you can use CAST or CONVERT
> functions for performing the conversion. URL:
> [https://dev.mysql.com/doc/refman/8.0/en/cast-functions.html]
> 3. Microsoft SQL Server: SQL Server supports casting from NUMBER types (INT,
> BIGINT, NUMERIC, etc.) to BYTES type (VARBINARY or IMAGE). You can use CAST
> or CONVERT functions for performing the conversion. URL:
> [https://docs.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql]
> 4. Oracle Database: Oracle supports casting from NUMBER types (NUMBER,
> INTEGER, FLOAT, etc.) to BYTES type (RAW). You can use UTL_RAW.CAST_TO_RAW
> function for performing the conversion. URL:
> [https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/TO_BINARY_DOUBLE.html]
>
> for the problem of bytes order may arise (little vs big endian).
>
> 1. Apache Hadoop: Hadoop, being an open-source framework, has to deal with
> byte order issues across different platforms and architectures. The Hadoop
> File System (HDFS) uses a technique called "sequence files," which include
> metadata to describe the byte order of the data. This metadata ensures that
> data is read and written correctly, regardless of the endianness of the
> platform.
> 2. Apache Avro: Avro is a data serialization system used by various big data
> frameworks like Hadoop and Apache Kafka. Avro uses a compact binary encoding
> format that includes a marker for the byte order. This allows Avro to handle
> endianness issues seamlessly when data is exchanged between systems with
> different byte orders.
> 3. Apache Parquet: Parquet is a columnar storage format used in big data
> processing frameworks like Apache Spark. Parquet uses a little-endian format
> for encoding numeric values, which is the most common format on modern
> systems. When reading or writing Parquet data, data processing engines
> typically handle any necessary byte order conversions transparently.
> 4. Apache Spark: Spark is a popular big data processing engine that can
> handle data on distributed systems. It relies on the underlying data formats
> it reads (e.g., Avro, Parquet, ORC) to manage byte order issues. These
> formats are designed to handle byte order correctly, ensuring that Spark can
> handle data correctly on different platforms.
> 5. Google Cloud BigQuery: BigQuery is a serverless data warehouse offered by
> Google Cloud. When dealing with binary data and endianness, BigQuery relies
> on the data encoding format. For example, when loading data in Avro or
> Parquet formats, these formats already include byte order information,
> allowing BigQuery to handle data across different platforms correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
