HyukjinKwon commented on a change in pull request #25379: [SPARK-28474][SQL]
Hive 0.12 JDBC client can not handle binary type
URL: https://github.com/apache/spark/pull/25379#discussion_r314940083
##########
File path:
sql/hive-thriftserver/v1.2.1/src/main/java/org/apache/hive/service/cli/ColumnValue.java
##########
@@ -195,7 +197,8 @@ public static TColumnValue toTColumnValue(Type type,
Object value) {
case DECIMAL_TYPE:
return stringValue(((HiveDecimal)value));
case BINARY_TYPE:
- return stringValue((String)value);
+ String strVal = value == null ? null :
UTF8String.fromBytes((byte[])value).toString();
+ return stringValue(strVal);
Review comment:
@wangyum, https://github.com/apache/spark/pull/25480 this might be the
cause. `UTF8String.toString()`.
```
new String(getBytes(), StandardCharsets.UTF_8);
```
It actually reads bytes with UTF-8. IIRC, Java mangles the data if it's not
conformed as specified encoding.
For some protocols, it might be unable to recognise those mangled strings
back to binary. It might be the cause.
BTW, we should allow arbitrary binary, for instance, for the use case like
images.
If you are unable to find the root cause, we can partially revert to the
failed protocols for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
