pitrou commented on a change in pull request #7611:
URL: https://github.com/apache/arrow/pull/7611#discussion_r449081453
##########
File path: r/src/array_from_vector.cpp
##########
@@ -1150,6 +1150,25 @@ std::shared_ptr<arrow::DataType>
InferArrowTypeFromVector<REALSXP>(SEXP x) {
return float64();
}
+template <>
+std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<STRSXP>(SEXP x) {
+ // See how big the character vector is
+ R_xlen_t n = XLENGTH(x);
+ int64_t size = 0;
+ for (R_xlen_t i = 0; i < n; i++) {
+ SEXP string_i = STRING_ELT(x, i);
+ if (string_i != NA_STRING) {
+ size += XLENGTH(Rf_mkCharCE(Rf_translateCharUTF8(string_i), CE_UTF8));
+ }
+ if (size > 2147483646) {
+ // Exceeds 2GB capacity of utf8 type, so use large
+ return large_utf8();
+ }
+ }
+
+ return utf8();
+}
Review comment:
Yes, ideally we'd have a `AdaptiveStringBuilder`-like clas.
##########
File path: r/src/array_from_vector.cpp
##########
@@ -1150,6 +1150,25 @@ std::shared_ptr<arrow::DataType>
InferArrowTypeFromVector<REALSXP>(SEXP x) {
return float64();
}
+template <>
+std::shared_ptr<arrow::DataType> InferArrowTypeFromVector<STRSXP>(SEXP x) {
+ // See how big the character vector is
+ R_xlen_t n = XLENGTH(x);
+ int64_t size = 0;
+ for (R_xlen_t i = 0; i < n; i++) {
+ SEXP string_i = STRING_ELT(x, i);
+ if (string_i != NA_STRING) {
+ size += XLENGTH(Rf_mkCharCE(Rf_translateCharUTF8(string_i), CE_UTF8));
+ }
+ if (size > 2147483646) {
+ // Exceeds 2GB capacity of utf8 type, so use large
+ return large_utf8();
+ }
+ }
+
+ return utf8();
+}
Review comment:
Yes, ideally we'd have a `AdaptiveStringBuilder`-like class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
