[
https://issues.apache.org/jira/browse/IMPALA-7044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482775#comment-16482775
]
Tim Armstrong commented on IMPALA-7044:
---------------------------------------
Yeah, so regardless I think we should pick an upper limit and make sure that we
test up to that limit to be sure it works well as part of fixing this bug.
> int32 overflow in HdfsTableSink::CreateNewTmpFile()
> ---------------------------------------------------
>
> Key: IMPALA-7044
> URL: https://issues.apache.org/jira/browse/IMPALA-7044
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0,
> Impala 2.12.0, Impala 2.13.0
> Reporter: Lars Volker
> Priority: Critical
> Labels: parquet
> Attachments: ct.sql
>
>
> When writing Parquet files we compute a minimum block size based on the
> number of columns in the target table in
> [hdfs-parquet-table-writer.cc:916|https://github.com/apache/impala/blob/master/be/src/exec/hdfs-parquet-table-writer.cc?utf8=%E2%9C%93#L916]:
> {noformat}
> 3 * DEFAULT_DATA_PAGE_SIZE * columns_.size();
> {noformat}
> For tables with a large number of columns (> ~10k), this value will get
> larger than 2GB. When we pass it to {{hdfsOpenFile()}} in
> {{HdfsTableSink::CreateNewTmpFile()}} it gets cast to a signed int32 and can
> overflow.
> This leads to error messages like the following:
> {noformat}
> I0516 16:13:52.755090 24257 status.cc:125] Failed to open HDFS file for
> writing:
> hdfs://localhost:20500/test-warehouse/lv.db/a/_impala_insert_staging/3c417cb973b710ab_803e898000000000/.3c417cb973b710ab-80
> 3e898000000000_411033576_dir/3c417cb973b710ab-803e898000000000_271567064_data.0.parq
> Error(255): Unknown error 255
> Root cause: RemoteException: Specified block size is less than configured
> minimum value (dfs.namenode.fs-limits.min-block-size): -1935671296 < 1024
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2417)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2339)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> @ 0x187b8b3 impala::Status::Status()
> @ 0x1fade89 impala::HdfsTableSink::CreateNewTmpFile()
> @ 0x1faeee7 impala::HdfsTableSink::InitOutputPartition()
> @ 0x1fb1389 impala::HdfsTableSink::GetOutputPartition()
> @ 0x1faf34a impala::HdfsTableSink::Send()
> @ 0x1c91bcd impala::FragmentInstanceState::ExecInternal()
> @ 0x1c8efa5 impala::FragmentInstanceState::Exec()
> @ 0x1c9e53f impala::QueryState::ExecFInstance()
> @ 0x1c9cdb2
> _ZZN6impala10QueryState15StartFInstancesEvENKUlvE_clEv
> @ 0x1c9f25d
> _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala10QueryState15StartFInstancesEvEUlvE_vE6invokeERNS1_15function_bufferE
> @ 0x1bd6cd4 boost::function0<>::operator()()
> @ 0x1ec18f9 impala::Thread::SuperviseThread()
> @ 0x1ec9a95 boost::_bi::list5<>::operator()<>()
> @ 0x1ec99b9 boost::_bi::bind_t<>::operator()()
> @ 0x1ec997c boost::detail::thread_data<>::run()
> @ 0x31a527a thread_proxy
> @ 0x7f30246a8184 start_thread
> @ 0x7f30243d503d clone
> {noformat}
> The signature of {{hdfsOpenFile()}} is as follows:
> {noformat}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags, int bufferSize,
> short replication, tSize blocksize);
> {noformat}
> {{tSize}} is typedef'd to {{int32_t}}.
> The comment of {{hdfsOpenFile()}} is explicit about this:
> {noformat}
> @param blocksize Size of block - pass 0 if you want to use the
> default configured values. Note that if you want a block size bigger
> than 2 GB, you must use the hdfsStreamBuilder API rather than this
> deprecated function.
> {noformat}
> If using {{hdfsStreamBuilder}} is not an option, we should be able to cap the
> blocksize to 2GB (or smaller values). It might result in a suboptimal storage
> layout, but will preserve correctness.
> An alternative would be to cap the maximum number of columns. In either case
> we should be explicit about the signed overflow, as it results in undefined
> behavior.
> I'm attaching a SQL file which creates a table with 11k columns and inserts a
> row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
