Hi Hop Community, I run Hop 2.0.0 locally (openjdk 11.0.15 2022-04-19 LTS) and have a remote server running Hop containerized: docker run -d \ --name=hop \ --hostname=hop \ -v /home/ec2-user/hop/data:/files \ -v /tmp:/tmp \ -e TZ=Europe/Berlin \ -e HOP_SERVER_USER="xx" \ -e HOP_SERVER_PASS="yy" \ -e HOP_SERVER_PORT=8080 \ -e HOP_SERVER_HOSTNAME=0.0.0.0 \ -e HOP_SERVER_METADATA_FOLDER="/files/metadata" \ -e HOP_SHARED_JDBC_FOLDER="/files/jdbc" \ -e HOP_LOG_PATH="/files/log/hop.log" \ -e HOP_SERVER_MAX_LOG_LINES=10000 \ -e HOP_SERVER_MAX_LOG_TIMEOUT=720 \ -e HOP_SERVER_MAX_OBJECT_TIMEOUT=720 \ -e AWS_ACCESS_KEY_ID="abc" \ -e AWS_SECRET_ACCESS_KEY="def" \ -e AWS_REGION="xx" \ --restart unless-stopped \ -m 16G \ apache/hop
I'm running a pipeline which takes table input from mysql and creates parquet output (to s3). Running the pipeline with local run config works great, but executing the same pipeline with remote run config on the server causes the error below. Any hints what might be causing the different behavior locally vs. remotely, and how to fix it, would be greatly appreciated. Thanks in advance! 2022/08/18 10:52:21 - Parquet File Output .0 - ERROR: org.apache.hop.core.exception.HopException: 2022/08/18 10:52:21 - Parquet File Output .0 - Error writing row to parquet file 2022/08/18 10:52:21 - Parquet File Output .0 - newLimit > capacity: (78 > 77) 2022/08/18 10:52:21 - Parquet File Output .0 - 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.hop.parquet.transforms.output.ParquetOutput.processRow(ParquetOutput.java:122) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:51) 2022/08/18 10:52:21 - Parquet File Output .0 - at java.base/java.lang.Thread.run(Thread.java:829) 2022/08/18 10:52:21 - Parquet File Output .0 - Caused by: java.lang.IllegalArgumentException: newLimit > capacity: (78 > 77) 2022/08/18 10:52:21 - Parquet File Output .0 - at java.base/java.nio.Buffer.createLimitException(Buffer.java:372) 2022/08/18 10:52:21 - Parquet File Output .0 - at java.base/java.nio.Buffer.limit(Buffer.java:346) 2022/08/18 10:52:21 - Parquet File Output .0 - at java.base/java.nio.ByteBuffer.limit(ByteBuffer.java:1107) 2022/08/18 10:52:21 - Parquet File Output .0 - at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:235) 2022/08/18 10:52:21 - Parquet File Output .0 - at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:67) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.xerial.snappy.Snappy.compress(Snappy.java:156) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:78) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writeDictionaryPage(ColumnChunkPageWriteStore.java:375) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.column.impl.ColumnWriterBase.finalizeColumnChunk(ColumnWriterBase.java:311) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:188) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.InternalParquetRecordWriter.checkBlockSizeReached(InternalParquetRecordWriter.java:158) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:140) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:310) 2022/08/18 10:52:21 - Parquet File Output .0 - at org.apache.hop.parquet.transforms.output.ParquetOutput.processRow(ParquetOutput.java:118) 2022/08/18 10:52:21 - Parquet File Output .0 - ... 2 more Cheers, Dieter
