Hi Pierre, Thank you for your return.
In order to do lean testing, the default configuration was used generally: * nifi-env.sh file has the default configuration. * bootstrap.conf file has the default configuration. * nifi.properties file has the default configuration except security configuration. * PutParquet Processor has the default configuration except configuration which I mentioned in the previous post. (But SNAPPY compression is not working) * ConvertAvroToParquet Processor has the default configuration. (SNAPPY compression is working correctly) * There is no custom processor in our NiFi environment. * There is no custom lib location in Nifi properties. As I mentioned in my previous email, this error happens only in NiFi 1.14.0 version. Thank you for helping, --Bilal -----Original Message----- From: Pierre Villard <pierre.villard...@gmail.com> Sent: 15 Eylül 2021 Çarşamba 13:50 To: dev@nifi.apache.org Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0) Hi, Is there something you changed on how the Snappy libraries are loaded in the NiFi JVM? I'm not familiar with this specifically but you may need more recent versions of the native libraries. Also, anything related to a snappy-java lib you added somewhere? /lib? custom processor? etc. I believe it may be related to https://github.com/apache/nifi/pull/4601 where we're adding snappy-java and maybe there is a version conflict there. Pierre Le mer. 15 sept. 2021 à 12:38, Bilal Bektas <bilal.bek...@obase.com> a écrit : > Hi Dev Team, > > To be able to detect the problem, is there anything you expect from us > or is there anything we can help with > > Thank you in advance, > > --Bilal > > > -----Original Message----- > From: Bilal Bektas <bilal.bek...@obase.com> > Sent: 1 Eylül 2021 Çarşamba 12:43 > To: dev@nifi.apache.org > Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi > 1.14.0) > > Hi Dev Team, > > I have tested different compression types which is a feature of > PutParquet and ConvertAvroToParquet Processors on different NiFi versions. > > To avoid confusion, i will give a summary information: > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet > Processor works correctly on NiFi 1.12.1 and 1.13.2 > * Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor works > correctly on NiFi 1.14.0; SNAPPY gives an error in the previous email > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of > ConvertAvroToParquet Processor works correctly on NiFi 1.12.1, 1.13.2 and > 1.14.0. > > > PutParquet - Properties: > * Hadoop Configuration Resources: File locations > * Kerberos Credentials Service: Keytab service > * Record Reader: AvroReader Service (Embedded Avro Schema) > * Overwrite Files: True > * Compression Type: SNAPPY > * Other Properties: Default > > Error Log (nifi-app.log): > Class org.xerial.snappy.SnappyNative does not implement the requested > interface org.xerial.snappy.SnappyApi > > > Could you help in order to resolve this situation? > > Thank you in advance, > > --Bilal > > > > -----Original Message----- > From: Bilal Bektas <bilal.bek...@obase.com> > Sent: 3 Ağustos 2021 Salı 14:55 > To: dev@nifi.apache.org > Subject: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0) > > Hi Dev Team, > > SNAPPY compression feature of PutParquet Processor works correctly on > NiFi > 1.12.1 but NiFi 1.14.0 gives the following error. > > Could you help in order to resolve this situation? > > Thank you in advance, > > --Bilal > > > PutParquet - Config: > Hadoop Configuration Resources > > File locations > > Kerberos Credentials Service > > Keytab service > > Kerberos Relogin Period > > 4 Hours > > Record Reader > > AvroReader Service (Embedded Avro Schema) > > Compression Type > > SNAPPY > > > > Error Log (nifi-app.log): > 2021-08-03 14:13:01,955 ERROR [Timer-Driven Process Thread-12] > o.a.nifi.processors.parquet.PutParquet > PutParquet[id=6caab337-68e8-3834-b64a-1d2cbd93aba8] Failed to write > due to > java.lang.IncompatibleClassChangeError: Class > org.xerial.snappy.SnappyNative does not implement the requested > interface > org.xerial.snappy.SnappyApi: java.lang.IncompatibleClassChangeError: > Class org.xerial.snappy.SnappyNative does not implement the requested > interface org.xerial.snappy.SnappyApi > java.lang.IncompatibleClassChangeError: Class > org.xerial.snappy.SnappyNative does not implement the requested > interface org.xerial.snappy.SnappyApi > at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:380) > at > org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67) > at > org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81) > at > org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92) > at > org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167) > at > org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:168) > at > org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59) > at > org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387) > at > org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186) > at > org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29) > at > org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185) > at > org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124) > at > org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319) > at > org.apache.nifi.parquet.hadoop.AvroParquetHDFSRecordWriter.close(AvroParquetHDFSRecordWriter.java:49) > at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:534) > at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:466) > at > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:326) > at > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2466) > at > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2434) > at > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1822) > at > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:271) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > obase > TEL: +90216 527 30 00 > FAX: +90216 527 31 11 > [http://www.obase.com/images/signature/home.png]<http://www.obase.com> > [ http://www.obase.com/images/signature/facebook.png] < > https://www.facebook.com/obasesocial> [ > http://www.obase.com/images/signature/twitter.png] < > https://twitter.com/obasesocial> [ > http://www.obase.com/images/signature/linkedin.png] < > https://tr.linkedin.com/in/obase> > [http://www.obase.com/images/signature/obase25tr.png]<http://www.obase > .com > > > > Bu elektronik posta ve onunla iletilen b?t?n dosyalar sadece > g?ndericisi tarafindan almasi amaclanan yetkili gercek ya da t?zel > kisinin kullanimi icindir. Eger s?z konusu yetkili alici degilseniz bu > elektronik postanin icerigini aciklamaniz, kopyalamaniz, > y?nlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi > derhal silmeniz gerekmektedir. > OBASE bu mesajin icerdigi bilgilerin do?rulu?u veya eksiksiz oldugu > konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin > ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve > saklanmasindan sorumlu degildir. Bu mesajdaki g?r?sler yalnizca > g?nderen kisiye aittir ve OBASE g?r?slerini yansitmayabilir. > > Bu e-posta bilinen b?t?n bilgisayar vir?slerine karsi taranmistir. > > This e-mail and any files transmitted with it are confidential and > intended solely for the use of the individual or entity to whom they > are addressed. If you are not the intended recipient you are hereby > notified that any dissemination, forwarding, copying or use of any of > the information is strictly prohibited, and the e-mail should > immediately be deleted. OBASE makes no warranty as to the accuracy or > completeness of any information contained in this message and hereby > excludes any liability of any kind for the information contained > therein or for the information transmission, recepxion, storage or use > of such in any way whatsoever. The opinions expressed in this message > belong to sender alone and may not necessarily reflect the opinions of OBASE. > > This e-mail has been scanned for all known computer viruses. >
