JavaBean serialization with cyclic bean attributes
Hi, we have a more or less small problem with DataFrame creation. In our current project we have a more or less complex data model corresponding to documents and word positions in it. In the last month we refactor our architecture to use neo4j as persistence. Before that we used PostgreSQL and try to implement all old algorithms with spark. Currently we are not using a spark database connector but we will integrate it in the next few weeks. Now we have the problem, that neo4j in combination with spring data had a impact on our domain models such that we have the a reference of models the current object relates to. That ends in cyclic references and we don’t know how to resolve this issue. The problem is, that we already implement a database abstraction layer in a fashion that we don’t have to worry about how to get the data, we only have to specify which data we need. So we use reflection at a few locations. We have now a few approaches in mind: 1. Ask our domain for simple objects without cyclic references 2. Implement our own Encoders Are we missing something. We appreciate every hint so resolve this. Kind Regards, Pascal Stammer - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Running Spark und YARN on AWS EMR
Hi, I am running a Spark 2.1.x Application on AWS EMR with YARN and get following error that kill my application: AM Container for appattempt_1500320286695_0001_01 exited with exitCode: -104 For more detailed output, check application tracking page:http://ip-172-31-35-192.eu-central-1.compute.internal:8088/cluster/app/application_1500320286695_0001Then, click on links to logs of each attempt. Diagnostics: Container [pid=9216,containerID=container_1500320286695_0001_01_01] is running beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical memory used; 3.3 GB of 6.9 GB virtual memory used. Killing container. I already change spark.yarn.executor.memoryOverhead but the error still occurs. Does anybody have a hint for me which parameter or configuration I have to adapt. Thank you very much. Regards, Pascal Stammer
Re: Running Spark und YARN on AWS EMR
Hi Takashi, thanks for your help. After a further investigation, I figure out that the killed container was the driver process. After setting spark.yarn.driver.memoryOverhead instead of spark.yarn.executor.memoryOverhead the error was gone and application is executed without error. Maybe it will help you as well. Regards, Pascal > Am 17.07.2017 um 22:59 schrieb Takashi Sasaki <tsasaki...@gmail.com>: > > Hi Pascal, > > The error also occurred frequently in our project. > > As a solution, it was effective to specify the memory size directly > with spark-submit command. > > eg. spark-submit executor-memory 2g > > > Regards, > > Takashi > >> 2017-07-18 5:18 GMT+09:00 Pascal Stammer <stam...@deichbrise.de>: >>> Hi, >>> >>> I am running a Spark 2.1.x Application on AWS EMR with YARN and get >>> following error that kill my application: >>> >>> AM Container for appattempt_1500320286695_0001_01 exited with exitCode: >>> -104 >>> For more detailed output, check application tracking >>> page:http://ip-172-31-35-192.eu-central-1.compute.internal:8088/cluster/app/application_1500320286695_0001Then, >>> click on links to logs of each attempt. >>> Diagnostics: Container >>> [pid=9216,containerID=container_1500320286695_0001_01_01] is running >>> beyond physical memory limits. Current usage: 1.4 GB of 1.4 GB physical >>> memory used; 3.3 GB of 6.9 GB virtual memory used. Killing container. >>> >>> >>> I already change spark.yarn.executor.memoryOverhead but the error still >>> occurs. Does anybody have a hint for me which parameter or configuration I >>> have to adapt. >>> >>> Thank you very much. >>> >>> Regards, >>> >>> Pascal Stammer >>> >>> > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org >
Logging in unit tests
Hi all, I am writing unit tests for my spark application. In the rest of the project I am using log4j2.xml files to configure logging. Now I am running in some issues and need the full executor and driver logs but I can’t get logging working in my unit tests. Any hint how to configure the logging? I appreciate any help! Regards, Pascal Stammer
Re: Netty Issues
Hi, I don’t know how this should help. We use maven shade plugin. This behavior currently happen in local unit tests. Pascal > Am 21.08.2017 um 12:58 schrieb 周康 <zhoukang199...@gmail.com>: > > Use maven shade plugin may help > > 2017-08-21 18:43 GMT+08:00 Pascal Stammer <stam...@deichbrise.de > <mailto:stam...@deichbrise.de>>: > Hi all, > > i got following exception: > > 17/08/21 12:33:56 ERROR TransportClient: Failed to send RPC > 5493448667271613330 to /10.210.85.3:52482 <http://10.210.85.3:52482/>: > java.lang.AbstractMethodError > java.lang.AbstractMethodError > at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73) > at > io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107) > at > io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811) > at > io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724) > at > io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111) > at > io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739) > at > io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731) > at > io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817) > at > io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724) > at > io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305) > at > io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739) > at > io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731) > at > io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) > at > io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1090) > at > io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1137) > at > io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1079) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445) > at > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > at java.lang.Thread.run(Thread.java:745) > > > We think that this is caused by incompatible versions of netty. We also have > a transitive dependency in neo4j dependencies. They are using > io.netty:netty-all.4.1.8.Final … Does anybody can provide some help? > > Regards, > Pascal Stammer > > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > >
Netty Issues
Hi all, i got following exception: 17/08/21 12:33:56 ERROR TransportClient: Failed to send RPC 5493448667271613330 to /10.210.85.3:52482: java.lang.AbstractMethodError java.lang.AbstractMethodError at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73) at io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731) at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1090) at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1137) at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1079) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745) We think that this is caused by incompatible versions of netty. We also have a transitive dependency in neo4j dependencies. They are using io.netty:netty-all.4.1.8.Final … Does anybody can provide some help? Regards, Pascal Stammer - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
