Encountered unregistered class ID
Hi to all, I'm running a job (with Flink 0.10-SNAPSHOT) that reads some parquet-thrift objects and then it performs some joins and I receive the following exception: Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: Encountered unregistered class ID: 93 at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784) Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 93 at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225) at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235) at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71) at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328) at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781) What could be the cause of this? Best, Flavio
RE: Application-specific loggers configuration
Hi ! Yes, we’re starting our job with “flink run --jobmanager yarn-cluster” So it’s perfect, we’ll use your fix and, when it’s out, we’ll switch to flink 0.9.1. B.R. From: Aljoscha Krettek [mailto:aljos...@apache.org] Sent: mardi 25 août 2015 19:25 To: user@flink.apache.org Subject: Re: Application-specific loggers configuration Hi Gwenhaël, are you using the one-yarn-cluster-per-job mode of Flink? I.e., you are starting your Flink job with (from the doc): flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar If you are, then this is almost possible on the current version of Flink. What you have to do is copy the conf directory of Flink to a separate directory that is specific to your job. There you make your modifications to the log configuration etc. Then, when you start your job you do this instead: export FLINK_CONF_DIR=/path/to/my/conf flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar You can easily put this into your startup script, of course. I said almost possible because this requires a small fix in bin/flink. Around line 130 this line: FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf needs to be replaced by this line: if [ -z "$FLINK_CONF_DIR" ]; then FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf; fi (We will fix this in the upcoming version and the 0.9.1 bugfix release.) Does this help? Let us know if you are not using the one-yarn-cluster-per-job mode, then we'll have to try to find another solution. Best, Aljoscha On Tue, 25 Aug 2015 at 16:22 Gwenhael Pasquiers mailto:gwenhael.pasqui...@ericsson.com>> wrote: Hi, We’re developing the first of (we hope) many flink streaming app. We’d like to package the logging configuration (log4j) together with the jar. Meaning, different application will probably have different logging configuration (ex: to different logstash ports) … Is there a way to “override” the many log4j properties files that are in flink/conf./*.properties ? In our environment, the flink binaries would be on the PATH, and our apps would be : - Jar file - App configuration files - Log configuration files - Startup script B.R. Gwenhaël PASQUIERS
Re: Encountered unregistered class ID
Hi Flavio! That exception means that the Kryo serializers are not in sync. The writers have registered types that the readers do not know. Two possible reasons that I can think of from the top of my head: 1) Do you manually register types? Are you registering new types in the middle of your program? 2) Flink may not be passing the Kryo registration properly down to the sort reader. Greetings, Stephan On Wed, Aug 26, 2015 at 9:04 AM, Flavio Pompermaier wrote: > > Hi to all, > I'm running a job (with Flink 0.10-SNAPSHOT) that reads some > parquet-thrift objects and then it performs some joins and I receive the > following exception: > > Caused by: java.io.IOException: Thread 'SortMerger spilling thread' > terminated due to an exception: Encountered unregistered class ID: 93 > at > org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784) > Caused by: com.esotericsoftware.kryo.KryoException: Encountered > unregistered class ID: 93 > at > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) > at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225) > at > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235) > at > org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71) > at > org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522) > at > org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328) > at > org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781) > > What could be the cause of this? > > Best, > Flavio > > >
Re: Encountered unregistered class ID
I'm running my job from my Eclipse and I don't register any Kryo class in the env. On Wed, Aug 26, 2015 at 10:34 AM, Stephan Ewen wrote: > Hi Flavio! > > That exception means that the Kryo serializers are not in sync. The > writers have registered types that the readers do not know. > > Two possible reasons that I can think of from the top of my head: > > 1) Do you manually register types? Are you registering new types in the > middle of your program? > > 2) Flink may not be passing the Kryo registration properly down to the > sort reader. > > > Greetings, > Stephan > > > On Wed, Aug 26, 2015 at 9:04 AM, Flavio Pompermaier > wrote: > >> >> Hi to all, >> I'm running a job (with Flink 0.10-SNAPSHOT) that reads some >> parquet-thrift objects and then it performs some joins and I receive the >> following exception: >> >> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' >> terminated due to an exception: Encountered unregistered class ID: 93 >> at >> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784) >> Caused by: com.esotericsoftware.kryo.KryoException: Encountered >> unregistered class ID: 93 >> at >> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) >> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) >> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) >> at >> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211) >> at >> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225) >> at >> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235) >> at >> org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71) >> at >> org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522) >> at >> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328) >> at >> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781) >> >> What could be the cause of this? >> >> Best, >> Flavio >> >> >> >
Re: Encountered unregistered class ID
Hi Flavio, can you share a minimal version of your program to reproduce the issue? On Wed, Aug 26, 2015 at 10:36 AM, Flavio Pompermaier wrote: > I'm running my job from my Eclipse and I don't register any Kryo class in > the env. > > On Wed, Aug 26, 2015 at 10:34 AM, Stephan Ewen wrote: > >> Hi Flavio! >> >> That exception means that the Kryo serializers are not in sync. The >> writers have registered types that the readers do not know. >> >> Two possible reasons that I can think of from the top of my head: >> >> 1) Do you manually register types? Are you registering new types in the >> middle of your program? >> >> 2) Flink may not be passing the Kryo registration properly down to the >> sort reader. >> >> >> Greetings, >> Stephan >> >> >> On Wed, Aug 26, 2015 at 9:04 AM, Flavio Pompermaier > > wrote: >> >>> >>> Hi to all, >>> I'm running a job (with Flink 0.10-SNAPSHOT) that reads some >>> parquet-thrift objects and then it performs some joins and I receive the >>> following exception: >>> >>> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' >>> terminated due to an exception: Encountered unregistered class ID: 93 >>> at >>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:784) >>> Caused by: com.esotericsoftware.kryo.KryoException: Encountered >>> unregistered class ID: 93 >>> at >>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) >>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) >>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) >>> at >>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:211) >>> at >>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:225) >>> at >>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(KryoSerializer.java:235) >>> at >>> org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(TupleSerializerBase.java:71) >>> at >>> org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:522) >>> at >>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1328) >>> at >>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781) >>> >>> What could be the cause of this? >>> >>> Best, >>> Flavio >>> >>> >>> >> > >
[0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Hi, Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m address since it does not find it any longer. Regards, Arnaud L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Hi Arnaud, usually, you don't have to manually specify the JobManager address manually with the -m argument, because it is reading it from the conf/.yarn-session.properties file. Give me a few minutes to reproduce the issue. On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud wrote: > Hi, > Using last nightly build, it seems that if you call yarn-session.sh with > -nm option to give a nice application name, then you cannot submit a job > with flink run without specify the ever changing -m address > since it does not find it any longer. > > Regards, > > Arnaud > > -- > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. >
RE: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Ooops… Seems it was rather a write problem on the conf dir… Sorry, it works! BTW, it’s not really nice to have an application write the configuration dir ; it’s often a root protected directory in usr/lib/flink. Is there a parameter to put that file elsewhere ? De : Robert Metzger [mailto:rmetz...@apache.org] Envoyé : mercredi 26 août 2015 14:42 À : user@flink.apache.org Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails. Hi Arnaud, usually, you don't have to manually specify the JobManager address manually with the -m argument, because it is reading it from the conf/.yarn-session.properties file. Give me a few minutes to reproduce the issue. On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud mailto:al...@bouyguestelecom.fr>> wrote: Hi, Using last nightly build, it seems that if you call yarn-session.sh with -nm option to give a nice application name, then you cannot submit a job with flink run without specify the ever changing -m address since it does not find it any longer. Regards, Arnaud L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Hi Arnaud, I think my answer to Gwenhaël could also be helpful to you: are you using the one-yarn-cluster-per-job mode of Flink? I.e., you are starting your Flink job with (from the doc): flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar If you are, then this is almost possible on the current version of Flink. What you have to do is copy the conf directory of Flink to a separate directory that is specific to your job. There you make your modifications to the log configuration etc. Then, when you start your job you do this instead: export FLINK_CONF_DIR=/path/to/my/conf flink run -m yarn-cluster -yn 4 -yjm 1024 -ytm 4096 ./examples/flink-java-examples-0.10-SNAPSHOT-WordCount.jar You can easily put this into your startup script, of course. I said almost possible because this requires a small fix in bin/flink. Around line 130 this line: FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf needs to be replaced by this line: if [ -z "$FLINK_CONF_DIR" ]; then FLINK_CONF_DIR=$FLINK_ROOT_DIR_MANGLED/conf; fi (We will fix this in the upcoming version and the 0.9.1 bugfix release.) Does this help? Let us know if you are not using the one-yarn-cluster-per-job mode, then we'll have to try to find another solution. Forget the stuff about one-yarn-cluster-per-job but the other stuff should be right what you want. I hope it helps. Best, Aljoscha On Wed, 26 Aug 2015 at 15:19 LINZ, Arnaud wrote: > Ooops… Seems it was rather a write problem on the conf dir… > > Sorry, it works! > > > > BTW, it’s not really nice to have an application write the configuration > dir ; it’s often a root protected directory in usr/lib/flink. Is there a > parameter to put that file elsewhere ? > > > > > > *De :* Robert Metzger [mailto:rmetz...@apache.org] > *Envoyé :* mercredi 26 août 2015 14:42 > *À :* user@flink.apache.org > *Objet :* Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session > -nm), flink run without -m fails. > > > > Hi Arnaud, > > > > usually, you don't have to manually specify the JobManager address > manually with the -m argument, because it is reading it from the > conf/.yarn-session.properties file. > > > > Give me a few minutes to reproduce the issue. > > > > On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud > wrote: > > Hi, > Using last nightly build, it seems that if you call yarn-session.sh with > -nm option to give a nice application name, then you cannot submit a job > with flink run without specify the ever changing -m address > since it does not find it any longer. > > Regards, > > Arnaud > > > -- > > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. > > >
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Great ;) Not yet, but you are the second user to request this. I think I'll put the file somewhere else now. On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud wrote: > Ooops… Seems it was rather a write problem on the conf dir… > > Sorry, it works! > > > > BTW, it’s not really nice to have an application write the configuration > dir ; it’s often a root protected directory in usr/lib/flink. Is there a > parameter to put that file elsewhere ? > > > > > > *De :* Robert Metzger [mailto:rmetz...@apache.org] > *Envoyé :* mercredi 26 août 2015 14:42 > *À :* user@flink.apache.org > *Objet :* Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session > -nm), flink run without -m fails. > > > > Hi Arnaud, > > > > usually, you don't have to manually specify the JobManager address > manually with the -m argument, because it is reading it from the > conf/.yarn-session.properties file. > > > > Give me a few minutes to reproduce the issue. > > > > On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud > wrote: > > Hi, > Using last nightly build, it seems that if you call yarn-session.sh with > -nm option to give a nice application name, then you cannot submit a job > with flink run without specify the ever changing -m address > since it does not find it any longer. > > Regards, > > Arnaud > > > -- > > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. > > >
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Can't we write the file to the system's temp directory or the user home? IMHO this is more standard practice for these type of session information. On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger wrote: > Great ;) > > Not yet, but you are the second user to request this. > I think I'll put the file somewhere else now. > > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud > wrote: >> >> Ooops… Seems it was rather a write problem on the conf dir… >> >> Sorry, it works! >> >> >> >> BTW, it’s not really nice to have an application write the configuration >> dir ; it’s often a root protected directory in usr/lib/flink. Is there a >> parameter to put that file elsewhere ? >> >> >> >> >> >> De : Robert Metzger [mailto:rmetz...@apache.org] >> Envoyé : mercredi 26 août 2015 14:42 >> À : user@flink.apache.org >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session >> -nm), flink run without -m fails. >> >> >> >> Hi Arnaud, >> >> >> >> usually, you don't have to manually specify the JobManager address >> manually with the -m argument, because it is reading it from the >> conf/.yarn-session.properties file. >> >> >> >> Give me a few minutes to reproduce the issue. >> >> >> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud >> wrote: >> >> Hi, >> Using last nightly build, it seems that if you call yarn-session.sh with >> -nm option to give a nice application name, then you cannot submit a job >> with flink run without specify the ever changing -m address >> since it does not find it any longer. >> >> Regards, >> >> Arnaud >> >> >> >> >> >> >> L'intégrité de ce message n'étant pas assurée sur internet, la société >> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces >> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous >> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir >> l'expéditeur. >> >> The integrity of this message cannot be guaranteed on the Internet. The >> company that sent this message cannot therefore be held liable for its >> content nor attachments. Any unauthorized use or dissemination is >> prohibited. If you are not the intended recipient of this message, then >> please delete it and notify the sender. >> >> > >
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Yep. I think the start-*.sh scripts are also writing the PID to tmp. On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels wrote: > Can't we write the file to the system's temp directory or the user > home? IMHO this is more standard practice for these type of session > information. > > On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger > wrote: > > Great ;) > > > > Not yet, but you are the second user to request this. > > I think I'll put the file somewhere else now. > > > > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud > > wrote: > >> > >> Ooops… Seems it was rather a write problem on the conf dir… > >> > >> Sorry, it works! > >> > >> > >> > >> BTW, it’s not really nice to have an application write the configuration > >> dir ; it’s often a root protected directory in usr/lib/flink. Is there a > >> parameter to put that file elsewhere ? > >> > >> > >> > >> > >> > >> De : Robert Metzger [mailto:rmetz...@apache.org] > >> Envoyé : mercredi 26 août 2015 14:42 > >> À : user@flink.apache.org > >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session > >> -nm), flink run without -m fails. > >> > >> > >> > >> Hi Arnaud, > >> > >> > >> > >> usually, you don't have to manually specify the JobManager address > >> manually with the -m argument, because it is reading it from the > >> conf/.yarn-session.properties file. > >> > >> > >> > >> Give me a few minutes to reproduce the issue. > >> > >> > >> > >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud > > >> wrote: > >> > >> Hi, > >> Using last nightly build, it seems that if you call yarn-session.sh with > >> -nm option to give a nice application name, then you cannot submit a job > >> with flink run without specify the ever changing -m address > >> since it does not find it any longer. > >> > >> Regards, > >> > >> Arnaud > >> > >> > >> > >> > >> > >> > >> L'intégrité de ce message n'étant pas assurée sur internet, la société > >> expéditrice ne peut être tenue responsable de son contenu ni de ses > pièces > >> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous > >> n'êtes pas destinataire de ce message, merci de le détruire et d'avertir > >> l'expéditeur. > >> > >> The integrity of this message cannot be guaranteed on the Internet. The > >> company that sent this message cannot therefore be held liable for its > >> content nor attachments. Any unauthorized use or dissemination is > >> prohibited. If you are not the intended recipient of this message, then > >> please delete it and notify the sender. > >> > >> > > > > >
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Therefore, my change will include a configuration option to set a custom location for the file. On Wed, Aug 26, 2015 at 5:55 PM, Maximilian Michels wrote: > The only problem with writing the temp is that it will be gone after a > restart. While this is not important for PIDs because the system has > been restarted anyways, this can actually be a problem if you want to > resume a YARN cluster after you have restarted your system. > > On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger > wrote: > > Yep. I think the start-*.sh scripts are also writing the PID to tmp. > > > > On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels > wrote: > >> > >> Can't we write the file to the system's temp directory or the user > >> home? IMHO this is more standard practice for these type of session > >> information. > >> > >> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger > >> wrote: > >> > Great ;) > >> > > >> > Not yet, but you are the second user to request this. > >> > I think I'll put the file somewhere else now. > >> > > >> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud < > al...@bouyguestelecom.fr> > >> > wrote: > >> >> > >> >> Ooops… Seems it was rather a write problem on the conf dir… > >> >> > >> >> Sorry, it works! > >> >> > >> >> > >> >> > >> >> BTW, it’s not really nice to have an application write the > >> >> configuration > >> >> dir ; it’s often a root protected directory in usr/lib/flink. Is > there > >> >> a > >> >> parameter to put that file elsewhere ? > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> De : Robert Metzger [mailto:rmetz...@apache.org] > >> >> Envoyé : mercredi 26 août 2015 14:42 > >> >> À : user@flink.apache.org > >> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application > (yarn-session > >> >> -nm), flink run without -m fails. > >> >> > >> >> > >> >> > >> >> Hi Arnaud, > >> >> > >> >> > >> >> > >> >> usually, you don't have to manually specify the JobManager address > >> >> manually with the -m argument, because it is reading it from the > >> >> conf/.yarn-session.properties file. > >> >> > >> >> > >> >> > >> >> Give me a few minutes to reproduce the issue. > >> >> > >> >> > >> >> > >> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud > >> >> > >> >> wrote: > >> >> > >> >> Hi, > >> >> Using last nightly build, it seems that if you call yarn-session.sh > >> >> with > >> >> -nm option to give a nice application name, then you cannot submit a > >> >> job > >> >> with flink run without specify the ever changing -m > >> >> address > >> >> since it does not find it any longer. > >> >> > >> >> Regards, > >> >> > >> >> Arnaud > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> L'intégrité de ce message n'étant pas assurée sur internet, la > société > >> >> expéditrice ne peut être tenue responsable de son contenu ni de ses > >> >> pièces > >> >> jointes. Toute utilisation ou diffusion non autorisée est interdite. > Si > >> >> vous > >> >> n'êtes pas destinataire de ce message, merci de le détruire et > >> >> d'avertir > >> >> l'expéditeur. > >> >> > >> >> The integrity of this message cannot be guaranteed on the Internet. > The > >> >> company that sent this message cannot therefore be held liable for > its > >> >> content nor attachments. Any unauthorized use or dissemination is > >> >> prohibited. If you are not the intended recipient of this message, > then > >> >> please delete it and notify the sender. > >> >> > >> >> > >> > > >> > > > > > >
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
Nice. More configuration options :) On Wed, Aug 26, 2015 at 5:58 PM, Robert Metzger wrote: > Therefore, my change will include a configuration option to set a custom > location for the file. > > On Wed, Aug 26, 2015 at 5:55 PM, Maximilian Michels wrote: >> >> The only problem with writing the temp is that it will be gone after a >> restart. While this is not important for PIDs because the system has >> been restarted anyways, this can actually be a problem if you want to >> resume a YARN cluster after you have restarted your system. >> >> On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger >> wrote: >> > Yep. I think the start-*.sh scripts are also writing the PID to tmp. >> > >> > On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels >> > wrote: >> >> >> >> Can't we write the file to the system's temp directory or the user >> >> home? IMHO this is more standard practice for these type of session >> >> information. >> >> >> >> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger >> >> wrote: >> >> > Great ;) >> >> > >> >> > Not yet, but you are the second user to request this. >> >> > I think I'll put the file somewhere else now. >> >> > >> >> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud >> >> > >> >> > wrote: >> >> >> >> >> >> Ooops… Seems it was rather a write problem on the conf dir… >> >> >> >> >> >> Sorry, it works! >> >> >> >> >> >> >> >> >> >> >> >> BTW, it’s not really nice to have an application write the >> >> >> configuration >> >> >> dir ; it’s often a root protected directory in usr/lib/flink. Is >> >> >> there >> >> >> a >> >> >> parameter to put that file elsewhere ? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> De : Robert Metzger [mailto:rmetz...@apache.org] >> >> >> Envoyé : mercredi 26 août 2015 14:42 >> >> >> À : user@flink.apache.org >> >> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application >> >> >> (yarn-session >> >> >> -nm), flink run without -m fails. >> >> >> >> >> >> >> >> >> >> >> >> Hi Arnaud, >> >> >> >> >> >> >> >> >> >> >> >> usually, you don't have to manually specify the JobManager address >> >> >> manually with the -m argument, because it is reading it from the >> >> >> conf/.yarn-session.properties file. >> >> >> >> >> >> >> >> >> >> >> >> Give me a few minutes to reproduce the issue. >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud >> >> >> >> >> >> wrote: >> >> >> >> >> >> Hi, >> >> >> Using last nightly build, it seems that if you call yarn-session.sh >> >> >> with >> >> >> -nm option to give a nice application name, then you cannot submit a >> >> >> job >> >> >> with flink run without specify the ever changing -m >> >> >> address >> >> >> since it does not find it any longer. >> >> >> >> >> >> Regards, >> >> >> >> >> >> Arnaud >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> L'intégrité de ce message n'étant pas assurée sur internet, la >> >> >> société >> >> >> expéditrice ne peut être tenue responsable de son contenu ni de ses >> >> >> pièces >> >> >> jointes. Toute utilisation ou diffusion non autorisée est interdite. >> >> >> Si >> >> >> vous >> >> >> n'êtes pas destinataire de ce message, merci de le détruire et >> >> >> d'avertir >> >> >> l'expéditeur. >> >> >> >> >> >> The integrity of this message cannot be guaranteed on the Internet. >> >> >> The >> >> >> company that sent this message cannot therefore be held liable for >> >> >> its >> >> >> content nor attachments. Any unauthorized use or dissemination is >> >> >> prohibited. If you are not the intended recipient of this message, >> >> >> then >> >> >> please delete it and notify the sender. >> >> >> >> >> >> >> >> > >> >> > >> > >> > > >
Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session -nm), flink run without -m fails.
The only problem with writing the temp is that it will be gone after a restart. While this is not important for PIDs because the system has been restarted anyways, this can actually be a problem if you want to resume a YARN cluster after you have restarted your system. On Wed, Aug 26, 2015 at 3:34 PM, Robert Metzger wrote: > Yep. I think the start-*.sh scripts are also writing the PID to tmp. > > On Wed, Aug 26, 2015 at 3:30 PM, Maximilian Michels wrote: >> >> Can't we write the file to the system's temp directory or the user >> home? IMHO this is more standard practice for these type of session >> information. >> >> On Wed, Aug 26, 2015 at 3:25 PM, Robert Metzger >> wrote: >> > Great ;) >> > >> > Not yet, but you are the second user to request this. >> > I think I'll put the file somewhere else now. >> > >> > On Wed, Aug 26, 2015 at 3:19 PM, LINZ, Arnaud >> > wrote: >> >> >> >> Ooops… Seems it was rather a write problem on the conf dir… >> >> >> >> Sorry, it works! >> >> >> >> >> >> >> >> BTW, it’s not really nice to have an application write the >> >> configuration >> >> dir ; it’s often a root protected directory in usr/lib/flink. Is there >> >> a >> >> parameter to put that file elsewhere ? >> >> >> >> >> >> >> >> >> >> >> >> De : Robert Metzger [mailto:rmetz...@apache.org] >> >> Envoyé : mercredi 26 août 2015 14:42 >> >> À : user@flink.apache.org >> >> Objet : Re: [0.10-SNAPSHOT ] When naming yarn application (yarn-session >> >> -nm), flink run without -m fails. >> >> >> >> >> >> >> >> Hi Arnaud, >> >> >> >> >> >> >> >> usually, you don't have to manually specify the JobManager address >> >> manually with the -m argument, because it is reading it from the >> >> conf/.yarn-session.properties file. >> >> >> >> >> >> >> >> Give me a few minutes to reproduce the issue. >> >> >> >> >> >> >> >> On Wed, Aug 26, 2015 at 2:39 PM, LINZ, Arnaud >> >> >> >> wrote: >> >> >> >> Hi, >> >> Using last nightly build, it seems that if you call yarn-session.sh >> >> with >> >> -nm option to give a nice application name, then you cannot submit a >> >> job >> >> with flink run without specify the ever changing -m >> >> address >> >> since it does not find it any longer. >> >> >> >> Regards, >> >> >> >> Arnaud >> >> >> >> >> >> >> >> >> >> >> >> >> >> L'intégrité de ce message n'étant pas assurée sur internet, la société >> >> expéditrice ne peut être tenue responsable de son contenu ni de ses >> >> pièces >> >> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si >> >> vous >> >> n'êtes pas destinataire de ce message, merci de le détruire et >> >> d'avertir >> >> l'expéditeur. >> >> >> >> The integrity of this message cannot be guaranteed on the Internet. The >> >> company that sent this message cannot therefore be held liable for its >> >> content nor attachments. Any unauthorized use or dissemination is >> >> prohibited. If you are not the intended recipient of this message, then >> >> please delete it and notify the sender. >> >> >> >> >> > >> > > >
HadoopDataOutputStream maybe does not expose enough methods of org.apache.hadoop.fs.FSDataOutputStream
Hi, I’ve noticed that when you use org.apache.flink.core.fs.FileSystem to write into a hdfs file, calling org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(), it returns a HadoopDataOutputStream that wraps a org.apache.hadoop.fs.FSDataOutputStream (under its org.apache.hadoop.hdfs.client .HdfsDataOutputStream wrappper). However, FSDataOutputStream exposes many methods like flush, getPos etc, but HadoopDataOutputStream only wraps write & close. For instance, flush() calls the default, empty implementation of OutputStream instead of the hadoop one, and that’s confusing. Moreover, because of the restrictive OutputStream interface, hsync() and hflush() are not exposed to Flink ; maybe having a getWrappedStream() would be convenient. (For now, that prevents me from using Flink FileSystem object, I directly use hadoop’s one). Regards, Arnaud L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Re: HadoopDataOutputStream maybe does not expose enough methods of org.apache.hadoop.fs.FSDataOutputStream
I think that is a very good idea. Originally, we wrapped the Hadoop FS classes for convenience (they were changing, we wanted to keep the system independent of Hadoop), but these are no longer relevant reasons, in my opinion. Let's start with your proposal and see if we can actually get rid of the wrapping in a way that is friendly to existing users. Would you open an issue for this? Greetings, Stephan On Wed, Aug 26, 2015 at 6:23 PM, LINZ, Arnaud wrote: > Hi, > > > > I’ve noticed that when you use org.apache.flink.core.fs.FileSystem to > write into a hdfs file, calling > org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(), it returns a > HadoopDataOutputStream that wraps a > org.apache.hadoop.fs.FSDataOutputStream (under its > org.apache.hadoop.hdfs.client .HdfsDataOutputStream wrappper). > > > > However, FSDataOutputStream exposes many methods like flush, getPos etc, > but HadoopDataOutputStream only wraps write & close. > > > > For instance, flush() calls the default, empty implementation of > OutputStream instead of the hadoop one, and that’s confusing. Moreover, > because of the restrictive OutputStream interface, hsync() and hflush() are > not exposed to Flink ; maybe having a getWrappedStream() would be > convenient. > > > > (For now, that prevents me from using Flink FileSystem object, I directly > use hadoop’s one). > > > > Regards, > > Arnaud > > > > > > > > > > -- > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. >
New contributor tasks
Hi, I've setup Flink on my local linux machine and ran few examples as well. Also setup the Intellij IDE for the coding environment. Can anyone please let me know if there are any beginner tasks which I can take a look for contributing to Apache Flink codebase. I am comfortable in Java and Scala programming. Please let me know. Thanks, Naveen
Re: New contributor tasks
Hi Naveen, There is a guide document [1] about contribution in homepage. Please read first before contributing. Maybe a document of coding guidelines [2] would be helpful to you. You can find some issues [3] to start contributing to Flink in JIRA. The issues are labeled as `starter`, `newbie`, or `easyfix`. Happy contributing! Regards, Chiwan Park [1] http://flink.apache.org/how-to-contribute.html [2] http://flink.apache.org/coding-guidelines.html [3] https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20starter%20ORDER%20BY%20priority%20DESC > On Aug 27, 2015, at 2:06 PM, Naveen Madhire wrote: > > Hi, > > I've setup Flink on my local linux machine and ran few examples as well. Also > setup the Intellij IDE for the coding environment. Can anyone please let me > know if there are any beginner tasks which I can take a look for contributing > to Apache Flink codebase. > > I am comfortable in Java and Scala programming. > > Please let me know. > > Thanks, > Naveen
Re: New contributor tasks
Additionally, If you have any questions about contributing, please send a mail to dev mailing list. Regards, Chiwan Park > On Aug 27, 2015, at 2:11 PM, Chiwan Park wrote: > > Hi Naveen, > > There is a guide document [1] about contribution in homepage. Please read > first before contributing. Maybe a document of coding guidelines [2] would be > helpful to you. You can find some issues [3] to start contributing to Flink > in JIRA. The issues are labeled as `starter`, `newbie`, or `easyfix`. > > Happy contributing! > > Regards, > Chiwan Park > > [1] http://flink.apache.org/how-to-contribute.html > [2] http://flink.apache.org/coding-guidelines.html > [3] > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20starter%20ORDER%20BY%20priority%20DESC > >> On Aug 27, 2015, at 2:06 PM, Naveen Madhire wrote: >> >> Hi, >> >> I've setup Flink on my local linux machine and ran few examples as well. >> Also setup the Intellij IDE for the coding environment. Can anyone please >> let me know if there are any beginner tasks which I can take a look for >> contributing to Apache Flink codebase. >> >> I am comfortable in Java and Scala programming. >> >> Please let me know. >> >> Thanks, >> Naveen > > > >
