Re: Select Columns from Dataframe in Java
Hi, Have a look here - https://repost.aws/knowledge-center/spark-driver-logs-emr-cluster. Usually, you have application logs out-of-the-box in the driver stdout. It looks like "s3://aws-logs--us-east-1/elasticmapreduce/j-35PUYZBQVIJNM/containers/application_1572839353552_0008/container_1572839353552_0008_01_01/stdout.gz". Grisha On Sat, Dec 30, 2023 at 2:49 PM PRASHANT L wrote: > Hi Grisha > This is Great :) It worked thanks alot > > I have this requirement , I will be running my spark application on EMR > and build a custom logging to create logs on S3. Any idea what should I do? > or In general if i create a custom log (with my Application name ), where > will logs be generated when run in cluster mode (since in cluster mode jobs > are executed all over different machine) > > On Sat, Dec 30, 2023 at 1:56 PM Grisha Weintraub < > grisha.weintr...@gmail.com> wrote: > >> In Java, it expects an array of Columns, so you can simply cast your list >> to an array: >> >> array_df.select(fields.toArray(new Column[0])) >> >> >> >> >> On Fri, Dec 29, 2023 at 10:58 PM PRASHANT L >> wrote: >> >>> >>> Team >>> I am using Java and want to select columns from Dataframe , columns are >>> stored in List >>> equivalent of below scala code >>> * array_df=array_df.select(fields: _*)* >>> >>> >>> When I try array_df=array_df.select(fields) , I get error saying Cast to >>> Column >>> >>> I am using Spark 3.4 >>> >>
Re: Select Columns from Dataframe in Java
Hi Grisha This is Great :) It worked thanks alot I have this requirement , I will be running my spark application on EMR and build a custom logging to create logs on S3. Any idea what should I do? or In general if i create a custom log (with my Application name ), where will logs be generated when run in cluster mode (since in cluster mode jobs are executed all over different machine) On Sat, Dec 30, 2023 at 1:56 PM Grisha Weintraub wrote: > In Java, it expects an array of Columns, so you can simply cast your list > to an array: > > array_df.select(fields.toArray(new Column[0])) > > > > > On Fri, Dec 29, 2023 at 10:58 PM PRASHANT L wrote: > >> >> Team >> I am using Java and want to select columns from Dataframe , columns are >> stored in List >> equivalent of below scala code >> * array_df=array_df.select(fields: _*)* >> >> >> When I try array_df=array_df.select(fields) , I get error saying Cast to >> Column >> >> I am using Spark 3.4 >> >
Re: Select Columns from Dataframe in Java
In Java, it expects an array of Columns, so you can simply cast your list to an array: array_df.select(fields.toArray(new Column[0])) On Fri, Dec 29, 2023 at 10:58 PM PRASHANT L wrote: > > Team > I am using Java and want to select columns from Dataframe , columns are > stored in List > equivalent of below scala code > * array_df=array_df.select(fields: _*)* > > > When I try array_df=array_df.select(fields) , I get error saying Cast to > Column > > I am using Spark 3.4 >
Select Columns from Dataframe in Java
Team I am using Java and want to select columns from Dataframe , columns are stored in List equivalent of below scala code * array_df=array_df.select(fields: _*)* When I try array_df=array_df.select(fields) , I get error saying Cast to Column I am using Spark 3.4