[
https://issues.apache.org/jira/browse/LIVY-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Larry McCay updated LIVY-759:
-----------------------------
Fix Version/s: 0.9.0
> Dataset Union not working for large datasets when submitting spark jon
> through livy
> -----------------------------------------------------------------------------------
>
> Key: LIVY-759
> URL: https://issues.apache.org/jira/browse/LIVY-759
> Project: Livy
> Issue Type: Bug
> Reporter: Mayukh Hazra
> Priority: Blocker
> Fix For: 0.9.0
>
>
> Hi,
> Am trying to perform an union of two large datsets (Coming from ORC or
> parquet file). We have that business logic in a java file and we are
> submitting the jar through *Livy* service for spark-submit. But it is ending
> with error stating
> *java.lang.IllegalStateException: No YARN application is found with tag
> livy-batch-43-0prln6cp in 120 seconds. This may be because 1) spark-submit
> fail to submit application to YARN; or 2) YARN cluster doesn't have enough
> resources to start the application in time. Please check Livy log and YARN
> log to know the details.*
> Our code looks like below
> Dataset<Row> mainDataset = spark.sql("Select * from tab1");
> Dataset<Row> anotherDataset; (This also has same number of columns);
> Dataset<Row> newDataset=mainDataset.union(anotherDataset);
>
> *Notes:*
> Row count of the datasets are approx 60K
> Column count of the datasets are approx 300
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
