[
https://issues.apache.org/jira/browse/SPARK-48009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841213#comment-17841213
]
Rushikesh Kavar commented on SPARK-48009:
-----------------------------------------
import org.apache.spark.sql.SaveMode;
import org.example.avro.Writer;
import java.util.ArrayList;
import java.util.List;
public class OverrideAvro {
public static void main(String[] args) {
// C:\Users\kavarus\testing\spark-testing\data
Writer.writeAvro(getMockData(),
"C:\\Users\\kavarus\\testing\\spark-testing\\data", SaveMode.Overwrite);
}
public static List<Modal> getMockData() {
List<Modal> lst = new ArrayList<>();
lst.add(new Modal("1", "Test1", 26));
lst.add(new Modal("2", "Test2", 28));
return lst;
}
}
> Specifications for Apache Spark hadoop Avro append operation
> ------------------------------------------------------------
>
> Key: SPARK-48009
> URL: https://issues.apache.org/jira/browse/SPARK-48009
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.3
> Reporter: Rushikesh Kavar
> Priority: Minor
>
> Consider a path /a/b/c
> Assume, I write the avro to folder using apache spark.
> After it is written, Assume I try to append dataset to this to folder.
> I want to see the specification of what happens in case of append.
> After doing PoC, I found out that when dataet which is appended is having
> same schema as of existing data, data gets just appended. But I want to see
> clear docs of what happens exactly in case of append.
> I am attaching my testing java code.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
